AI Score & Quality Measures (Legacy AI)

When to stop AI training and ensure high-quality results.

Training your AI model improves the accuracy of topic assignments, but at some point, it’s time to move forward. This article explains how to know when your model is ready, how to interpret the AI Score, and what steps to take if your results still need improvement.

When Should I Stop AI Training?

You can stop training your model when you’re confident in the quality of the results. We recommend using a combination of:

• Manual Review — Check a sample of your responses. If the topic assignments make sense and align with human judgment, that’s a strong sign the model is ready.
• AI Score — This score reflects how confidently the AI assigns topics. Use it as a benchmark for overall performance.

Review each topic 2–3 times to get a solid understanding of the model’s performance.

What Is the AI Score?

The AI Score shows how confident the model is in its own topic assignments. While the theoretical maximum is 100, this is never achieved in practice.

Here’s how to interpret it:

Scenario	Typical AI Score
A single person assigning topics twice	~90
Two different people assigning topics	70–80
Human-like AI performance	~70

How to Improve the AI Score

If your score is lower than expected, or if the topic assignments seem off, try the following strategies:

1. Simplify Your Topics

Large, overlapping topic sets can confuse the AI. Consider combining similar topics or removing less relevant ones.

Example: Combine “Late Delivery” and “Shipping Delays” into “Delivery Issues”.

2. Use Clear, Descriptive Labels

The AI learns from your topic names. Labels should be specific and easily understandable.

Avoid: “Misc.”, “Other”, “X1”

Use: “Website Usability”, “Suggestions for Improvement”

3. Review More Data

The more examples the AI has to learn from, the better it performs. If you have historical data with topics already assigned but not yet uploaded to Caplena, contact support to help you ingest it into your account.

4. Review Low-Confidence Topics

In the chart and table below, you’ll find topics where the AI has lower confidence in its assignments — these appear below the red line. These are the best candidates for additional review and refinement.

Why Are Some Topics Missing a Score?

If a topic is only assigned to a few responses, the AI might not have enough data to calculate a score. In rare cases, the AI may also struggle with topics that have unclear or overly abstract labels.