A lot is said about hallucinations, and we’re all aware that when AI returns a factually incorrect response there is the potential for serious consequences. Similarly, when safety and security standards and expectations are breached.
For reputable providers of conversational AI solutions optimising for accuracy and safety is standard. However, factual accuracy at the sentence level doesn’t automatically mean conversational success. Conversational failure will be a certainty unless the interaction expectations of a user are met.
Great conversational AI is much more than accuracy and safety
Organisations cannot afford to focus solely on factual accuracy and safety. Getting a pass mark for accuracy but failing at the conversational level can have negative financial, social, and reputational impact.
AI designed to avoid being wrong, offensive, or unsafe is critically necessary. But humans expect a human conversation. They don’t want an AI conversation that is repetitive, predictable, overly cautious, or devoid of personality
User interaction expectations
When users interact with a conversational AI system, the expectation is for basic conversational norms to shape the conversational flow. And with AI increasingly sounding more human-like, if basic conversational norms are not followed, they become more noticeable (and frustrating) to the user.
The main user expectation when interacting with conversational AI is for the conversation to follow the same social progression and norms as a human-to-human interaction would.
Basic conversational norms
Users expect AI interactions to take the form of basic conversational norms such as:
- Remembering what’s been said
- Not having to unnecessarily repeat things
- Update your understanding if you’ve misunderstood me
- Not going off on a tangent
- Not attributing or saying things that haven’t been said
- Saying if you’re assuming something, or don’t know
- Not continually apologising and fixing the problem
- Not getting flustered or giving up if something goes wrong
- Knowing when the conversation has come to an end
Conversational success
The foundation of conversational success is all about understanding and context. There must be an understanding of what words mean, what is being referred to in the conversation, what is hoped to be accomplished, the emotional significance of the situation, the sort of response that would actually be of help and knowing how this event fits into any larger story.
It is possible that understanding in any of these areas may be weaker or stronger. Simply asking whether the AI understood the user, is too basic if conversational success is the wanted outcome.
To ensure organisations can ensure both factual accuracy and conversational success, Creative Virtual enables analysis of understanding across the areas of intent, context, trust, clarification as well as accuracy and safety,
Users remember the conversational fails
Successful mitigation of conversational failures doesn’t happen by accident; it must be designed.
The user stuck in the ‘I don’t understand loop’, being overly apologised to, told they said or meant something they haven’t, or whose intention hasn’t been understood remembers these interactions. Whilst the exchange may be factually accurate, there can very well be interactional failures. And for the user it is a bad experience.
Clarification failure
Everyone will have experienced a time where the AI repeatedly asks for more information, even when you’ve clarified it previously. Instead of using that clarification input to narrow the uncertainty, the AI keeps asking for more information. It gets stuck in a repair loop it can’t untangle itself from. Users want progressive clarification, not a conversational reset.
Conversational boredom
Humans very quickly exit boring conversational partners in real life, yet in conversational AI not all providers are measuring or discussing potential conversational AI boredom, to their detriment.
Conversational AI providers should be designing with anti-boredom capabilities to ensure that repetitive or over long responses can be avoided.
Illusion of understanding
AI has become particularly good at mirroring language. But mirroring language does not equal understanding intent. Unfortunately, for many users they think it does.
We all know that conversational AI can produce extremely coherent, relevant, and contextually appropriate responses. It would be natural for users to infer that the AI has genuinely understood their meaning, intentions, or situation, especially when it has followed standard conversational norms. Overestimating and over-trusting what the AI knows or understands can have significant consequences in certain situations.
Linguistically compliant but not intent compliant
AI being obedient, taking the user input literally does not necessarily mean it has helped or satisfied the user.
A response can be grammatically correct, relevant, factually accurate and compliant with the explicit instructions given. But it can still be wrong as it is not relevant to the user situation. When the user objective is lost because the AI has obediently taken instructions literally, the result can be a huge conversational failure.
Conversational benchmarks
Accuracy benchmarks used by many organisations miss a critical aspect of successful conversational AI: conversational success. Take two systems. Each could produce the same correct answer. However, how they got to the answer is not the same. One gets there in four steps, the other in ten. These are very different user experiences. One is a conversational failure the other a conversational pass.
Measuring AI success solely for accuracy would give both experiences the same score. If a human rated the experiences, they would likely give very different ratings to each one.
Measuring interactional quality
Evaluating only answer correctness misses the interactional elements of a conversation. It is the interactional elements that influence successful ongoing user adoption of AI.
Beyond accuracy, organisations must be measuring whether the conversation was coherent, recoverable, and aligned to the user’s intent.
Of course, conversational AI solutions must be optimised for accuracy and safety. Equally important is that they are optimised for continual interactional quality. Traditional AI objectives must be partnered with new conversational objectives based around understanding.
If conversational quality is not part and parcel of the conversational design, users will simply stop interacting, no matter how accurate and safe.
