From pilot to operationalization
What happens when conversational AI projects move from pilot to operationalization? A new type or operational discipline is required. Organizations need to make sure they are structured and staffed in a way to manage AI conversations at scale.
To do this effectively requires a new operational discipline to be active within a company. The need for Conversation Operations (Conversation Ops), is one of many new disciplines that have been identified as a new requirement arising from the ever increasing use of AI across organizations.
Different from AI Ops which is concerned with “keeping AI systems running”, Conversation Ops is all about “keeping the AI interaction working”.
People to AI interactions
The communication between people and enterprise AI is critically important because it impacts a company financially, reputationally and operationally. This is why Conversation Ops is now a real need with the function having responsibility for designing, monitoring and continuously improving enterprise AI to people interactions.
Organizations must be capable of ensuring conversations between AI and people are consistently producing the right outcome. The quality of a conversation is not confined to if the answer was correct, it is multidimensional, which is more complex than a model simply being able to answer a question.
For example, technically correct answers may have misunderstood the user’s goal, or a RAG system could retrieve wrong answers, it may be that when a conversation fails there is no useful handoff to a human, or a model update could change successful conversation patterns.
The multiple factors that must be taken into account to ensure quality conversations, need to be brought together to ensure consistency of quality at scale.
Enterprise AI
As AI becomes how work gets done across a company, organizations that know how to operate conversations will be the best equipped to deliver the right outcomes and ensure the user accomplishes what they want to do. This requires organizations to view conversations as operational assets.
The introduction of AI into all areas of an organization has introduced a new operational layer – Conversation Ops – to the current software, data, security, customer experience and infrastructure operations. Organizations need to be operationally AI literate, connecting multiple departments and functions across the business to ensure the right conversation to deliver the right outcomes and ensure a positive human experience.
Conversation Ops responsibilities
Conversation Ops should not be seen as a singular department. Its prime responsibility is owner of the quality and performance of AI interactions at scale and therefore it must span multiple areas – product, customer experience, knowledge management, governance, AI engineering and AI ops.
From quality, evaluation and analytics, to design, escalation, knowledge, through to governance and improvement, Conversation Ops is continuously tracking outcome metrics (e.g. task completion, resolution, and conversion rates etc.), conversation metrics (e.g. repetition, abandonment and escalation rates etc.), quality metrics (e.g. factual accuracy, relevance, policy compliance, tone/brand adherence etc.), and operational metrics (e.g. failure and regression rates, performance by use case/persona/channel etc.).
With responsibility for ensuring that every interaction is useful, accurate and appropriate, as well as being compliant, auditable and safe, Conversation Ops is making sure that AI is delivering so that the user is able to accomplish their objective in the most effortless way possible.
Conversation Ops – the future
The advancement of conversational AI is not just about having better models. It will be about knowing how to operate conversations. Conversation quality is different to model quality, as a great model can still produce poor enterprise conversations. If the prompt is bad, knowledge is bad, workflow is bad, escalation is bad and context is bad, then the experience will be bad.
With Conversation Ops owning quality and performance across each of these, and more, then quality conversations will be assured. Conversations need to be seen as operational assets, and Conversational Ops an operational discipline that connects the technical operations of AI to the human experience of interacting with it. It’s important to remember that AI output is both a conversation and an outcome.
