You have done it! You have identified, built, and deployed your new Natural Language Processing (NLP) solution. The work stops here, right? Unfortunately, the answer is no, and it’s a good thing. Customer care is an ongoing process – making slight adjustments is easier & cheaper than waiting it out and starting a complex & expensive fix. That is why monitoring is so crucial. Besides ensuring that the solution works as expected, you also want to see its impact on how your customers interact with you. In this blog, we will touch on important concepts related to monitoring a customer care solution, powered by NLP.
Monitor the progress using these guidelines:
1. Any feedback is good feedback
If you ever wondered what every famous machine learning solution has in common, it is the ability to gather feedback in an affordable way. Search engines can improve because they gather what you search for, and which result you clicked on. Social media platforms can recommend content accurately because they are able to collect information whether you liked, shared, or watched something. They are able to do so because they designed their solution with monitoring in mind.
When you build chatbots, it is very useful to add confirmation dialogs to gather feedback on whether the user is satisfied with the bot’s understanding. If you want to improve your FAQ, it is valuable to see which links are clicked on most. In order to provide tools to help your call center agents, gathering feedback on the length of the conversation or how long it took before resolving an issue is crucial.
2. Pick KPIs based on the problem, not the solution
With machine learning technologies such as NLP, many fall into the trap of evaluating the quality of their solution with machine learning metrics. How well does the model perform on a test set, what kind of mistakes it makes, how long does it take to make a prediction, etc. While these metrics are important and will focus the efforts of your Data Scientists, they are not the most valuable metrics. Your customer does not care whether your solution is powered by a deep neural network, hard-coded „if statements“, or if there is a human performing the task, and neither should you.
A chatbot should not be judged by the accuracy of its intent recognition model. Instead, it should be judged by how many conversations lead to a solution, how many steps it took to get there, how many solutions were cut short because a customer was frustrated, and so on.
3. Do not overfocus on a single KPI
Using KPIs is crucial as it creates an objective truth that can then be used to evaluate whether a situation has improved or not. While it is very convenient to focus on a single number, there is no single number that can accurately tell a whole story.
Customer care is complex, and whether you are doing “better” does not depend on a single factor. Having a very short resolution time for calls could indicate that your agents are highly efficient, or that people are calling about issues so simple, they should not have to call in the first place. Your NPS score might decrease which could indicate that you are doing worse than before. But it could also mean that you solved all simple issues, and you are left with queries that are more difficult to solve and might result in frustrated customers.
4. Be transparent
Gathering feedback, KPIs, and all sorts of metrics is only useful if they are made available in a transparent manner. Building dashboards that can get accessed by many and be updated frequently, if not in real-time, is one of the most underrated aspects of a successful NLP project. Machine learning and data science can do many things, but it is extremely difficult to beat the insights and conclusions that humans can make when provided with the correct information.
Conclusions
Customer care is a fast-paced world, as a bad experience can quickly lead to a PR nightmare. It is therefore important to be on top of things and be aware of where your flaws are. Monitoring your solution may be the most important aspect discussed in this series of blogs. It is also where the loop closes back. With good monitoring in place, you will notice if new issues start popping up or if your customers start to choose different channels to communicate with you.
Happen to miss the other relevant NLP blogs? Don’t worry! You can check them out here:
This blog is written by Valentin Calomme – AI Engineer & Business Line Manager at Mediaan Conclusion.
