How to analyze the quality of training a chatbot?Some important metrics to evaluate the training of a chatbot.
Daniela SéBlockedUnblockFollowFollowingJul 1Currently, chatbots are in evidence.
We can say that most companies, which have websites, have some bot and even without knowing (as there are exceptionally well-trained and implemented bots), most people have already interacted with one.
Along with the bots, come a infinity of concepts, such as: entities, intents, dialog node, workspace, skill and a lot of them.
There is also a need to evaluate the training and performance of the service provided by the bots.
And there, dear reader, is that we have entered … the Super Data Scientists.
Giving a very superficial brushstroke, let’s align the following:AI Model: is composed of a set of intents and entities.
AI Provider: it is the tool that uses the AI model to classify new messages that bot users send (example of that engine is Watson).
If the “AI Model” is fine, the “AI Provider” will perform as expected during user dialogs with chatbot.
Similarly, it would be like evaluating the content that a teacher teaches (AI Model) to his students, thereby ensuring that, if the content is correct, the student (bot) is able to apply that knowledge on a day to day basis.
One of the points of attention, recommended by IBM, is the non-existence of repeated expressions in training, whether in relation to intentions or in relation to entities.
In this post, I show how to evaluate the existence or not of these repetitions.
So come on, let’s get going…1- INTENTSIntent mean what a user expects the bot to support.
Example of intent used in bot training.
Structure:In the bot workspace there are several objects, one of them is the ‘intents’.
Inside ‘intents’: [] are all intentions used in bot training.
Each intention has the {‘intent’ and ‘examples’} objects.
Within the ‘intent’ object is the name of the intention and, within the ‘examples’: [] are all examples of expressions for that intention, and these examples are reported inside ‘examples’ objects: [{‘text’: ‘’},…,{‘text’: ‘’}].
It is the expressions within {‘text’: ‘’} that should not be repeated.
Structure of intents in the workspace — Watson Assistant.
Analysis:To check, if the bot was trained with repeated examples for intents, I created a script on the jupyter notebook, available on my github ❤Import the required libraries and create the objects with the credentials in the watson assistant;Create the “original_workspace” object with the workspace used in bot training;Make the loop for that will go through the examples of each of the intents, looking for repetitions;2- ENTITIESEntities mean complements of information, that is, parameters of the actions that the user wishes to execute.
Example of entities used in bot training.
Structure:In the bot workspace there are several objects, one of which is ‘entities’.
Inside the ‘entities’: [] object are all entities used to train the bot.
Each entity has {‘entity’ and ‘values’}.
Inside the object ‘entity’ is the name of the entity and, within the ‘values’: [] object are the values and synonyms for that entity.
Values are reported within ‘values’: [{‘value’: ‘’}] and synonyms are reported within ‘values’: [{‘synonyms’: [‘’]}].
Structure of entities in the workspace — Watson Assistant.
Analysis:Continuing the script I made available on my github.
In the case of entities, it is recommended that there be no repetitions of expressions between entity names, values, and synonyms.
Make the loop for that will cycle through entity names, values, and synonyms by looking for repeats:Thanks for reading.
Show Me The CODE!To use the code I shared in github, you must install python and jupyter notebook or use the platform created by IBM that allows you to perform analysis in the cloud.
To do this, simply instantiate your version of Data Science Experience.
Steps:Acess the DSX.
Create a new project.
On the Assets tab, click Create a notebook.
Access my code in github.
Copy and paste.
Inside the notebook, change the credentials according to the Assistant you want to analyze.
Done!.. More details