Bot Metrics

These are bot metrics I found Googling. It will be updated as I find more.

While bot metrics can include KPIs like daily active users, being in the early days of bots, it would be better to focus on how well bots work in fulfilling the customers’ experience. Most other metrics related to funnels and sales should still be applicable.

From “Towards a Method For Evaluating Naturalness in Conversational Dialog Systems”

Metric Type Data Collection Method
Total elapsed time Efficiency Quantitative Analysis
Total number of user/system turns Efficiency Quantitative Analysis
Total number of system turns Efficiency Quantitative Analysis
Total number of turns per task Efficiency Quantitative Analysis
Total elapsed time per turn Efficiency Quantitative Analysis
Number of re-prompts Qualitative Quantitative Analysis
Number of user barge-ins Qualitative Quantitative Analysis
Number of inappropriate system responses Qualitative Quantitative Analysis
Concept Accuracy Qualitative Quantitative Analysis
Turn correction ratio Qualitative Quantitative Analysis
Ease of usage Qualitative Questionnaire
Clarity Qualitative Questionnaire
Naturalness Qualitative Questionnaire
Friendliness Qualitative Questionnaire
Robustness regarding misunderstandings Qualitative Questionnaire
Willingness to use system again Qualitative Questionnaire

From “Different measurements metrics to evaluate a chatbot system”:

  • Dialogue efficiency in terms of matching type.
  • Dialogue quality metrics based on response type.
  • Users’ satisfaction assessment based on an open-ended request for feedback.