Moveo.AI announced that after rigorous comparison, its custom LLM tuned for CX outperforms GPT-4-0613 in all grading dimensions, except Markdown, where GPT-4 performs better. The evaluation was based on a random sample of hundreds of entries from Moveo’s production data, which neither our LLM nor GPT-4 had encountered before. Each entry was converted into a prompt consisting of the user question, conversation history, grounding knowledge from the collection documents, live instructions, and custom instructions.
This press release features multimedia. View the full release here: https://www.businesswire.com/news/home/20240723855013/en/
As can be clearly seen in this table, Moveo’s custom LLM outperformed GPT-4 in four critical dimensions that are the cornerstone of a great Customer Experience: Hallucination, Repetitions, Disambiguation, and Readability. The two models are equal in Language while GPT-4 performs better only in Markdown use. (Graphic: Business Wire)
Methodology
The grading process assessed Moveo’s LLM and GPT-4 responses across 8 dimensions that capture critical traits within the CX setting:
- Hallucination
- Repetition
- Disambiguation
- Live agent handover
- Readability
- Language
- Markdown, and
- Latency
Each dimension received a score, determining which LLM provided a better response. To evaluate the performance of the different models, Moveo used a separate GPT-4 instance as a “grader,” performing a single API call for each of the samples.
Results
Moveo’s custom LLM outperforms GPT-4-0613 in all grading dimensions, except in Markdown, where GPT-4 performs better in stylistic formatting. Most importantly, it is worth mentioning that in terms of hallucination, GPT-4 performs worse, which could hurt Customer Experience. For example, if GPT-4 provides incorrect information about a product, it could lead to potential liabilities, customer dissatisfaction, and increased support requests.
Moveo’s LLM responds in only 5 seconds, while GPT-4 takes at least 18 seconds. In that time, Moveo.AI could have handled more than 4 inquiries, significantly enhancing support efficiency and customer satisfaction.
According to Panos Karagiannnis, CEO of Moveo.AI, “Enterprises need vertical-specific LLMs as every customer interaction is an opportunity to build trust and loyalty. By minimizing hallucinations and connecting to real-time information systems, our LLM significantly beats GPT-4, reduces the risk of customer dissatisfaction and potential liabilities, and sets a new standard in CX”.
To learn more about Moveo’s proprietary LLMs, please visit: https://moveo.ai/
About Moveo.AI
Moveo.AI is a Conversational AI platform transforming how enterprises interact with customers. Moveo’s LLM, trained on historical and real-time CX data, powers GenAI agents to seamlessly connect to real-time data and unstructured knowledge bases to provide accurate and contextually relevant answers to inquiries.
View source version on businesswire.com: https://www.businesswire.com/news/home/20240723855013/en/