Baidu Ranks Top In Chinese ChatGPT-Style Chatbot Tests

Matthew Broersma,
Linkedin
chatgpt, smartphone, ai, artificial intelligence, chatbot

Tests find Baidu’s Ernie Bot ranks above Alibaba’s Tongyi Qianwen but behind OpenAI’s ChatGPT as competition heats up

Baidu’s Ernie Bot ranked highest in Xinhua Institute tests of chatbots that have been made available in recent months by Chinese tech firms seeking to compete with ChatGPT.

Xinhua Institute, a think tank affiliated with the Xinhua news agency, said its tests found Ernie Bot led Alibaba’s Tongyi Qianwen, SparkDesk from voice recognition firm iFlytek, and SenseChat from image recognition company SenseTime.

But the Baidu chatbot still trailed Microsoft-backed OpenAI’s latest GPT-4 and GPT-3.5, which powered the version of ChatGPT released to the public last November.

The researchers tested general capabilities such as basic language skills, logical reasoning and subject knowledge in fields such as mathematics, physics, finance and literature.

Baidu chief executive Robin Li speaks at the World Artificial Intelligence Conference in July 2021. Image credit: Baidu
Baidu chief executive Robin Li speaks at the World Artificial Intelligence Conference in July 2021. Image credit: Baidu

Competition

They also tested the bots’ ability to improve productivity in fields such as journalism, painting, design, marketing, law and research.

ChatGPT captured the public’s imagination on its release last year, quickly gaining more than 100 million users, and competitors including Google and Amazon quickly announced their own versions of the technology.

In China, where ChatGPT is not officially available, Ernie Bot became the first OpenAI competitor last March.

But Baidu suffered an backlash as investors reacted to what they felt was a disappointing prerecorded demonstration of the chatbot’s capabilities.

‘Subjectivity’

Baidu’s shares have experienced a surge since the end of last month, when the company announced it would soon launch a new version of the large language model (LLM) powering Ernie Bot.

But they are still trading about 20 percent below their peak in early February, when investors were hotly awaiting the company’s initial chatbot announcement.

A different test of chatbots by Clue, a Chinese website that tracks AI research, found Smart Brain from cybersecurity firm Qihoo 360 was the best performer, followed by SparkDesk.

Xinhua Institute acknowledged in its report that tests were subject to time and conditional constraints that may result in “a certain degree of subjectivity”.