Italian Regulator Bans AI Chatbot Replika Over Data Concerns

Italy’s data protection regulator has banned the firm behind AI chatbot Replika from gathering data within the country.

The action by the Garante appears to be the first official sanction against Luka, the San Francisco-based company whose service is based on technology from Microsoft-backed OpenAI.

Replika is based on OpenAI’s GPT-3 language model, the same technology that powers OpenAI’s popular ChatGPT chatbot, but Luka has tailored it for the purpose of providing users with virtual “friends” intended to make them feel better.

For many users that means a simulated romantic relationship, with the company’s paid subscription service offering features such as an increased level of intelligence, customiseable avatars, sexting, voice calls and augmented reality (AR).

Inappropriate interactions

Late last year some users began complaining that even the free version of Replika was offering unwanted sexually charged comments, and the Garante said it was concerned that this could expose underage users to harm.

The risks include “first and foremost, the fact that they are served replies which are absolutely inappropriate to their age”, the regulator said in a statement.

It said Luka lacks a proper legal basis for processing children’s data under the EU’s GDPR data regulations, and that it fails to carry out age checks on users.

“A provisional limitation on data processing was imposed by the Italian Garante on the US-based company that has developed and operates the app; the limitation will take effect immediately,” the Garante stated.

Age verification

“There is actually no age verification mechanism in place: no gating mechanism for children, no blocking of the app if a user declares that they are underage,” the regulator said. “During account creation, the platform merely requests a user’s name, email account and gender.”

It said Luka had 20 days to communicate measures it had taken to comply with the order.

The firm faces a fine of up to 20 million euros (£18m), or 4 percent of its total worldwide annual turnover, if it fails to comply.

Luka was an early API partner for GPT-3, which sources its interactions from material found across the internet.

OpenAI, whose own ChatGPT bot became the fastest-growing consumer app in history after its public launch late last year, is preparing to publish an even more convincing version, GPT-4.

The tool’s high profile has sparked a debate over the implications of so-called generative AI models, especially when their responses become impossible to distinguish from those of a human.