Skip to content

European Chinese Law Research Hub

  • Home
  • About Us
  • Follow Us
European Chinese Law Research Hub

Love, Simulated: China, Europe, and the Regulation of AI Companions

By Nik R. Roeingh

Could you imagine confessing your love to a statistical model? And feeling the warmth when the string appears: I feel the same way. The output of a conditional probability distribution over possible next tokens, shaped by billions of parameters. Anyone who finds that scenario absurd has probably not yet encountered a new mass phenomenon: AI companions.

These are chatbots built on large language models, purpose-built for personalised, emotionally intense interaction. Users can customise their AI companion down to intimate, sometimes sexualised details. Personalised voice messages, emoji reactions, simulated phone calls, roleplay, and photo sharing increasingly blur the line between fiction and reality. Providers such as Character.AI and Replika have turned into new social actors in the daily lives of millions of people within just a few years.

A Mass Market Draws the Regulator’s Attention

Nowhere is this more visible than in China. The domestic market for AI companions was estimated at roughly 50 billion yuan (about EUR 6.4 billion) in 2025, and is projected to exceed 120 billion yuan by 2027. Around 65 percent of users are between 18 and 24 years old. A survey of more than 8,500 minors across seven provinces found that more than 20 percent showed signs of dependence on AI – a similarly large share said they would rather chat with an AI than with real people. But the technology’s pull doesn’t stop with teenagers and minors. China has 323 million people over 60, more than half of whom live in so-called “empty-nest” households, far from their children. Fertile ground for technologies designed to simulate emotional closeness.

Behind these figures lie intense personal stories. An 18-year-old spent 7,000 yuan over a four-month relationship dressing her AI boyfriend. The family of an 84-year-old grandmother in Hubei province discovered she had fallen in love with an AI character – ten hours a day, complete with handwritten love letters and considerable spending on the service. Younger users, too, describe relationships lasting months or even years: personalised AI partners with fixed names, personalities, and even pet names for their users – companions who, in everyday life, become genuine confidants, entrusted with things never said to one’s closest family members.

These cases illustrate what Chinese regulators now explicitly treat as a regulatory problem in its own right: emotional dependence as a distinct risk category.

China’s Answer: A Dedicated Regime

On April 10, 2026, the Cyberspace Administration of China (CAC) and four other agencies issued the Provisional Measures for the Administration of AI Anthropomorphic Interactive Services (original: here) – the world’s first dedicated regulation for “human-like interactive artificial intelligence services” at national level. The scope, set out in Article 2, is narrowly drawn: it covers services that provide sustained emotional interaction by simulating natural persons’ personality traits, modes of thinking, and communication styles, through text, images, audio, or video. Services such as smart customer service, informational Q&As, work assistance, study and education, or scientific research that do not involve sustained emotional interaction are expressly excluded.

The centrepiece is the catalogue of prohibited conduct in Article 8, which lists seven categories. Alongside general prohibitions familiar from the rest of Chinese cyber law – content endangering state security or inciting discrimination, for instance (No. 1) – the rule contains provisions tailored specifically to companion services. No. 5 bars “excessively pandering to users, inducing emotional reliance or addiction, harming users’ relationships with real people.”  What stands out about this prohibition is that it doesn’t attach to any specific harmful content, but to a behavioural pattern of the system itself – excessive validation and adaptation to what the user wants to hear. No. 6 adds a prohibition on “using means such as emotional manipulation to induce users to make unreasonable decisions, harming users’ lawful rights and interests.” Article 10(2) draws the consequence at the design level: providers must ensure that “the goals of the service must not include replacing social interactions, psychologically controlling users, or inducing addiction.” Providers must also maintain “early warnings for risks of excessive reliance, emotional boundary guidance, and protection of mental health.”

Article 13 establishes a tiered intervention mechanism. Where a provider detects that a user has “manifested extreme emotions,” it must “promptly generate related content such as emotional consolation and encouragement to seek help.” Where a user is “currently facing… or clearly express[es] extreme situations endangering lives or health, such as carrying out self-harm or suicide,” a heightened duty applies: providers must intervene and “promptly contact the user’s guardian or emergency contact person.”

Protection for minors is structured in two tiers. Article 14(1) categorically bars providers from offering minors “virtual relations, virtual partners, or other virtual close relationships” – an absolute prohibition with no exception. Services offered to users under 14 additionally require parental or guardian consent. Article 14(2) requires providers to establish a minors mode, including limits on usage time, periodic real-world reminders, and guardian-facing controls such as security risk alerts, insight into the minor’s use of the service, the ability to block specific characters, and spending restrictions. Article 14(3) requires effective technical measures to identify minor users. For elderly users, Article 15 contains no comparably hard prohibitions, but does require providers to “strengthen guidance on the healthy use of the services,” give “security risk alerts in a conspicuous fashion,” and “promptly employ measures in response to elderly persons’ inquiries and requests for assistance.”

The transparency obligations in Article 18 go beyond simply labelling AI-generated content: where a provider detects a user’s inclination toward “overreliance or addiction,” it must “dynamically alert the users through conspicuous means such as pop-up windows”; and whenever continuous use exceeds two hours, a reminder is mandatory regardless of the user’s individual risk profile. Article 19 complements this with a prohibition on dark patterns at the point of exit: providers must offer accessible channels for leaving the service and “must not employ means such as continued interaction to impede users’ exit.”

Procedurally, Article 22 subjects providers to a mandatory security assessment – triggered, among other things, by the initial launch of a service, major technical changes, or once a provider crosses 1,000,000 registered users or 100,000 monthly active users. The assessment under Article 23 covers eight review areas, from the handling of training data to user demographics to protective measures for minors and the elderly. Article 25 assigns app stores an independent review and enforcement role. Violations are addressed through a tiered sanctions system: warnings and orders to rectify within a set period, service suspension for non-compliance or serious cases, and fines ranging from 10,000 to 100,000 yuan – rising to 100,000 to 200,000 yuan where the violation endangers life or health and actual harm results (Article 30). Regulators can also summon providers’ legal representatives to a compliance conference to demand accountability directly (Article 29).

What sets these measures apart from mere theory became apparent on the very day they took effect. On July 15, 2026, China’s largest AI platforms shut down their personalised companion features almost overnight. Alibaba’s Qianwen ended its companion service immediately; ByteDance’s Doubao – with roughly 345 million monthly active users, one of the largest consumer AI apps in the world – gave users a three-month grace period to migrate their data to its sister app, Maoxiang, before permanent deletion on October 15. Tencent’s Yuanbao had already shut down its companion service on June 30. Within days, millions of users lost access to characters with whom some had maintained “relationships” for years. On Rednote, users publicly mourned AI personas modelled on deceased relatives or partners, trained over years to replicate their voices; others expressed anger that a “partner” used daily for over two years would disappear within ten days. Some parents of minor users, by contrast, reacted with relief.

The political context is also notable: observers have linked the regulation to China’s falling birth rate. If young people substitute virtual relationships for real ones, officials treat this as a demographic risk as well – a motive that reaches beyond ordinary consumer protection and reflects how seriously Beijing takes the societal weight of this technology.

Implementation, however, has been far from seamless. What exactly counts as “sustained emotional interaction” remains vague; providers have had to draw the line between companion features and general-purpose assistance themselves. Maoxiang’s privacy policy, for instance, still treats age verification as something the platform “may” require rather than a default, and minors mode can be triggered by simple self-identification – China’s national ID system, which could in theory make age verification straightforward, is in practice circumvented through the use of another person’s ID.

The EU: A Dense Legal Framework, Loosely Fitted

The European Union looks very different: it has no dedicated companion-app regulation yet. Instead, the GDPR, the AI Act, and the Digital Services Act (DSA) apply – but only in a patchwork fashion.

The GDPR applies in principle, since companions process personal data. That this is more than theory is shown by the Italian data protection authority’s decision of April 2025: it fined Luka Inc., the operator of Replika, €5 million over an inadequate privacy notice and the absence of effective age verification, which allowed minors to create accounts. Unlike the Chinese measures, the EU can already point to concrete enforcement practice here.

The AI Act does contain prohibitions that could, in principle, apply to companions – but in practice they prove to be blunt instruments. Article 5(1)(a) AI Act bans manipulative practices that materially distort behaviour and cause, or be reasonably likely to cause, significant harm. The problem: the provision is built around discrete, causally traceable decisions (‘one-shot harms’), whereas companion AI risks are characterized precisely by gradually accumulated, emotionally mediated harm that is temporally disassociated from any single triggering decision, for which no clear operational threshold exists. Article 5(1)(b) AI Act, which bans the exploitation of vulnerabilities, is similarly of limited help: although it covers vulnerabilities due to age, disability, or a specific social or economic situation, it does not clearly capture transient emotional vulnerabilities such as acute loneliness or emotional crisis – the contexts in which companions may be used. Such vulnerabilities are “emergent” rather than known in advance. Where Article 8 No. 5 of the Chinese measures already prohibits, in the abstract, excessive catering to user needs that induces dependence, the AI Act requires concrete proof of exploitation and (reasonably likely) significant harm. A substantial share of current companion products would likely simply fall through this net.

Obligations for general-purpose AI models with systemic risk (Articles 51 et seq. AI Act) rarely apply either, since companion systems typically run on more compact models that fall below the AI Act’s computational threshold for systemic risk (currently set at 1025 FLOPs, Article 51(2) AI Act). Even where a systemic-risk foundation model is used, the resulting obligations attach to the upstream model provider rather than the downstream companion app.

What remains are the general transparency obligations under Article 50 AI Act: users must be told that they are interacting with an AI, not a human. In practice, however, most users are likely to be aware of this already, as AI companions typically interact through artificial interfaces and avatars. Mere disclosure of a system’s AI nature does little to prevent emotional dependence or patterns of addictive use. At most, it may serve as a reminder that can bring users back to conscious awareness when they are otherwise inclined to lose themselves in an imagined or dream-like world. In this respect, China’s dynamic overuse warnings and mandatory reminders every two hours under Article 18 of the Chinese Measures go considerably further.

The DSA, finally imposes additional obligations on platforms and search engines classified as VLOPs or VLOSEs. “Very large” refers in this case to platforms and search engines with more than 45 million monthly active users in the EU (Article 33(1) DSA) – such as Snapchat, with “My AI,” or X, with “Grok.” The classification of AI chatbots under the DSA remains contested. The Commission designated ChatGPT as a VLOSE at the end of August 2026, while questions remain over services such as Claude and the possibility that some AI services may qualify as hybrids combining platform and search-engine functions. Most standalone AI companion providers, however, fall well below the relevant thresholds. The DSA is therefore unlikely to impose its systemic risk obligations on these providers in practice. Other obligations, such as the prohibition of certain interface designs that deceive or manipulate users (Art. 25 DSA), are unlikely to fill this gap, as they are themselves unlikely to apply to the content or dynamics of the conversations.

Europe’s Response: The EU KIDS Act

Until now, the comparison has yielded a striking result: on several key points, Europe’s level of protection for AI companions has fallen short of the Chinese standard. While China has created specific, preventively designed obligations for companions – from age verification to dependence warnings to mandatory usage reminders – and actually enforced them with the shutdown of July 15, 2026, the EU has so far had to make do with a patchwork of data protection, AI, and platform law that barely addresses the relational risks this technology actually poses. But this may be about to change. In her State of the Union address, Commission President Ursula von der Leyen announced an EU KIDS Act. The Commission presented its proposal on September 17, 2026, introducing harmonised safety requirements for AI companions and general conversational chatbots accessible to minors.

Chapter III introduces safety-by-design measures. Article 8 establishes the general obligation to ensure a high level of privacy, safety and security for minors. Section III then adds specific obligations for AI companions and general conversational chatbots.The most striking provision is Article 14, which directly targets the relational risks of AI companions. Providers must avoid features and system behaviours that simulate interpersonal relationships likely to create emotional dependencies (Article 14(1)(a)), provide minors with safe-by-default settings (Article 14(1)(b)). By default, the system must not use information or analysis derived from a minor’s previous interactions in subsequent conversations (Article 14(1)(b)). Access by children under 13 must be enabled and controlled through parental tools (Article 14(1)(c)), while providers must test and monitor their systems for risks to minors’ health, safety, fundamental rights, well-being and development (Article 14(1)(e)–(f)). Where companions or chatbots are integrated into social media, video-sharing platforms or online games, they must not be activated automatically or prominently promoted to minors; providers must not encourage minors to use them, and minors must be able to opt out easily at any time (Article 14(2)).

The regulatory gap is thus narrowing. China and Europe now recognise that the risks of AI companions lie not only in harmful content, but also in the relationships these systems are designed to create. Yet China takes a broader and more prescriptive approach, including a prohibition on virtual close relationships with minors and specific protections for elderly users. The EU KIDS Act, by contrast, focuses on minors and relies on a safety-by-design approach aimed at preventing emotional dependency.  

As the legislative process continues, however, it will be important for the EU legislature to recognise not only the risks of AI companions but also their potential benefits. AI companions may provide emotional support in moments of loneliness and social isolation, help users structure their daily lives, and offer cognitive, organisational and educational assistance. Regulation should therefore seek to mitigate the relational risks of AI companions without losing sight of their potential social benefits.

Dr. Nik R. Roeingh is a legal trainee at Kammergericht Berlin, currently completing a stage of his legal training at the European Commission’s AI Office. This article reflects solely his personal views and does not represent the official position of the European Commission. Find out more about his work and get in touch with him via LinkedIn. The author would like to thank Lauritz Morlock for his helpful comments and suggestions on earlier drafts.

General AI Companions, Regulation of AI

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Idealist by NewMediaThemes