What should AI companies prove before giving conversational agents to children?
A new World Economic Forum white paper sets out child-focused practices spanning design, privacy, learning, harmful interactions and accountability. It is useful guidance, but not a binding standard or an outcome study, and its public summary reports no sample or systematic-review method.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
Research topic
What evidence and incentives are needed to make generative, conversational and agentic AI safer and more beneficial for children
At a glance
- 1The 30 September white paper covers age-appropriate design, beneficial use, harmful interactions, privacy, learning, accountability and inclusion across the AI life cycle.
- 2It also identifies regulation, procurement, assurance, investment, shared infrastructure and organisational accountability as levers that could make child-safety practices more likely to be implemented.
- 3The report is guidance, not a law or controlled evaluation. Its public summary provides no child sample, intervention denominator, systematic search protocol or measured safety outcome.
Living evidence record
Impact record IAI-1HNH8MY
Evidence stage
Observed
Confidence
Supported
Reporting basis
Source analysis
Independent support
Not yet
Record status
Monitoring
Last checked
30 September 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what World Economic Forum published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
The report treats child safety as a product-lifecycle duty
A World Economic Forum white paper published on 30 September argues that children's safety and wellbeing must be considered throughout the AI life cycle, not added after a conversational product has reached families and schools. The report groups its proposed practices around age-appropriate design, beneficial use, harmful interactions, privacy, learning, accountability and inclusion. It covers generative, conversational and agentic systems because memory, personalisation, persuasion and autonomous actions can change both the usefulness and the risk of an interaction.
That framing is more practical than a generic promise to make AI 'safe'. Developers decide what data a system remembers, how it responds to distress, whether it can contact other services, which commercial prompts it displays and when a human is brought in. Deployers decide where the system appears and what claims are made about it. Schools, parents and children cannot compensate for every hidden design choice. The paper therefore places duties across organisations rather than treating safety as a matter of individual self-control.[1]
Age-appropriate design is wider than an age gate
Age assurance may help a service apply different protections, but it can also create privacy risks if platforms collect identity documents, biometric estimates or more behavioural data. A child-centred design approach must ask what the product can do after a user enters: whether persuasive features are limited, sensitive memories expire, advertising is separated from advice, default settings minimise disclosure and high-risk actions require adult or institutional oversight.
The report's emphasis on inclusion is important because one age check will not work equally for every child. Some young people lack standard identity documents, share devices or use assistive technology. Language, disability and cultural context can change whether warnings are understood and whether a complaint route is usable. A protection that excludes vulnerable children or pushes them towards unregulated services may create a different harm rather than solving the first one.[1]
Benefits should be demonstrated, not assumed from engagement
AI tools can support learning, creativity and accessibility, but time spent with a product is not evidence of educational benefit or wellbeing. A credible evaluation should define the intended benefit, compare it with an appropriate alternative and measure whether gains persist. For a tutoring system, that could mean independent assessment of learning rather than session length. For an assistive tool, it could mean whether a child completes a task with more autonomy without surrendering unnecessary data.
The same principle applies to AI companions. A fluent response may feel supportive while providing inaccurate information, encouraging dependency or mishandling a disclosure of abuse or self-harm. Companies need pre-release tests that reflect children's actual language and situations, clear escalation rules and monitoring for harms after launch. Serious incidents should be reportable and auditable without forcing families to prove a product's internal behaviour from screenshots alone.[1]
Incentives determine whether practices survive commercial pressure
The WEF paper looks beyond voluntary checklists to the incentives created by regulation, procurement, assurance, investment, shared infrastructure and organisational accountability. Procurement is especially concrete: a school or public agency can require a documented risk assessment, data-retention limits, independent testing, incident reporting and an exit plan before buying a service. Investors and boards can ask whether safety teams have authority, resources and access to product decisions rather than existing only as advisers.
Assurance must be specific about scope. A one-time audit of a model does not automatically cover the surrounding app, its memory, advertising, tool access or later updates. Shared testing infrastructure could help smaller organisations evaluate risks without recreating every benchmark, but common tools should not become a ceiling. Products used by children still need local testing, accessible redress and clear responsibility when a supplier, school and platform each control part of the experience.[1]
What the paper establishes—and what would change our assessment
This is a policy white paper, not a clinical or educational trial. The public publication page does not provide a participant sample, child denominator, systematic evidence-search method or measured reduction in harm. Its value is to organise practices and institutional levers; it cannot show that companies already follow them or that any one practice improves outcomes. The recommendations are not binding law, and enforcement will depend on the countries and institutions that adopt them.
Confidence would rise if companies publish product-level child-risk assessments, independent red-team results, incident rates, subgroup testing and evidence that protective changes improve outcomes without excluding children. Regulators and procurers should state measurable requirements and preserve access for independent researchers. The assessment would weaken if the white paper becomes a badge used without public evidence, if age checks expand surveillance, or if organisations cite engagement as proof of benefit. The test is whether children are safer and more capable in practice, not whether a company can point to a principles document.[1]
What this means for people
- Children could gain more useful learning and accessibility tools if benefits are tested and persuasive, data-intensive or autonomous features are constrained by default.
- Parents and schools need clear evidence, complaint routes and supplier accountability rather than being expected to audit opaque systems themselves.
Global context
The report is international, but child rights, education systems, privacy rules and age-assurance requirements differ widely. Wealthier institutions may be able to buy independent audits and safer enterprise products while families and schools elsewhere rely on free consumer systems. Global guidance is useful only if protections can be implemented across languages, devices and resource levels without excluding children or normalising invasive identity checks.
What the evidence does not yet show
- The source is a WEF policy white paper rather than a peer-reviewed experiment, law or binding technical standard.
- The public summary does not report a participant sample, child denominator, systematic-review protocol or measured effect on safety, learning or wellbeing.
- Implementation and enforcement will differ across countries, schools, families, platforms and regulatory systems.
What to watch next
- Product-level child-risk assessments, independent testing and public incident reporting from AI companies.
- Procurement rules that specify memory, data-retention, advertising, escalation and audit requirements for products used by children.
- Evaluations that include children across ages, languages, disabilities and living circumstances while protecting their privacy.
Evidence trail
Sources used for this report
Links checked 30 September 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Society & Media
UNICEF frames AI companions as a child-rights issue
UNICEF's report When AI Becomes a Friend examines how companion systems can affect privacy, development, safety, participation and children's relationships with people and institutions.
4 min · 1 source
Society & Media
Former Google safety chief warns AI could amplify harms seen in social media
Reuters reports testimony from a former Google safety executive who argued that companion-like AI and persuasive systems could expose children to harms more directly than conventional social feeds.
4 min · 1 source
Society & Media
Council of Europe weighs generative AI's democratic benefits and risks
A Council of Europe study examines how generative AI may widen participation and access to information while also enabling manipulation, unequal influence and pressure on public debate.
4 min · 1 source
Reader discussion
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.