Did AI-assisted teaching improve computational thinking?
A 14-week pilot at one Chinese medical college found larger gains in an AI-assisted class than in a conventional class. But one intact class received each condition, so the 120-student study cannot separate the teaching package from class, teacher, time-on-task or other group differences.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1Two intact university classes totalling 120 students completed a 14-week comparison; the AI-assisted class scored higher on the final composite measure, with Cohen's d of 0.97.
- 2The intervention combined intelligent resources, adaptive sequencing, multimodal interaction and a fuzzy multi-dimensional evaluation system rather than testing one tool in isolation.
- 3Because class and treatment were perfectly confounded, the result is a promising two-class pilot, not a generalisable causal estimate of AI's effect on learning.
Research topic
Whether an integrated AI-assisted teaching package improves computational-thinking outcomes beyond conventional university instruction
The answer: scores improved, but the design cannot isolate AI
Students in the AI-assisted class made larger measured gains in computational thinking than students in the comparison class during a 14-week pilot at Puyang Medical College in Henan, China. The final composite difference produced t(118)=5.36, p<0.001 and Cohen's d=0.97. The largest reported differences were in abstraction, with d=1.03, and evaluation with generalisation, with d=0.98. An analysis adjusting for pre-test scores still found a group difference.
Those are sizeable effects, but they do not establish that the adaptive engine caused them. The study assigned one intact class to the full AI-assisted package and one intact class to conventional instruction. Class membership and treatment were therefore the same variable. Any difference in teachers, peer dynamics, timetable, motivation, resource exposure or time spent online could move the result, and no statistical adjustment can remove a factor that exists only at class level when there is one class per condition.
The most accurate interpretation is feasibility: the researchers implemented a coherent four-layer programme for a full term, students completed it, and the intervention class improved on the chosen measure. The study generates a testable model for larger trials. It does not justify a claim that AI-assisted teaching will produce the same improvement in other institutions, subjects or student populations.[1]
The Impact Brief · Free
Follow the evidence in education.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
What the four-layer package actually included
The first layer managed learning resources. Materials were organised so students could receive content suited to the topic and their current position in the course. The second layer used deep knowledge tracing—a long short-term memory model intended to estimate changing mastery from prior interactions—to recommend an adaptive learning path. The third added multimodal interaction support, while the fourth evaluated computational thinking across several dimensions and competency levels using a fuzzy comprehensive evaluation framework.
This matters because the intervention was not simply access to a chatbot. It combined new resources, sequencing, feedback, interaction and assessment. A higher score could reflect the package as a whole, increased structure, additional practice or closer monitoring. It cannot be attributed cleanly to generative AI, deep knowledge tracing or any one interface. Schools considering adoption should treat the programme as a complex instructional redesign rather than a software switch.
The outcome framework covered decomposition, abstraction, algorithmic thinking, debugging and evaluation with generalisation across three competency tiers. Mid-course checks at weeks five and nine used shortened parallel forms. The authors explicitly note that those abbreviated forms were not formally equated to the full assessment, so the interim trajectory should be read descriptively rather than as a precise growth curve.[1]
The 120-student comparison and its denominator
The quasi-experiment included 120 students in two existing classes at a single medical college. Each class received one condition in its entirety. The paper reports pre- and post-intervention computational-thinking measures and the 14-week delivery period. Written consent was obtained, students could withdraw without academic penalty and could ask for behavioural logs to be excluded. The college's research ethics committee approved the protocol.
The comparison group received conventional instruction, while the experimental group received the integrated four-layer package. The published abstract describes an ANCOVA that adjusted the final comparison for baseline scores. That helps address individual pre-test differences, but it does not create independent treatment units. The effective unit of assignment is the class, leaving one unit in each arm and no way to estimate variation between classes.
The researchers also analysed behavioural correlates. Task-completion volume and debugging efficiency were the strongest predictors of learning gains among the collected measures. Medium-experience learners appeared to benefit most from adaptive sequencing, while low-experience learners gained mainly on foundational dimensions. These subgroup findings are exploratory: dividing a 120-student sample reduces precision, and behaviour may reflect engagement or prior ability rather than cause improvement.[1]
Why the effect size should not be treated as a product claim
Cohen's d of 0.97 describes the standardised difference observed between these two classes. It is not a forecast that another school will gain nearly one standard deviation. With only one class in each condition, the estimate includes every unmeasured feature that separated those groups. Novelty, teacher attention, platform availability and different amounts of practice could all contribute.
The outcome itself was the study's composite computational-thinking measure, not an external examination, long-term retention test or independent project completed after the intervention. The paper reports no evidence here about whether students transferred skills to a new subject, retained them months later or became less dependent on automated support. Measurement tied closely to an intervention can capture genuine learning while still overestimating broader capability.
Nor does the study isolate costs. Building and maintaining adaptive resources, reviewing model output, supporting students and monitoring data can create substantial teacher and technical work. A scalable programme needs evidence about preparation time, accessibility, device and connectivity requirements, error correction and the students who disengage—not only average scores among those who complete the course.[1]
Practical implications for teachers and institutions
For teachers, the useful takeaway is the instructional architecture. Resources, adaptation, interaction and evaluation were designed together around named thinking skills. That is more defensible than giving students an unrestricted tool and hoping usage becomes learning. A local pilot could preserve this structure while simplifying the technology and measuring which component changes practice.
For students, adaptive sequencing may help match practice to current mastery, but it can also narrow what a system thinks they are ready to attempt. Learners should be able to see why material was recommended, ask for a different route and reach a teacher when the inferred level is wrong. Debugging and explanation should remain assessed human capabilities rather than being silently performed by the system.
For institutions, the safest next step is a limited, supported trial with independent assessment and a pre-agreed exit plan. Procurement should cover data minimisation, retention, accessibility, bias, teacher override and audit logs. The study's large within-site difference is a reason to investigate the model carefully, not a reason to replace conventional teaching across a curriculum.[1]
What would change the assessment
A stronger test would randomise many classes or students across several institutions, stratify by prior experience and use the same teachers or balanced teaching assignments across conditions. An active control should match resources, time-on-task and feedback while withholding the adaptive component. A dismantling design could then test whether resource curation, sequencing, multimodal interaction or the evaluation framework produces the gain.
Outcomes should include blinded performance on unfamiliar problems, retention months later, independent project quality, error detection, workload and accessibility. Researchers should publish attrition, missing data, system errors and results for students with different prior attainment and support needs. Formal equating of interim and final instruments would make learning trajectories interpretable.
Until then, the finding remains encouraging but local: one AI-assisted class outscored one conventional class after a term-long package. The work was funded by a project on a campus intelligent learning assistant based on generative AI, and the authors declared no competing interests. Replication that separates treatment from class is necessary before the effect can guide wide adoption.[1]
What this means for people
- Teachers may gain a clearer structure for adaptive practice, but need protected time and authority to review recommendations and correct system errors.
- Students need transparent pathways, teacher access and assessment that preserves their own ability to decompose, debug and generalise.
- School and university leaders should treat the result as a pilot signal and avoid buying a broad rollout before independent learning and workload evidence exists.
Global context
The pilot took place at one medical college in Henan, China, with two intact classes and a locally designed campus-learning project. Curriculum, class size, device access, teaching roles and assessment culture vary internationally. The four-layer architecture can inform replication, but the reported effect should not be transferred to other education systems without multi-site testing.
What the evidence does not yet show
- Only two intact classes at one Chinese medical college participated, with one class per condition; class and treatment effects cannot be separated.
- The intervention bundled resources, adaptive sequencing, multimodal support and evaluation, so the contribution of any one AI component is unknown.
- Week-five and week-nine abbreviated assessments were not formally equated to the full instrument and support only descriptive trajectory claims.
- The study did not establish long-term retention, transfer to unfamiliar tasks, independent project performance or reduced dependence on AI support.
- Costs, teacher workload, accessibility, system errors and infrastructure requirements were not central outcomes in the reported comparison.
What to watch next
- Multi-class, multi-site randomised studies that separate class, teacher and treatment effects.
- Active-control and dismantling trials identifying which layer of the package produces measurable benefit.
- Blinded tests of transfer, delayed retention, debugging and independent problem solving.
- Outcomes by prior experience, disability, device access and connectivity, with attrition reported.
- Teacher workload, implementation cost, error handling and student override in real deployment.
Living evidence record
Impact record IAI-074PZGN
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
10 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Scientific Reports published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 10 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Education
How did AI change a teaching team?
A two-year action-research study found that AI-related discussion inside one Taiwanese biostatistics teaching team was selective and clustered: 4,697 LINE messages showed fewer AI ties than general communication, but strong reciprocity once those ties formed. The eight-node network cannot show that AI improved student learning or that the pattern generalises.
9 min · 1 source
Education
Did ChatGPT improve writing scores in a 16-week Taiwan course?
Twelve university students used ChatGPT throughout a five-stage inquiry-writing course. Their closed-book writing scores rose by 2.79 points, but the change was not statistically significant and the study had no comparison group.
8 min · 1 source
Education
What should design students know about AI?
Researchers in Guangzhou developed a 23-item scale covering technical skills, tool use, ethics and originality, perceived value, and independent evaluation of AI output. Two Chinese student samples supported the five-factor structure, but the instrument still needs cross-cultural and outcome validation.
8 min · 1 source
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.