Did ChatGPT improve writing scores in a 16-week Taiwan course?
Twelve university students used ChatGPT throughout a five-stage inquiry-writing course. Their closed-book writing scores rose by 2.79 points, but the change was not statistically significant and the study had no comparison group.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1The study followed 12 English-major students in one 16-week composition course at a national university in Taiwan; there was no comparison group or random assignment.
- 2Mean closed-book writing scores rose from 78.13 to 80.92, a 2.79-point change that was not statistically significant: t(11) = -1.63, p = .13.
- 3Qualitative materials suggested increasingly specific prompting and selective use of AI output, but incomplete interaction logs and teacher-led course design limit what can be inferred.
Research topic
How students used ChatGPT when it was embedded across a five-stage inquiry-writing course, and whether their closed-book argumentative writing scores changed
The clearest measured result is a non-result
A new peer-reviewed study from Taiwan offers a useful counterweight to claims that simply adding a chatbot improves student writing. Twelve university students completed a 16-week English-as-a-foreign-language composition course in which ChatGPT was threaded through a five-stage inquiry cycle: Engage, Explore, Explain, Elaborate and Evaluate. The students' average closed-book argumentative-writing score increased, but the change did not meet the study's threshold for statistical significance.
The mean rose from 78.13 before the course to 80.92 afterwards, an increase of 2.79 points. The paired-samples test produced t(11) = -1.63 and p = .13, with Cohen's d = 0.47. A moderate effect-size estimate in a group this small is not proof of benefit: sampling uncertainty is wide, there was no control group, and the two tests used different topics. The responsible reading is that this class showed a modest upward pattern that could reflect the intervention, ordinary teaching, practice, topic difficulty or chance.[1]
The Impact Brief · Free
Follow the evidence in education.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
What the course actually asked students to do
The research took place in an intact undergraduate English-major course at a national university in Taiwan during autumn 2024. ChatGPT was not presented as a free-standing answer generator. Students used it during a sequence that moved from framing a topic and designing a questionnaire to interpreting findings, drafting sections and evaluating a completed research report. The instructor also trained students to specify roles, constraints and output formats in their prompts and to compare AI suggestions with peer and teacher feedback.
That design matters because the intervention was a package: an inquiry framework, normal instruction, peer discussion, reflective work and ChatGPT use. The study cannot isolate which element produced any change. It also cannot tell whether the same students would have developed similar habits in a well-taught inquiry course without the chatbot. The paper explicitly describes the research as exploratory rather than causal, a boundary that schools should preserve when considering adoption.[1]
The assessment measured a related but different kind of writing
At the beginning and end of the semester, students wrote an IELTS-style Task 2 essay for 40 minutes without ChatGPT or online resources. Two experienced EFL teachers scored task response, coherence and cohesion, lexical resource, and grammatical range and accuracy. Inter-rater reliability for the overall score was reported as ICC(2,2) = 0.80, with a 95% bootstrap confidence interval from 0.70 to 0.87.
The pre-test asked whether social media is more harmful or helpful, while the post-test concerned whether computer translation makes foreign-language learning a waste of time. Those prompts are not identical, and neither directly measures the extended research-writing tasks central to the course. Task response and grammatical range showed the largest mean gains, with p values of .09 and .08, while coherence and lexical-resource changes were smaller. None of those comparisons establishes that ChatGPT caused improvement.[1]
The qualitative record points to a teaching practice, not an efficacy verdict
The study also analysed 30 peer-discussion recordings, staged writing artefacts, submitted ChatGPT interaction screenshots or links, and end-of-course reflective journals. The authors report that students moved from broad requests toward more phase-specific prompting: asking for neutral questionnaire wording during exploration, candidate themes during analysis, or a whole-text clarity check during evaluation. Students sometimes revised or rejected suggestions rather than accepting them wholesale.
Those observations are useful because they make the learning process visible. They suggest that structured tasks can turn AI use into something teachers can discuss and critique. But the interaction logs were incomplete and used as supplementary evidence, so the record does not capture deleted, informal or unsubmitted conversations. Reflective journals were the only source for students' overall perceptions. Themes about agency or critical filtering therefore describe this small class; they do not measure independent learning gains across a wider student population.[1]
For teachers, the practical lesson is about scaffolding
The most defensible application is not a claim that ChatGPT raises scores. It is a design hypothesis: tie AI use to a defined stage of inquiry, require students to record and evaluate suggestions, and keep peers and instructors in the feedback loop. A teacher could ask students to explain why they accepted or rejected an AI-proposed survey question, identify evidence that supports an interpretation, and preserve versions showing how a draft changed. Those activities make judgment and authorship assessable.
The design also surfaces risks. If a course grades only polished final prose, students may optimise for fluent output instead of learning. If prompting becomes a substitute for subject knowledge, they may lack the basis to reject a plausible error. And if access to capable tools, devices or language support differs, an AI requirement can widen rather than narrow opportunity. Schools need an accessibility plan, transparent rules, protected routes for opting out and assessments that still reveal what students can do without the system.[1]
Funding and researcher roles deserve context
Taiwan's National Science and Technology Council supported the work under grant NSTC 113-2410-H-194-095. The paper declares no competing interests. The sole author designed the study and course intervention, collected the data, analysed the quantitative and qualitative material, and wrote the manuscript. Two EFL teachers independently rated the writing assessments, and subsets of qualitative material were double-coded, with initial agreement reported at 84% for peer-discussion segments and 88% for reflective journals.
That level of disclosure helps readers understand the study, but it cannot remove expectancy effects when the researcher is also closely involved with the intervention. Students knew they were participating, and a small intact class can be unusually responsive to a teacher's emphasis. Independent replication by instructors who did not design the framework would make the evidence more persuasive.[1]
What would change the assessment
A stronger test would compare multiple classes across institutions, ideally allocating similar students to the same writing curriculum with and without the structured ChatGPT component. It should pre-register primary outcomes; use equivalent or counterbalanced writing prompts; score work blind to group and timepoint; and report confidence intervals, attrition and tool usage. Direct ratings of research reports would better match the intervention than relying mainly on a timed argumentative essay.
Longer follow-up would answer whether students retain prompting and critical-evaluation skills after support is withdrawn. Process logs could show which AI interactions precede productive revision, while interviews and classroom observation could test whether reported agency matches behaviour. Until then, the paper supports careful experimentation with structured AI pedagogy. It does not show that ChatGPT improved writing scores, that the five-stage method is superior to other teaching, or that the approach will transfer beyond one 12-person Taiwanese class.[1]
What this means for people
- Students should not be told that the study proves ChatGPT improves writing; its measured score change was uncertain.
- Teachers can use the five-stage structure as a testable lesson-design idea while keeping evaluation and authorship explicit.
- Universities need accessible alternatives and assessments that show what students can do independently as well as with AI support.
Global context
The study comes from an English-major course in Taiwan, where students were writing in a foreign language and working within a specific university curriculum. That makes it relevant to international EFL education but not automatically transferable to secondary schools, first-language writing, other disciplines or institutions with different assessment rules. Cross-country replication should separate the effects of language proficiency, teacher practice, tool access and local academic-integrity expectations rather than treating one classroom as a universal model.
What the evidence does not yet show
- The sample contained only 12 students from one English-major course at one university in Taiwan.
- There was no control group or random assignment, so the study cannot attribute score changes to ChatGPT or the 5E framework.
- The pre- and post-tests used different prompts, and the timed essay did not directly measure the extended inquiry-writing skills taught in the course.
- The overall 2.79-point gain was not statistically significant at p = .13.
- Submitted ChatGPT logs were incomplete, and reflective journals were the sole evidence for overall student perceptions.
- The sole author designed and taught the intervention as well as leading data collection and analysis.
What to watch next
- Multi-class controlled studies with larger samples and independent instructors.
- Blind scoring of equivalent writing tasks and direct assessment of full research reports.
- Complete process logs that link specific AI interactions to revisions without surveilling students unnecessarily.
- Evidence on retention, authorship, academic integrity and differential effects by language proficiency or access.
- Comparisons with equally intensive inquiry teaching that does not use generative AI.
Living evidence record
Impact record IAI-0LGIGJ0
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Asian-Pacific Journal of Second and Foreign Language Education published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 5 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Education
How did AI change a teaching team?
A two-year action-research study found that AI-related discussion inside one Taiwanese biostatistics teaching team was selective and clustered: 4,697 LINE messages showed fewer AI ties than general communication, but strong reciprocity once those ties formed. The eight-node network cannot show that AI improved student learning or that the pattern generalises.
9 min · 1 source
Education
Does open AI access still let teachers see student judgement?
In one 33-manuscript university course, structural writing clustered near the top while evidence use and theoretical interpretation still varied. The case supports assessment redesign, not a claim that AI improved learning.
7 min · 1 source
Education
Does AI help students think about learning?
A 33-study meta-analysis found a positive average for technology-supported metacognition, but almost all effects came from self-report and the review did not show that AI outperformed other educational technology.
8 min · 1 source
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.