Can zero-shot AI find what buyers value?
A peer-reviewed study turns 12,900 Amazon smartwatch reviews into product priorities without task-specific training. A 200-review human check supports the sentiment labels, but the attribute taxonomy and strategic conclusions were not independently validated.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1The pipeline analysed more than 12,900 Amazon smartwatch reviews collected from July to December 2024, using keyword extraction, several language models, BART-MNLI classification and importance–performance mapping.
- 2On a random 200-review human-annotated check, the zero-shot sentiment classifier reported 0.88 accuracy and a macro F1 score of 0.86; positive labels were easier than neutral ones.
- 3Only sentiment, not the review-to-attribute assignments or business recommendations, received a labelled validation test. The evidence is one product category, one platform and one historical review period.
Research topic
Whether a zero-shot language-model pipeline can convert unlabelled consumer reviews into reliable attribute-level product priorities without task-specific training data
The study builds a four-stage consumer-analysis pipeline
Researchers from Universitas Tidar, Sejong University and Kookmin University asked whether a business can organise consumer feedback without first paying for a large, product-specific labelled dataset. Their case study collected more than 12,900 Amazon smartwatch reviews through the Amazon Customer Feedback API from July to December 2024. One copy of the text was lowercased, tokenised, stripped of punctuation and stopwords, and lemmatised for keyword extraction. A raw-text copy was retained so the later classifier could use context and ordinary phrasing.
The proposed workflow is sequential rather than a single all-purpose chatbot prompt. KeyBERT first compared review and candidate-term embeddings, including one- and two-word phrases. Maximal Marginal Relevance then balanced relevance against repetition. A similarity threshold reduced 17,736 unconstrained candidates to 240. Language models filtered that list further, a zero-shot classifier mapped reviews to product attributes, another zero-shot step labelled sentiment, and an importance–performance analysis combined how often an attribute appeared with the average polarity of the reviews assigned to it. That separation makes the design inspectable, although errors in an early stage can still travel into every later result.[1]
The Impact Brief · Free
Follow the evidence in finance & business.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
Several models agreed on the broad vocabulary but not the exact list
The authors compared Llama 3.3 70B, GPT-5 nano, Gemini 2.5 Flash and DeepSeek R1 T2 as filters for the 240 candidate terms. Their retained lists ranged from 28 keywords for Llama to 75 for DeepSeek, with repeated concepts such as battery, display, strap, GPS, connectivity, notifications and waterproofing. The team selected the broader DeepSeek output so less frequent attributes would not be discarded before classification. Two human reviewers also screened the candidate pool to remove irrelevant terms and merge overlapping expressions.
A prompt then consolidated the vocabulary into eight categories: health monitoring, connectivity, display technology, design and comfort, software and smart features, activity and location tracking, power and battery management, and general functionality. For large-scale assignment, the study used the open facebook/bart-large-mnli natural-language-inference model. Each review and the eight category labels were supplied to the Hugging Face zero-shot pipeline, and the highest-scoring label became that review's product attribute. This is a reproducible classification rule, but the paper does not report a labelled evaluation of whether those assignments were correct, whether multi-attribute reviews were forced into an unsuitable single category, or how often the models disagreed on the taxonomy itself.[1]
A 200-review check supports the sentiment component
The same BART-MNLI model classified each review as positive, neutral or negative. Researchers converted those classes to scores of plus one, zero and minus one. Across the complete corpus, 8,524 reviews were labelled positive, 2,311 neutral and 2,087 negative, reported as 65.8%, 17.8% and 16.4%. To test this stage, two independent annotators labelled a random sample of 200 reviews covering different products, brands and expressions; disagreements were resolved by discussion to create a consensus reference label.
Against that reference, the automated sentiment labels achieved 0.88 overall accuracy and a macro-averaged F1 score of 0.86. Positive reviews produced precision of 0.91, recall of 0.93 and F1 of 0.92. Neutral reviews were harder, with precision of 0.82, recall of 0.79 and F1 of 0.80; negative reviews reached 0.88, 0.86 and 0.87. The check is useful because it measures the chosen model on the actual corpus instead of citing performance from an unrelated benchmark. It remains modest: 200 examples are about 1.6% of the dataset, the consensus process removes an independent measure of ambiguous cases, and the paper does not report confidence intervals or annotator agreement before reconciliation.[1]
The business map measures attention and sentiment, not demand
For each attribute, the study treated frequency as importance and mean sentiment as performance. That placed design and comfort, connectivity and health-monitoring features among the more prominent, positively perceived topics, while power and battery management emerged as an area of dissatisfaction. Brand-level summaries suggested Apple, Samsung and Garmin attracted much of the discussion while showing only moderate differences in sentiment. These outputs can help a product team decide which themes to investigate, especially when manually reading thousands of reviews would be slow.
But frequency is not the same as purchase importance. A feature may be mentioned because it failed, because a listing emphasised it, because one brand has many more reviews or because reviewers repeat a familiar complaint. A star rating, return, warranty claim, sales outcome or controlled choice experiment would answer a different question. Amazon reviewers are also a selected population: people who bought, used and chose to review a smartwatch, during one six-month period. The pipeline cannot establish that silent customers, prospective buyers or people in another market value the same attributes. Product managers should read the quadrants as a structured listening aid, not a causal ranking of investments or evidence that changing one feature will increase sales.[1]
Human review remains part of the method
Although described as zero-shot, the complete workflow is not hands-off automation. People selected thresholds, reviewed candidate keywords, reconciled attribute categories, chose the widest model output and created the human sentiment reference. Those decisions may be sensible, but they shape the final map and would need governance in a commercial setting. Teams should log model versions and prompts, preserve review dates, test for duplicated or manipulated reviews, and give analysts a way to inspect uncertain or multi-topic cases rather than accepting one label automatically.
The study was supported by Korean government information-technology and research grants and by Sejong University's industry–academic cooperation foundation; the authors declared no competing interests. It does not report whether any smartwatch manufacturer supplied data or influenced the analysis. Its strongest contribution is methodological: a relatively accessible pipeline can compress a large unlabelled corpus, and its sentiment component agrees well with a small human check. The paper is not evidence that any named brand leads the market, that automated review mining should replace customer research, or that the same accuracy will hold for another product category or language.[1]
What would change the assessment
Confidence would rise with an external test fixed before development: reviews from another retailer, time period, country and product category, with labelled attribute and sentiment assignments and brand-stratified error reporting. A multi-label comparator would show whether forcing one attribute per review loses important information. Publishing the collection protocol, code, prompts, model snapshots and de-identified examples would allow independent teams to reproduce the pipeline and test sensitivity to thresholds and model substitutions.
The more consequential business claim also needs outcome evidence. A prospective team could use the map to choose a product change, record the decision before launch, and compare sales, returns, support contacts or customer satisfaction against a credible control. Until that happens, the study supports zero-shot review analysis as a promising research and triage tool. It does not demonstrate that the inferred priorities are the right investments, that a 0.88 sentiment classifier is accurate enough for every decision, or that consumer preference can be reduced to mentions and polarity alone.[1]
What this means for people
- Faster review analysis could help product teams find recurring usability and battery complaints that deserve investigation.
- Customers can be misrepresented if ambiguous, multi-topic or non-English reviews are forced into a single label without inspection.
- Business decisions should preserve a route for human review and should not infer individual reviewer intent from an aggregate model score.
Global context
The research joins Indonesian engineering and Korean AI teams and tests a globally familiar e-commerce problem on English-language Amazon smartwatch reviews. The method could lower the cost of exploratory consumer analysis, but retailer cultures, review manipulation, product mixes, language and purchasing power vary across markets. Local validation is necessary before the same categories or sentiment thresholds guide decisions elsewhere.
What the evidence does not yet show
- The evidence comes from one product category, one retailer and reviews collected between July and December 2024.
- Only 200 reviews received a human sentiment check; confidence intervals and pre-consensus annotator agreement were not reported.
- The study did not validate the review-to-attribute assignments against a labelled reference set.
- Attribute frequency was treated as importance, which may not match willingness to pay, purchase decisions or business value.
- The pipeline includes human screening and model-selection decisions, so it is not a fully automatic or model-independent method.
- Transfer to other languages, retailers, products and later model versions remains untested.
What to watch next
- External replication on another retailer, language, product category and later time period.
- Human-labelled, multi-label evaluation of attribute assignment as well as sentiment.
- Sensitivity tests across prompts, thresholds and open versus proprietary model versions.
- Prospective evidence that a priority inferred from reviews improves a measured customer or business outcome.
Living evidence record
Impact record IAI-18RL59V
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Scientific Reports published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 5 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Finance & Business
Does Micron's record quarter prove the AI infrastructure boom will last?
Micron reported $54.23 billion in quarterly revenue and $32 billion of customer commitments under long-term supply agreements. The figures show extraordinary demand for memory and storage, but the company's outlook is not proof that every AI investment will earn a return.
6 min · 3 sources
Finance & Business
Is AI financing becoming circular?
A Bank for International Settlements analysis maps 1,246 AI firms and 972 investment relationships, finding that commercial links accompanied 16.1% of AI-to-AI deals by count but 46.4% by disclosed value. The pattern is consequential, yet it does not show that the transactions were improper, unprofitable or certain to spread financial stress.
10 min · 5 sources
Finance & Business
Can an LLM explain a fraud alert without seeing transaction data?
An Indian research team tested a graph detector, fixed decision rules and a constrained language model on a 1,000-transaction Nigerian sample. The design limits what the LLM can invent, but simpler tabular models detected fraud better and no auditors judged the explanations.
9 min · 1 source
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.