Imagine a marketing team submitting an AI-assisted report to request more budget. The manager checks the sales figures against the source records. They match. One question remains: are we judging the campaign by sales, or by profit after the relevant costs?
Checking the arithmetic and choosing what to calculate require separate judgments. This marketing scenario is hypothetical, offered as a way to examine a reporting process. It is not a result from a marketing experiment.
What the study found
Anthropic published the study on October 27, 2026. Of 61 recruits, 11 opted out of using AI. The main analysis covered 50 analysts using Claude Code with Claude Opus 4.8 at High effort on a fictional farming-cooperative task. Hidden randomized instructions deliberately strengthened the agent's preferences. Anthropic's study design
Across four analytical choices, participants matched the assigned AI lean 86% of the time, against 50% expected without influence from random assignment. This does not mean 86% were wrong, or describe incorrect final recommendations. Both options could be defensible. Full paper, sections 1.1 and 4
The choices concerned the outcome metric, comparison method, target population and weighting. They preceded the recommendation. The task's analytical choices
The registered final-recommendation estimate was 17% of the discontinue-to-rollout scale, with extension halfway between, and was not statistically significant at .05 (p = .069). Using only the four randomly assigned analytical leans predicted 82% of recommendations, versus a 61% majority baseline. This was exploratory, in-sample and excluded extension recommendations. It was not validated predictive accuracy on new tasks. Full paper, Figure 1 and section 1.1
Back to the marketing report
Consider two reports built from the same transaction records. One asks which campaign generated more sales. The other asks which left more profit after discounts, product costs and the expenses explicitly included in the calculation. They may point in different directions without an arithmetic error. They answer different questions.
A team seeking new customers might accept lower initial profit. A manager protecting short-term profit needs to know which costs and time period the report covers. The reason for choosing a measure belongs before the chart, where a reader can examine it before reaching the recommendation.
The population matters too. Studying only people who purchased answers a different question from studying everyone who received an offer. Giving every customer equal weight also answers a different question from weighting by purchase value. The business owner should confirm what they want to learn, and about whom.
This article proposes asking AI to present another defensible analytical approach before producing the main report, explaining how each approach changes the question being answered. The study did not establish that this practice reduces deferral. It is a process proposal for teams to try and evaluate.
Three questions before approval
- What decision are we making, and how does this metric address it? Ask for the objective, metric definition, included costs and time period.
- Who does this result apply to? Identify included and excluded groups and the weighting rule, so the conclusion stays within the analysis's scope.
- What alternative did we consider and reject, and why? Request another reasonable approach and an explanation of how it changes the result or its meaning.
The report's author should answer in their own words, recording which suggestions came from AI and which criteria the business owner chose. Then review the calculations, source data and recommendation.
Keep the limits attached
The study used one fictional task and one model, with deliberately strengthened AI preferences. Participants knew causal inference but were new to farming. Generalization to other workplaces remains unknown. The no-AI group self-selected and differed in background, preventing causal claims about time savings or errors. The researchers' limitations
A reviewer's name establishes responsibility. Written criteria help that reviewer understand which question the report actually answers. For the team's next report, ask the business owner to answer these three questions before handing the analysis to AI.
Original AI-generated editorial illustration showing alternative analytical lenses over the same data, rather than experimental results.
