LLM‑Generated Data Outperforms Human Experiments in Persuasion Games

In the July 27, 2026 issue of the Journal of Artificial Intelligence Research (JAIR), Eilam Shapira, Omer Madmon, Roi Reichart and Moshe Tennenholtz report that large language models (LLMs) can generate synthetic datasets that allow neural predictors to forecast human decisions in strategic persuasion games with higher accuracy than when trained directly on real human experiments ¹. The authors argue that generating choice data via an LLM removes the need for costly human studies while potentially revealing novel strategies missed by traditional methods.

The work focuses on “language-based persuasion games,” a class of economic settings where agents communicate through natural language to convince each other about preferences or intentions. Unlike simple binary-choice tasks, these games require modeling not only individual preferences but also beliefs about others’ reasoning—a challenge noted since Schelling’s 1960 exploration of strategic stability. In their experiments, the team used an off-the-shelf LLM to simulate thousands of game rounds, producing dialogue transcripts paired with final decisions. They then trained machine-learning models to predict participants’ choices from those dialogues. When tested against a held‑out set of real human interactions collected in controlled laboratory studies, the LLM‑trained models achieved higher accuracy than counterparts trained directly on the human data.

To verify robustness, the researchers performed cross‑validation across multiple game variants differing in payoff structures and linguistic constraints. Across all conditions, the LLM-generated training sets consistently yielded better predictive performance, improving accuracy by roughly four to six percentage points. Ablation tests revealed that removing the model’s ability to reason about counterfactuals degraded predictions more sharply than eliminating raw textual features, underscoring the importance of belief modelling.

While the results are encouraging, the authors caution against overgeneralizing. Distributional shifts between synthetic and observed behavior may still exist, potentially limiting transfer to other domains or populations ¹. The paper echoes earlier attempts such as Zhao et al.’s 2024 study which showed limited success for simpler bias simulations but struggled with complex scenarios. By contrast, Shapira et al. demonstrate that an appropriately fine-tuned LLM can capture strategic nuances required for persuasion games.

The implications extend beyond economics. If language models can reliably generate high-quality behavioral datasets, fields ranging from marketing to public policy could reduce reliance on expensive field experiments. Experts emphasize careful validation: “LLM‑generated data should be used as a supplement rather than a replacement until we understand how closely they mirror real human reasoning” ¹.

In sum, this JAIR contribution marks a significant step toward using artificial intelligence not only as a predictor but also as a creative partner in generating experimental data for social sciences.