In 2026, there was a preprint posted by Sydney Sears and Deena Skolnick Weisberg looking at people's ability to identify whether a short story was written by AI, whether they preferred AI-written or human-written stories, and whether that perception of enjoyment was affected by what they were told about authorship. *(Hat tip to science communicator [Alex Dainis](https://linktr.ee/alexdainis) for creating a nice summary of this one.)* "[Can people tell the difference between stories written by a human or by an AI system?](https://osf.io/preprints/psyarxiv/jkh6p_v5)" Sydney Sears & Deena Skolnick Weisberg, preprint accepted at the journal, Judgment and Decision Making, June 2026. (See pdf at the end of this page.) **Abstract** How do people perceive AI-generated fictional stories? In Study 1, participants (1,682 adults recruited from Prolific) read either a human-written or AI-generated story. Half were told that their story was written by a human, and half were told that their story was written by ChatGPT. Participants rated the AI-generated story as higher quality and more absorbing, but they rated the story better on both measures when they were told it was written by a human. In Studies 2 and 3, participants (905 adults recruited from Prolific) read both types of story and guessed each story’s origins. Participants were no better than chance at differentiating between the human-written and AI-generated stories. Self-reported expertise with AI, but not with fictional literature, was positively correlated with correct story identification. AI programs thus can generate creative work that is viewed as being at least as good as human-generated work, but are not perceived as being able to do so. Keywords: artificial intelligence, Turing test, fictional stories, creative writing --- #### Can humans identify whether AI wrote a short story? In one of the three studies conducted by the authors, they sought to decide whether humans can correctly identify authorship of short stories. Participants were given pairs of stories -- each pair consisting of one human-written story and one AI-written story -- and they were asked to identify which was written by a human. So, if the participants have no skill (or anti-skill?), their success probability should be $\theta_0 = 0.5$. The natural probability model for the number of correct guesses is $X \sim \mathrm{Binom}(n = 424, \theta)$. Under the null hypothesis that $\theta = \theta_0 = 0.5$, the expected number of correct guesses would be $\mu_0 = n \theta_0 = 424 \times 0.5 = 212$ and the standard deviation would be $\sigma_0 = \sqrt{n \theta_0 (1-\theta_0)} = 10.30$. As a result, a two-sided $2 \sigma$-test would have critical values $212 \pm 2\times10.30$. | Component | Value | | ---------------------: | :-------------------------------------------------------------------- | | Null Hypothesis | $H_0: \theta = 0.5$, the humans have no bias in assessing authorship. | | Alternative Hypothesis | $H_A: \theta \neq 0.5$ | | Test Statistic | $x =$ the number of correct guesses | | Rejection Criterion | $x<191.40$ or $x > 232.60$ | We can visualize the null hypothesis as follows: ![[Bot_or_not_2sigma.png]] The experimental outcome is summarized in the following table: | Correct guesses | Incorrect guesses | Total | | :-------------: | :---------------: | :---: | | 167 | 257 | 424 | Since the experimental outcome for the number of correct guesses was 167, we can reject the null hypothesis that humans have no bias in assessing authorship of these stories. *In fact, the humans are significantly bad at it!* There are a number of latent variables that could play a role in this: for example gender, familiarity/enthusiasm for AI, pre-existing critical reading skills. The authors of the study took a closer look at possible contributions from these various factors using more sophisticated techniques. It's worth a read, particularly to look at what happens to people's assessment of how they enjoyed the stories when they were told (or lied to!) about who authored the stories. ###### Potential project: Study the role of the stories that were presented A hidden variable in the analysis is whether one of the pairs of stories was particularly difficult for participants to identify authorship. | | Pair 1 | Pair 2 | Pair 3 | Total | | -----------------------: | :----: | :----: | :----: | ----- | | **Guessed human author** | | | | | | **Guessed AI author** | | | | | | **Total** | | | | | --- ![[2026 Sears Bot or Not.pdf]]