Research talk:Newsletter/2024/August

See also the discussion page of the Signpost version of this issue

Identical text perceived as *less* credible when presented as a Wikipedia article than as simulated ChatGPT or Alexa output

I see the article as problematic in the way it claims that the format in the experiment is in line with Wikipedia. The article format is just plain text with no hyperlinks, neither intrawiki links nor to external credible sources. For Wikipedia, the main way of raising credibility is via inline external references to credible sources. Intrawiki links may possible also contribute to (perceived) credibility. I see the same problem with missing references and hyperlinks in "Do You Trust ChatGPT? – Perceived Credibility of Human and AI-Generated Content" (Figure 1 panel c). Furthermore, the article ("Conversational presentation mode increases...") writes the phrase "a static text-based online encyclopedia like Wikipedia". It bothers me. Wikipedia is an interactive resource where you can following links and edit text in a collaborative way. In a peer-review, I would have been reluctant to accept their claim that the presentation format is like Wikipedia. — Finn Årup Nielsen (fnielsen) (talk) 03:57, 17 September 2024 (UTC)

It's always good to be skeptical of a paper's methodology of course, and to highlight an experiment's limitations.

But note that Figure 1 only shows the "unbranded" version. They also tested a "branded" Wikipedia mockup which did include citation links and wikilinks. (Unfortunately the authors didn't include a screenshot of that version in the paper itself, only in the supplement. That also meant that I wasn't able to include it in this review because unlike the paper itself, the supplement wasn't published under a free license. But I posted an example on Twitter instead.)

Yes, it looks like these hyperlinks were only simulated in a screenshot. But then again we know from other research that readers only very rarely click reference links (similar for the act of "edit[ing] text in a collaborative way"). That should make one optimistic that this limitation didn't affect the experiment's results too much.

Regards, HaeB (talk) 17:47, 2 October 2024 (UTC) (Tilman)