Li, Haider, Kim, Luo, Luo, Yuan, Watts, Vidal, Callison-Burch and colleagues built WorldStateMedia by processing Fineweb, a large web crawl corpus, and matching URLs to outlets tracked in the State Media Monitor database, which categorizes outlets by level of state control.
State media articles now dominate English web crawl content
A new dataset of 118 million articles from 136 countries shows editorially controlled outlets surged from 39% to 93% of English state media pages since 2013, and retrieval over such media shifts LLM answers toward state claims.
Big Tech
Bryan Li · Samar Haider · Grace Kim · Fiona Luo · Jinqi Luo · Yuan Yuan · +3 more
University of Pennsylvania · University of California, Berkeley · University of California, Los Angeles · Google · Northwestern University
Research Digest··3 min read
The authors introduce WorldStateMedia, an open dataset of 118 million state media articles in 114 languages, built by linking Fineweb web crawl data to outlet-level state control annotations.
§