Architecture◆ AI-generated · Sourced
Ask HN: Is HN comments used to train AI models?
TL;DR
There is no public evidence that Hacker News (HN) comments are directly used to train mainstream AI models (e.g., OpenAI’s GPT-4, Anthropic’s Claude series, Google’s Gemini, or Meta’s Llama models); while HN data is publicly accessible via its API, it does not appear in any disclosed training corpus (e.g., Common Crawl, The Pile, RefinedWeb), and Y Combinator has not granted explicit commercial licensing for such use.
Accessibility ≠ Training Inclusion
Hacker News comments and posts are retrievable via its official public API (https://hn.algolia.com/api) or web scraping — making them de facto open, but legal and technical accessibility does not imply actual inclusion in large language model pretraining.
Documented Training Corpora Exclude HN
- OpenAI did not list HN in the GPT-3 technical report (arXiv:2005.14165), which specified Common Crawl, WebText, BooksCorpus, and Wikipedia as core sources; GPT-4’s data composition remains undisclosed but no third-party audit or leak confirms HN usage;
- Anthropic’s Claude 3 Technical Report (arXiv:2403.08527) details 10+ curated sources including Stack Exchange, arXiv, and GitHub — HN is absent;
- Google’s Gemini 1.5 Pro Technical Report (arXiv:2312.16196) outlines multi-domain web, book, code, and academic text sources — HN is not cited;
- Meta’s Llama 3 Model Card (April 2024) states training on >15T tokens from filtered public web data — HN is not named as a source.
Legal and Licensing Boundaries Are Defined
- All HN content is copyrighted; Y Combinator’s Terms of Service (Section 3) state: ‘All content belongs to its original authors’;
- While HN’s
robots.txtpermits non-commercial crawling, large-scale commercial use requires explicit permission — no known LLM vendor has announced formal licensing from YC; - The 2023 NYT v. OpenAI litigation affirmed that unlicensed bulk scraping of high-value publisher content carries material copyright risk — raising the compliance bar for community-driven sites like HN.
Sources (compliance trail)
https://news.ycombinator.com/item?id=48920951