Newly unsealed court documents from the ongoing copyright battle between The New York Times and the partnership of OpenAI and Microsoft have revealed striking internal admissions. According to the unredacted filings, top executives at both companies privately acknowledged that their AI training practices could be described as theft and posed an existential threat to the journalism industry.
The filings provide a rare glimpse into the private communications of the people building the worlds most prominent generative AI models. The documents detail how the companies allegedly bypassed paywalls, stripped copyright notices, and built massive datasets using scraped content, all while acknowledging the potential damage to the very sources that provided their training data.
Internal admissions of theft
One of the most significant revelations in the filing involves Brent Hecht, Microsoft's Director of Applied Science. In a January 2023 internal memo, Hecht reportedly described the scale of AI data collection as an astonishing theft of unprecedented proportions. He further characterized the practice as the largest theft of labor in human history.
This internal sentiment contrasts sharply with the public legal defenses typically mounted by AI companies, which argue that training models on publicly available internet data constitutes fair use.
A year later, in January 2024, Hecht authored a presentation describing a doom loop. He warned that the decline in traffic to original content sources would eventually hurt the performance of AI models and the entire web at the same time. The filing quotes a Microsoft document stating that it is highly unusual for an end-product to threaten the economic foundations of its essential suppliers, yet that is the situation the company created for its LLM business.
The substitution effect
A key pillar of the fair use defense is the requirement that a new product does not substitute or harm the market for the original work. However, the unredacted filings suggest that OpenAI and Microsoft executives were aware that their products were doing exactly that.
According to the documents:
- Microsoft's own data indicated that its Copilot answer engine caused click-through rates for The New York Times domain to drop by as much as 93 percent compared to traditional Bing search results.
- Nick Turley, the head of ChatGPT at OpenAI, wrote in internal communications that publishers face an existential threat because AI products are largely substitutive.
- Turley noted that these tools would become more and more substitutive as they improved.
- Microsoft CEO Satya Nadella agreed under oath earlier this year that interacting with a chatbot has substituted the need for a user to go to the underlying source website.
These admissions are particularly damaging to the argument that AI models are transformative rather than competitive. If a model provides the full value of a news article within its own interface, it effectively removes the economic incentive for the user to visit the publisher, thereby disrupting the employment of the people who generated the data in the first place.
Circumventing paywalls and stripping metadata
The filing also alleges that OpenAI and Microsoft used technical workarounds to gather data they knew was behind paywalls. In one exchange, OpenAI researcher Nick Ryder told OpenAI President Greg Brockman about a hack to get around the New York Times paywall. Brockman reportedly replied, "ah nice."
Beyond bypassing paywalls, the documents describe a deliberate effort to remove copyright notices from training data. Researchers reportedly wanted to avoid having the models output copyright notices to users, which led to the stripping of that metadata before the information reached the model.
The scale of the copying revealed in the filings is substantial. OpenAI's mid-training datasets reportedly contained more than 91,692 copies of works from The New York Times, the Daily News, and the Center for Investigative Reporting. Furthermore, a dataset derived from Common Crawl included more than 2 million documents from nytimes.com.
The filings also shed light on internal data-sharing initiatives between the two partners. Microsoft reportedly provided training data to OpenAI through projects codenamed Project Taxi and Project Mango. The latter resulted in a dataset containing copies of at least 160,903 unique works from various news publishers.
Executive testimony and the licensing debate
During a deposition earlier this year, Satya Nadella made comments that appeared to distance Microsoft from some of OpenAI's more aggressive scraping tactics. Nadella testified that anything that is paywalled should be licensed by anyone who wants to use it for grounding or training.
Nadella further stated that if he had been made aware that OpenAI had scraped and trained on information behind a paywall, he would have invoked Microsoft's right to require OpenAI to retrain its models.
Despite these statements, the filings allege that Microsoft was deeply involved in the acquisition and evaluation of these datasets. The plaintiffs argue that Microsoft used the GPT-3 training dataset, provided by OpenAI, to determine how to implement the models into its own commercial products.
What happens next?
These unredacted filings represent a significant escalation in the legal conflict. While the U.S. government recently filed a brief in support of OpenAI's position on fair use, these internal admissions could make it harder for judges to ignore the economic harm caused to publishers.
The case centers on whether AI's use of data is transformative. If the courts find that the technology is primarily a substitute for the original work, as internal communications from Microsoft and OpenAI executives seem to suggest, it could force a massive shift in how AI companies compensate content creators.
For now, the focus remains on the discovery phase and the unsealing of further exhibits. The outcome of this case will likely set the precedent for the entire AI industry's content supply chain and the future of digital journalism.
Filed under: AI, TechNews, Software, Microsoft, OpenAI, Copyright, Journalism