If AI won’t follow the rules, should the media even try?

2 hours ago7 min read

The digital landscape for publishers has become a treacherous ethical frontier, reminiscent of Asimov's fictional dilemmas about human-robot coexistence. When large language models and AI search engines systematically ingest and repurpose online content, they don't just redirect traffic—they challenge the fundamental economics of digital journalism.The core conflict isn't merely technical but philosophical: technology companies operate under what I'd term the 'digital frontier doctrine,' where anything publicly accessible becomes fair game for algorithmic consumption, while media organizations cling to traditional copyright frameworks that feel increasingly archaic in this new environment. Consider the recent revelations about Common Crawl, the foundational dataset that trained GPT-3.5 and countless other models. Despite receiving takedown requests from major publishers including The New York Times and Reuters, Common Crawl appears to have maintained this content in its archives while merely hiding it from public search—a digital sleight-of-hand that speaks volumes about the priorities of AI development.Executive director Rich Skrenta's statement that 'You shouldn't have put your content on the internet if you didn't want it to be on the internet' reveals a worldview where digital publication constitutes implicit consent for unlimited algorithmic harvesting, a perspective that would have seemed dystopian just a decade ago. Simultaneously, AI-powered browsers like Perplexity Comet and ChatGPT Atlas demonstrate another dimension of this access-at-all-costs mentality.As documented by Columbia Journalism Review, these systems successfully circumvent paywalls that would normally protect subscription content, effectively granting every user the technical sophistication of the most determined digital bypass artist. This represents what I call the 'scale paradox'—behaviors that might be ethically ambiguous but practically insignificant when performed by individual humans become systematically destructive when automated and deployed at internet scale.The implications extend beyond immediate revenue loss to threaten the entire subscription-based business model that many quality publications depend on for survival. What makes this conflict particularly fascinating from a policy perspective is the inherent contradiction in the technology sector's positioning.On one hand, AI companies frequently argue that individual content sources are easily replaceable in their training datasets—OpenAI's CEO has suggested they could train models without publisher data if necessary. Yet Common Crawl's apparent reluctance to fully comply with removal requests from prestigious publications suggests these sources carry disproportionate value, creating what economists would call a 'free rider problem' where AI companies benefit from content they haven't contributed to producing.The emergence of AI browsing agents introduces additional complexity to the consent framework. While OpenAI states that Atlas won't train on pages its agent accesses, the company acknowledges retaining this data for user memory functions—a distinction that feels increasingly semantic in an ecosystem where data retention inherently creates future utility.This technological arms race is pushing publishers toward more robust server-side paywalls, but such defensive measures represent additional costs for an industry already facing economic headwinds. The fundamental question isn't whether AI will continue accessing content—it will—but whether media organizations can establish frameworks that acknowledge their content's value while adapting to technological realities.The solution likely lies not in absolute blocking but in sophisticated content strategy that differentiates between training access, summary usage, and direct competition, potentially leveraging emerging standards like the Robots Exclusion Protocol for AI or developing new licensing models specifically for machine consumption. What's clear is that publishers possess more leverage than they might assume—the very aggressiveness of AI companies in seeking content access demonstrates its indispensable value. The challenge now is transforming that need into sustainable economic models before the foundational economics of quality journalism erode beyond repair.

#AI

#media

#content

#copyright

#paywalls

#regulation

#Common Crawl

#featured

Stay Informed. Act Smarter.

Get weekly highlights, major headlines, and expert insights — then put your knowledge to work in our live prediction markets.