A Marketplace for AI-Generated Adult Content and Deepfakes
By: Shalmoli Ghosh, Matthew R. DeVerna, Filippo Menczer
Potential Business Impact:
AI platform incentivizes harmful, explicit fake images.
Generative AI systems increasingly enable the production of highly realistic synthetic media. Civitai, a popular community-driven platform for AI-generated content, operates a monetized feature called Bounties, which allows users to commission the generation of content in exchange for payment. To examine how this mechanism is used and what content it incentivizes, we conduct a longitudinal analysis of all publicly available bounty requests collected over a 14-month period following the platform's launch. We find that the bounty marketplace is dominated by tools that let users steer AI models toward content they were not trained to generate. At the same time, requests for content that is "Not Safe For Work" are widespread and have increased steadily over time, now comprising a majority of all bounties. Participation in bounty creation is uneven, with 20% of requesters accounting for roughly half of requests. Requests for "deepfake" - media depicting identifiable real individuals - exhibit a higher concentration than other types of bounties. A nontrivial subset of these requests involves explicit deepfakes despite platform policies prohibiting such content. These bounties disproportionately target female celebrities, revealing a pronounced gender asymmetry in social harm. Together, these findings show how monetized, community-driven generative AI platforms can produce gendered harms, raising questions about consent, governance, and enforcement.
Similar Papers
Deepfakes on Demand: the rise of accessible non-consensual deepfake image generators
Computers and Society
Makes fake pictures of people easily.
Perpetuating Misogyny with Generative AI: How Model Personalization Normalizes Gendered Harm
Computers and Society
Stops AI from making fake, harmful pictures.
Advertising in AI systems: Society must be vigilant
Artificial Intelligence
AI might show ads to make money.