Research

Is Claude's Watermark Detectable? We Tested It

When Anthropic announced statistical watermarking for Claude in early 2025, it sparked intense debate about the implications for AI-generated content. Content creators, publishers, compliance teams, and researchers all had the same fundamental question: can these watermarks actually be detected in practice, and how reliably?

We set out to answer that question definitively. Over the course of three months, our research team conducted what we believe is the most comprehensive independent evaluation of Claude's watermarking system, testing detection accuracy across a wide range of content types, lengths, and real-world conditions.

Our Testing Methodology

We designed a rigorous testing protocol to evaluate detection under realistic conditions. Our dataset consisted of 10,000 text samples spanning five content categories: academic writing, marketing copy, technical documentation, creative fiction, and conversational text. For each category, we generated 1,000 samples using Claude Sonnet 4 with watermarking enabled and 1,000 samples of genuine human-written text from verified sources.

Each sample ranged from 100 to 5,000 words to test detection reliability across different text lengths. We used three detection approaches: our proprietary 47-test statistical engine, an open-source implementation of the Kirchenbauer et al. detection algorithm, and a third-party commercial detector for cross-validation.

To simulate real-world conditions, we also tested detection after common post-processing transformations: paraphrasing, translation and back-translation, format conversion, partial human editing (10%, 25%, and 50% of content modified), and copy-paste through different text editors.

Key Findings

Our results revealed several important patterns about Claude's watermark detectability:

Overall detection accuracy was 94.7% for texts over 300 words. This dropped to 82.3% for texts between 100 and 300 words, and fell to 61.8% for texts under 100 words. The watermark signal strengthens with length because more tokens provide more data points for statistical analysis. For practical purposes, we consider detection reliable only for texts exceeding 200 words.

False positive rates remained remarkably low at 0.4%. Out of 5,000 verified human-written samples, only 20 were incorrectly flagged as watermarked. Most false positives occurred in highly formulaic content such as legal boilerplate and standardized technical specifications, where human writing naturally exhibits lower entropy patterns similar to those introduced by watermarking.

Content type significantly affected detection accuracy. Technical documentation achieved the highest detection rate at 97.2%, likely because its structured nature preserves watermark patterns. Creative fiction was hardest to detect at 89.1%, possibly because the more varied token distributions in creative writing partially mask watermark signals. Marketing copy fell in between at 94.3%.

The Impact of Post-Processing

Perhaps our most practically relevant finding concerns how well watermarks survive common content transformations. Light paraphrasing, where approximately 15% of words are replaced with synonyms, reduced detection rates by only 8 percentage points on average. The watermark signal proved remarkably robust to surface-level changes.

However, more aggressive transformations had a larger impact. Translation to another language and back reduced detection to near-chance levels, confirming that the watermark is language-specific and does not survive semantic transfer through a different linguistic system. Similarly, when human editors modified more than 40% of the content, detection rates dropped below 60%.

Our aggressive removal mode, which specifically targets the statistical patterns used by Claude's watermarking system, reduced detection rates to under 8% while maintaining 95% semantic similarity with the original text. This represents a meaningful benchmark for organizations that need to manage watermark presence in their content.

Practical Implications

Based on our findings, we draw several conclusions for different stakeholder groups:

• For content teams: Claude's watermark is detectable with high reliability in standard-length content. If your organization needs to identify AI-generated text in submissions or editorial pipelines, statistical detection is a viable and accurate approach for texts over 200 words.
• For compliance officers: The low false positive rate (0.4%) makes watermark detection suitable for compliance workflows where false accusations would be problematic. However, we recommend using detection as one signal among several rather than as a sole determinant.
• For publishers: Watermarks survive light editing but not substantial rewriting. If your concern is identifying wholly AI-generated submissions, detection works well. If contributors are using AI for drafting and then extensively editing, detection becomes less reliable.
• For researchers: The length-dependent detection rates and content-type variations we observed suggest that watermark evaluation should always account for these factors. Blanket accuracy claims without context can be misleading.

Looking Ahead

AI watermarking technology continues to evolve rapidly. Anthropic has indicated plans to strengthen their watermarking system, and other providers including OpenAI and Google DeepMind are developing or refining their own approaches. We will continue updating our detection engine and publishing independent evaluations as the landscape develops.

Our full technical report, including raw data and methodology details, is available to Enterprise customers through their account dashboard. We welcome collaboration with academic researchers studying AI content provenance.