Rendered at 07:22:30 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
dSebastien 26 minutes ago [-]
We should watermark everything we create, AI-generated or not. This might shield our content from being stolen moving forward
pr337h4m 11 hours ago [-]
We are very fortunate open source models have reached parity for virtually all non-coding use cases.
NewsaHackO 11 hours ago [-]
Do open "source" models have have this watermarking enabled? How do you know?
nonethewiser 11 hours ago [-]
It doesn't matter if they are watermarked if there is no ability to verify the watermark.
NitpickLawyer 11 hours ago [-]
The way they explain it implies they're using this at the sampler level and not trained into the weights themselves. So unless you're using an inference library that does this, the open models will not have this kind of a watermark.
> When watermarking is used, choices are still made at random, but the source of the randomness is different. Instead of using an arbitrary random number generator to pick the next word, watermaking uses the key and a few words that come before to settle what word the model should pick.
> the watermark only changes the source of the randomness used to pick among words.
bonoboTP 11 hours ago [-]
This particular watermark doesn't live in the weights, but in the sampling process, so you can turn this one off in an open source LLM.
absoluteunit1 11 hours ago [-]
> Google DeepMind tested this impact by serving a model that used watermarking to a portion of their Gemini traffic and comparing thumbs-up and thumbs-down ratings. They found no statistically significant differences from the unwatermarked model. And in a controlled study, human raters comparing watermarked and unwatermarked answers side-by-side saw no difference in quality.
For some reason I had assumed testing this would be more sophisticated than just checking the thumbs up/down stats and user "vibes"
jonas21 11 hours ago [-]
It's not just checking user thumbs up/down. As your quote says, they also did a controlled study with people rating the results. What else would you want them to do? The whole point is that it needs to introduce a detectable statistical difference, but humans should not be able to perceive it as a quality difference.
absoluteunit1 8 hours ago [-]
> human raters comparing watermarked and unwatermarked answers side-by-side saw no difference in quality
Yes - maybe saying "vibes" was minimizing the effort but what I am trying to say is that even the controlled testing is just asking users whether quality is impacted or not. Which is subjective and thats what I meant by when I said "vibes"
Don't get me wrong - I have no idea how one would go about testing this with other methods; I was just stating my assumption.
Since they rolled this out to all users I had assumed there would be other testing involved.
bonoboTP 11 hours ago [-]
> What else would you want them to do?
Retest on benchmarks whether it accomplishes tasks with the same success rates. Prose is only one thing.
Messing with the randomness may make the problem solving capabilities weaker. Probably it doesn't but this is the answer to what else I would want them to do.
cube00 11 hours ago [-]
More unannounced testing on paying customers.
cj 11 hours ago [-]
The only function of the thumbs up/down buttons are to give feedback to Google. As a user it's pretty obvious that's the purpose of the button.
cube00 11 hours ago [-]
You were still tested on and your outputs messed with even if you didn't click on either button.
11 hours ago [-]
baliex 11 hours ago [-]
Genuine question, how else would they do it? And isn’t this practice the same as basically any agile-developed SaaS?
thevinter 11 hours ago [-]
I'm not a mathematician but to me it doesn't seem so far-fetched to think that there might exist some mathematical proof that ensures the indistinguishability
antonvs 11 hours ago [-]
We don’t have the ability to do that level of analysis of natural language text mathematically.
If we did, we probably wouldn’t need LLMs in the first place, i.e. we could just generate text using explicitly programmed algorithms.
arjie 11 hours ago [-]
Interesting. Here's the section of the EU Act that mandates this:
> Providers of AI systems, including general-purpose AI systems, generating synthetic audio, image, video or text content, shall ensure that the outputs of the AI system are marked in a machine-readable format and detectable as artificially generated or manipulated. Providers shall ensure their technical solutions are effective, interoperable, robust and reliable as far as this is technically feasible, taking into account the specificities and limitations of various types of content, the costs of implementation and the generally acknowledged state of the art, as may be reflected in relevant technical standards. This obligation shall not apply to the extent the AI systems perform an assistive function for standard editing or do not substantially alter the input data provided by the deployer or the semantics thereof, or where authorised by law to detect, prevent, investigate or prosecute criminal offences.
Do i understand correctly - to check watermark you need full model weights, of all org models. So running the check is basically the same as running every model once? That's really expensive
jluysvi 11 hours ago [-]
Opus 5 must be the pilot becuase it's writing style is so grating it has to be intentional. Let's hope they make it more subtle in the future.
nonethewiser 11 hours ago [-]
It does feel like it comes from somewhere specific. I mean maybe its just a diffuse set of reasons but it feels pretty abrupt.
herywort 11 hours ago [-]
Watermarking has no impact on style of writing
jluysvi 37 minutes ago [-]
So they say haha
visiondude 11 hours ago [-]
I’d like to better understand the minimum text length to get a confident result, i would presume it would need to be quite long, perhaps > 1000 words to get an accurate result.
nonethewiser 11 hours ago [-]
From what I’ve read its closer to 100-300 words. And maybe 500 words to include info identifying the prompter.
SubiculumCode 11 hours ago [-]
How I use claude in my grant writing.
I write a rough paragraph. I invoke /concise-mode skill (a supposed instruction that Claude used for their previous concise writing style), and ask it to revise for clarity. I re-read to ensure it says what I wanted, ask for another revision with a specific request, or manually edit.
This is a productivity enhancement for me. I am not writing art. I am delivering information for my research plan. While I would not mind a flag that indicated AI assisted for clarity, I do not want to be accused of using AI-wholesale. I put a lot of work into it, and I do not want to be maligned.
johnfn 11 hours ago [-]
> We will soon be offering a watermark detection API. We’re in the process of working out the details of its implementation.
Dumb question - doesn't this defeat the purpose of a watermark? i.e., anyone who wants to avoid detection can simply run `while (has_watermark(text)) text = slightly_rewrite_with_non_anthropic_llm(text)` until it's gone? I feel I am missing the intent of the watermark if it is so easily defeated.
nonethewiser 11 hours ago [-]
Yeah kinda. You cant be confident a negative is true.
It doesn’t undermine positives though. That’s just whatever the false positive rate is.
So if it comes back as anthropic generated, it most likely is. If it comes back as non-anthropic generated, we have no idea.
If anthropic didnt make it public there would only be a narrow path for governments or something to make requests. Its kind of fucked either way.
DonsDiscountGas 11 hours ago [-]
It's completely pointless without an API, unless you're thinking the API should be private or restricted. The public needs some way of identifying the watermark.
> anybody who wants to avoid detection can just
They can just use a different LLM. By far easier and more reliable than what you're suggesting. This whole watermarking requirement is better then nothing because meant people are profoundly lazy, but yes it is not hard to work around with any effort.
johnfn 11 hours ago [-]
Sure, but I imagined it'd be something like Anthropic handing over this API only to trusted third-parties, not everyone in the world.
euio757 11 hours ago [-]
> anyone who wants to avoid detection can simply run `while (has_watermark(text)) text = slightly_rewrite_with_non_anthropic_llm(text)` until it's gone?
What do you think the pricing per call of "has_watermark(...)" will be?
An important principle:
Never pay someone to remove a problem that they themselves created
nonethewiser 11 hours ago [-]
has_watermark isnt solving the problem though. slightly_rewrite_with_non_anthropic_llm is
omoikane 11 hours ago [-]
> while (has_watermark(text)) slightly_rewrite...
I understand what you are trying to say but I am not sure any watermark detection API would definitively return a true/false answer, I would have expected something more like a numeric confidence value. I am also not sure if the API would be deterministic.
johnfn 11 hours ago [-]
I'm not sure how that changes the question -- just add `has_watermark(text) < 0.5`.
pan69 11 hours ago [-]
Not sure either why they are providing an API to detect either and what you say make sense.
However, if my understanding is correct, the reason for the watermark / detections is that its not directly aimed at end-users, but to be able from them to detect if text was produced by one of their models so they don't use it as input in training data. So, yeah, in that context, not sure why they are announcing this with an ability for anyone to detect if it was produced by one of their models. Also, they are happy to ingest text produced by models they don't own? Maybe someone with more information can elaborate?
snowe2010 11 hours ago [-]
I thought the same thing. Maybe they can restrict it so that you can’t run the same text through multiple times with only one word differences. At least from an IP perspective that would start to get really expensive to rotate through IPs to get around a block like that.
WaitWaitWha 5 hours ago [-]
Is there a method to opt out of this for non-EU people? I do not see such option in the article.
brap 11 hours ago [-]
I’m entirely confident that this technically pointless, especially when you consider open models exist.
I believe they know damn well that this will lead nowhere, and are only doing this to mitigate criticism.
bonoboTP 11 hours ago [-]
The vast majority of users are not sophisticated enough to try to erase the traces, so this will be effective for the vast majority of AI generated text that regular people are upset about. Eg. lazy student essays.
efavdb 11 hours ago [-]
The method of identifying authorship isn’t new. I guess the main new thing here is to ensure Claude has a specified word distribution so you can identify its writing.
Curious if you can prompt Claude to sue some scrambling scheme and then unscramble to defeat this.
E.g. prompt Claude to write all sentence in reverse, or swap every 2 words etc.
Then use a script to put reorder in the right ordering?
11 hours ago [-]
himata4113 11 hours ago [-]
From what I understand when you re-tokenize the output you can simply look at how often certain tokens show up and the position of them, enough of these matches would result it watermarked text.
Let's say we are at token 431 and there is 49% to generate token 1 and 51% to generate token 2, we apply bias to our token 1 which would make it win causing a repeating pattern invisible to the human eye.
Now you apply this to multiple tokens and a reversible source of random you have a pretty strong watermarking system... That is rather annoying to defeat as you essentially have to rewrite most of the text. The alternative is to use a diffusion model and spray some gaps across non-literal information such as ids, links, etc.
Can anybody take a body of text and determine if it's from Claude or not? (Or if it's AI-generated or not)?
yapfrog 11 hours ago [-]
> How do I check if a piece of text was written by Claude?
> We will soon be offering a watermark detection API. We’re in the process of working out the details of its implementation.
Determining whether it's written by Claude will be possible in the future. But unless you know the LLM being used and the company behind that LLM offers a similar API, there's no easy way to tell if it's AI generated in general.
nonethewiser 11 hours ago [-]
There will be tools to scan across a broad spectrum
jti107 11 hours ago [-]
anthropic speed running its way into irrelevance. wtf would i use AI for writing that screams AI generated especially when I'm not in the EU and open models are so good now
0gs 10 hours ago [-]
this is so funny. it's literally just the claude voice. that's not just load-bearing, it's belt and braces
lowbloodsugar 11 hours ago [-]
>But if we could see the sequence of all the moves after the game (and we knew the value of pi), we could work out whether this was a game that likely used pi to determine its moves. The game that used pi is, in a sense, “watermarked”.
Wouldn't pi contain any such sequence of numbers? Therefore you'd have to allow only certain regions of pi, and therefore, its not random anymore and we could just shortcut the whole game?
whalesalad 12 hours ago [-]
Seems pretty easy to defeat by running text output through a random reworder process that would effectively repeat the same routine on low-stakes words, replacing them with similar ones. We learned this in high school, jumping through your paper and hitting random words with the thesaurus to 'sound smarter'
yapfrog 11 hours ago [-]
That will likely make the text output worse and you'll have to fix it yourself. Regardless even if you don't fix it, at that point you're not really using Claude to generate the final output anymore.
nonethewiser 11 hours ago [-]
The point is if you can trivially change it. This remains to be seen. Its jot very useful if it only detects things that were one shots.
whalesalad 11 hours ago [-]
Seems like claude is already making it worse by choosing a random token that might not be the best one.
> When watermarking is used, choices are still made at random, but the source of the randomness is different. Instead of using an arbitrary random number generator to pick the next word, watermaking uses the key and a few words that come before to settle what word the model should pick.
> the watermark only changes the source of the randomness used to pick among words.
For some reason I had assumed testing this would be more sophisticated than just checking the thumbs up/down stats and user "vibes"
Yes - maybe saying "vibes" was minimizing the effort but what I am trying to say is that even the controlled testing is just asking users whether quality is impacted or not. Which is subjective and thats what I meant by when I said "vibes"
Don't get me wrong - I have no idea how one would go about testing this with other methods; I was just stating my assumption.
Since they rolled this out to all users I had assumed there would be other testing involved.
Retest on benchmarks whether it accomplishes tasks with the same success rates. Prose is only one thing.
Messing with the randomness may make the problem solving capabilities weaker. Probably it doesn't but this is the answer to what else I would want them to do.
If we did, we probably wouldn’t need LLMs in the first place, i.e. we could just generate text using explicitly programmed algorithms.
> Providers of AI systems, including general-purpose AI systems, generating synthetic audio, image, video or text content, shall ensure that the outputs of the AI system are marked in a machine-readable format and detectable as artificially generated or manipulated. Providers shall ensure their technical solutions are effective, interoperable, robust and reliable as far as this is technically feasible, taking into account the specificities and limitations of various types of content, the costs of implementation and the generally acknowledged state of the art, as may be reflected in relevant technical standards. This obligation shall not apply to the extent the AI systems perform an assistive function for standard editing or do not substantially alter the input data provided by the deployer or the semantics thereof, or where authorised by law to detect, prevent, investigate or prosecute criminal offences.
https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng
It definitely makes Pangram's job a bit easier.
This is a productivity enhancement for me. I am not writing art. I am delivering information for my research plan. While I would not mind a flag that indicated AI assisted for clarity, I do not want to be accused of using AI-wholesale. I put a lot of work into it, and I do not want to be maligned.
Dumb question - doesn't this defeat the purpose of a watermark? i.e., anyone who wants to avoid detection can simply run `while (has_watermark(text)) text = slightly_rewrite_with_non_anthropic_llm(text)` until it's gone? I feel I am missing the intent of the watermark if it is so easily defeated.
It doesn’t undermine positives though. That’s just whatever the false positive rate is.
So if it comes back as anthropic generated, it most likely is. If it comes back as non-anthropic generated, we have no idea.
If anthropic didnt make it public there would only be a narrow path for governments or something to make requests. Its kind of fucked either way.
> anybody who wants to avoid detection can just
They can just use a different LLM. By far easier and more reliable than what you're suggesting. This whole watermarking requirement is better then nothing because meant people are profoundly lazy, but yes it is not hard to work around with any effort.
What do you think the pricing per call of "has_watermark(...)" will be?
https://x.com/i/status/2087235360690741690
An important principle: Never pay someone to remove a problem that they themselves created
I understand what you are trying to say but I am not sure any watermark detection API would definitively return a true/false answer, I would have expected something more like a numeric confidence value. I am also not sure if the API would be deterministic.
However, if my understanding is correct, the reason for the watermark / detections is that its not directly aimed at end-users, but to be able from them to detect if text was produced by one of their models so they don't use it as input in training data. So, yeah, in that context, not sure why they are announcing this with an ability for anyone to detect if it was produced by one of their models. Also, they are happy to ingest text produced by models they don't own? Maybe someone with more information can elaborate?
I believe they know damn well that this will lead nowhere, and are only doing this to mitigate criticism.
https://towardsdatascience.com/text-classification-and-the-b...
E.g. prompt Claude to write all sentence in reverse, or swap every 2 words etc. Then use a script to put reorder in the right ordering?
Let's say we are at token 431 and there is 49% to generate token 1 and 51% to generate token 2, we apply bias to our token 1 which would make it win causing a repeating pattern invisible to the human eye.
Now you apply this to multiple tokens and a reversible source of random you have a pretty strong watermarking system... That is rather annoying to defeat as you essentially have to rewrite most of the text. The alternative is to use a diffusion model and spray some gaps across non-literal information such as ids, links, etc.
> We will soon be offering a watermark detection API. We’re in the process of working out the details of its implementation.
Determining whether it's written by Claude will be possible in the future. But unless you know the LLM being used and the company behind that LLM offers a similar API, there's no easy way to tell if it's AI generated in general.
Wouldn't pi contain any such sequence of numbers? Therefore you'd have to allow only certain regions of pi, and therefore, its not random anymore and we could just shortcut the whole game?
Is poor proofreading a form of watermarking? Clever, I suppose, but they should consider running posts through Sol for clarity.
As bad as Claude Code’s writing is, it wouldn’t make that mistake.