Thank you Henk, a thought provoking post. Of course this was all possible before…. what’s happened is Google making it a huge amount easier to create highly credible yet entirely false imagery. It’s almost as if Google’s thinking - hey, we can make money from the disinformation game! And it won’t be the first time Google’s ethics have had a glitch. Google makes a huge amount of money from scam operators …. And does less than it should in validating exactly who is using its services to advertise. The dark world of the scammer is almost as American as Apple pie. There was this brilliant website - still up but no longer updated - where a satirical sweary robot went on forays into scam world. Saltydroid.info. A niche one, but it did cover Trump’s MLM scams… worth a check.
This is less bad than the article itself initially led me to believe. At first I thought this was letting people annotate Google Earth with fake imagery. But it's making slightly more convenient a step in a process that was already available:
Before ~2020: 1) find a Google Earth image, 2) use photoshop to alter it, 3) save the PNG, 4) disseminate the image
~2020-yesterday: 1) find a Google Earth image, 2) ask a genAI system to alter it, 3) receive a PNG, 4) disseminate the image
After: 1) find a Google Earth image, 2) ask Google Earth to alter it, 3) receive a PNG, 4) disseminate the image
Putting a nuclear plant in Iran this way has been easy since Photoshop and open satellite imagery; it has been trivial since the advent of easy image editing tools. As far as I can see this isn't going to affect any journalists or governments any more than any other doctored image would -- presumably the first thing an intelligence officer or journalist would do is check the image against the coordinates in Google Earth or some other open satellite database and see it's not there. As for more gullible actors (or gullible journalists and intelligence officers)... well, that has been a problem since the advent of image editing software.
Seems to me like a product manager had "add Gemini to Google Earth" as a marching order and this nothingburger feature is what they came up with.
Four things that are new, and none of them are about capability:
Google made a testable safety claim and it failed immediately. Their statement to me says they prevent image creation on harmful topics. Bomb crater on a Gaza hospital, refugees at the Mexican border, nuclear plant in Iran. Nothing refused, nothing softened, no warning.
Only Google can read Google's watermark. Their remedy is SynthID — ask Gemini or Lens. Hive, one of the biggest detectors outside Google, scanned my clip unprompted: 1% AI video, 0% deepfake, and 36% AI music on a silent audio track. The mark was there. Outside Google it reads as authentic, which is worse than reading as unknown.
Friction was the control. A fake US naval base ran to 950,000 views this year and was caught only because the cars sat in identical spaces in both frames. Six manual steps is six chances to get bored, and boredom is why he left the cars. Remove the last step and you don't get better fakes. You get more of them, from people who were never going to bother.
The damage that needs nobody fooled. Your model assumes the goal is deception. Increasingly it isn't. Every easy fake makes real imagery deniable — an official can now dismiss a genuine photograph of a genuine atrocity as AI. He doesn't need to make anything. He needs everyone to believe making it is trivial.
Is pretending that Claudiform words came from a mind not its own kind of misinformation?
By the way, in case you think I am basing this simply on a Pangram score, I am not. A few cases of semantic ungroundedness that jump out at me:
> A faction wants a hospital to look flattened, or intact, depending which one is holding it this week.
This is grammatically incorrect, but in a very LLM-generated way. You could easily correct it by saying "Two factions want", but that would break the rigid tricolon structure. However, there are other ways to maintain the rigid tricolon structure:
"A faction wants a hospital to look flattened or intact, depending on whether they are holding it this week."
So why would an LLM not typically output this? It has to do with the way that LLMs parse grammar. There are two sets of two here: the hypothetical faction A or faction B, and "flattened, or intact". One is interfering with the other: "flattened, or intact" looks like a list.
> One clause in there is checkable, so I checked it.
Both clauses are, in fact, checkable. But you only recounted one instance of checking to the LLM in your prompt. It doesn't think, so it doesn't think about whether the one you didn't recount is also checkable. It just deployed this rhetorical pattern, even though it's semantically ungrounded.
> The rest of the answer has a shape worth noticing.
"The rest of the answer" would be a part that comes after the section just referenced. But that's not after the section just referenced, it's before. LLMs famously are temporo-spatially ungrounded.
i agree that this is bad but this post is basically unreadable due to being saturated with ai generated drivel. if something is worth writing it’s worth writing yourself
Thank you Henk, a thought provoking post. Of course this was all possible before…. what’s happened is Google making it a huge amount easier to create highly credible yet entirely false imagery. It’s almost as if Google’s thinking - hey, we can make money from the disinformation game! And it won’t be the first time Google’s ethics have had a glitch. Google makes a huge amount of money from scam operators …. And does less than it should in validating exactly who is using its services to advertise. The dark world of the scammer is almost as American as Apple pie. There was this brilliant website - still up but no longer updated - where a satirical sweary robot went on forays into scam world. Saltydroid.info. A niche one, but it did cover Trump’s MLM scams… worth a check.
Why can't we check it against Google Earth itself? It's not editing the actual Google Earth imagery itself for others to see, right?
This is less bad than the article itself initially led me to believe. At first I thought this was letting people annotate Google Earth with fake imagery. But it's making slightly more convenient a step in a process that was already available:
Before ~2020: 1) find a Google Earth image, 2) use photoshop to alter it, 3) save the PNG, 4) disseminate the image
~2020-yesterday: 1) find a Google Earth image, 2) ask a genAI system to alter it, 3) receive a PNG, 4) disseminate the image
After: 1) find a Google Earth image, 2) ask Google Earth to alter it, 3) receive a PNG, 4) disseminate the image
Putting a nuclear plant in Iran this way has been easy since Photoshop and open satellite imagery; it has been trivial since the advent of easy image editing tools. As far as I can see this isn't going to affect any journalists or governments any more than any other doctored image would -- presumably the first thing an intelligence officer or journalist would do is check the image against the coordinates in Google Earth or some other open satellite database and see it's not there. As for more gullible actors (or gullible journalists and intelligence officers)... well, that has been a problem since the advent of image editing software.
Seems to me like a product manager had "add Gemini to Google Earth" as a marching order and this nothingburger feature is what they came up with.
Four things that are new, and none of them are about capability:
Google made a testable safety claim and it failed immediately. Their statement to me says they prevent image creation on harmful topics. Bomb crater on a Gaza hospital, refugees at the Mexican border, nuclear plant in Iran. Nothing refused, nothing softened, no warning.
Only Google can read Google's watermark. Their remedy is SynthID — ask Gemini or Lens. Hive, one of the biggest detectors outside Google, scanned my clip unprompted: 1% AI video, 0% deepfake, and 36% AI music on a silent audio track. The mark was there. Outside Google it reads as authentic, which is worse than reading as unknown.
Friction was the control. A fake US naval base ran to 950,000 views this year and was caught only because the cars sat in identical spaces in both frames. Six manual steps is six chances to get bored, and boredom is why he left the cars. Remove the last step and you don't get better fakes. You get more of them, from people who were never going to bother.
The damage that needs nobody fooled. Your model assumes the goal is deception. Increasingly it isn't. Every easy fake makes real imagery deniable — an official can now dismiss a genuine photograph of a genuine atrocity as AI. He doesn't need to make anything. He needs everyone to believe making it is trivial.
Google just made genocide denying a whole lot easier for some folks...
Quite ironic for this post to be Claudiform.
Is pretending that Claudiform words came from a mind not its own kind of misinformation?
By the way, in case you think I am basing this simply on a Pangram score, I am not. A few cases of semantic ungroundedness that jump out at me:
> A faction wants a hospital to look flattened, or intact, depending which one is holding it this week.
This is grammatically incorrect, but in a very LLM-generated way. You could easily correct it by saying "Two factions want", but that would break the rigid tricolon structure. However, there are other ways to maintain the rigid tricolon structure:
"A faction wants a hospital to look flattened or intact, depending on whether they are holding it this week."
So why would an LLM not typically output this? It has to do with the way that LLMs parse grammar. There are two sets of two here: the hypothetical faction A or faction B, and "flattened, or intact". One is interfering with the other: "flattened, or intact" looks like a list.
> One clause in there is checkable, so I checked it.
Both clauses are, in fact, checkable. But you only recounted one instance of checking to the LLM in your prompt. It doesn't think, so it doesn't think about whether the one you didn't recount is also checkable. It just deployed this rhetorical pattern, even though it's semantically ungrounded.
> The rest of the answer has a shape worth noticing.
"The rest of the answer" would be a part that comes after the section just referenced. But that's not after the section just referenced, it's before. LLMs famously are temporo-spatially ungrounded.
You should have blamed me, not AI. I'm Dutch . Thanks for the edits.
i agree that this is bad but this post is basically unreadable due to being saturated with ai generated drivel. if something is worth writing it’s worth writing yourself
Making irony work is hard because it requires a precise gap between expectation and reality.