Yesterday an image generation model was released publicly with weights anyone can download and run on their own hardware. It follows a year in which text generation became a commodity API call, image tools moved into open beta, and voice cloning dropped to the price of a streaming subscription. The security consequence is not science fiction and it is not next decade. It is that every heuristic organisations have spent fifteen years teaching employees — look for poor grammar, watch for odd phrasing, be suspicious of generic greetings — expired quietly, and most awareness programmes are still teaching them.
What actually changed is the cost of a convincing message
Nothing in the attack model is new. Business email compromise, supplier impersonation and executive fraud have been the most financially damaging categories for years, with reported losses running into billions annually. What changed is the unit economics of plausibility. Personalisation at scale. Constructing a tailored pretext once required a human to research the target. A model plus a public profile plus a company website now produces five thousand individually specific messages at negligible cost. Language coverage. Attacks previously arrived in whatever language the operator spoke. Fluent output in any commercially relevant language is now free, which removes the protection that non-English-speaking markets enjoyed by accident. Iteration. Lures can be tested, scored and rewritten continuously against what gets replies, in the same way a marketing team optimises a campaign.
Three attack classes that are real today
Fluent text impersonation. Vendor payment-change requests and executive instructions written in a correct internal register, referencing real projects, arriving mid-thread. Voice cloning for authorisation. A short sample of public audio is sufficient. The technique has been used in documented frauds for several years, and the tooling has become dramatically cheaper since. Synthetic media against identity verification. Remote onboarding checks built on a document photograph and a selfie were designed against a human forger, not a generative model.
Awareness training taught people to detect authorship. Authorship is now free
That is the sentence to take to the security steering committee. If the control is a human judgment about whether a message reads like a legitimate one, the control is gone — and pushing people to look harder makes it worse, because it produces confident wrong answers rather than escalation. The replacement is verification that does not depend on the plausibility of the request at all. Bank detail changes require confirmation on a number held in the vendor master file, retrieved by someone other than the person who received the request, and recorded as a dual-authorised change. High-value instructions require an out-of-band confirmation with a pre-agreed challenge that is never sent in writing. Payment runs carry hard limits, and newly added payees face a deliberate cooling period before a first large payment can be released. None of this asks anyone to judge whether an email sounds right. That is the point.
Do not buy a detector. Buy a control
There will be a great deal of software sold this year claiming to detect machine-generated content. Treat those claims sceptically: classifiers of this kind are unreliable, degrade as models improve, and produce exactly the wrong failure mode — a confident clean verdict on a well-crafted forgery. What does help is structural. Enforce sender authentication on your own domains so impersonation of your brand becomes harder. Monitor for lookalike domain registrations. Make external-sender warnings meaningful by ensuring they are rare. And move detection effort from message content to behaviour: a payment to a new beneficiary, an unusual amount, a change of banking detail within days of an invoice, a login from an unexpected location. Behaviour is far harder to synthesise than prose.
Rewrite the awareness programme
Five changes, in order of value. Stop teaching spelling and grammar tells, and say plainly that they no longer work. Teach four pressure signatures instead: urgency, secrecy, unusual authority, and any change to payment details. These persist regardless of how well the message is written, because they are properties of the fraud rather than of the text. Simulate the channels actually being used — text messages and voice calls, not only email. Measure the reporting rate rather than the click rate. An organisation where thirty per cent report a simulation is far safer than one where five per cent click and nobody says anything. And make the cost of a false alarm visibly zero. Publicly thank the accounts clerk who delayed a legitimate payment to verify it, because every hesitation you punish buys the attacker a successful transfer later.
Practical Guidance for AI Threat Defense Strategy
- Remove authorship judgment from your controls and rebuild verification around out-of-band confirmation.
- Require dual authorisation and callback on every bank detail change, using the number already on file.
- Impose a cooling period and value cap on first payments to new beneficiaries.
- Enforce domain authentication and monitor for lookalike registrations.
- Shift monitoring from message content to payment and login behaviour.
- Update awareness content this quarter to retire the grammar tell explicitly.
- Run voice and text simulations, and report on escalation rather than clicks.
- Review remote identity verification if you onboard customers with document photographs and selfies.
The Regional Angle
Three regional considerations make this less theoretical here than the global commentary suggests. The first is that the most widely cited voice-cloning fraud in the world happened in this market. Court filings disclosed last year described a bank manager in the Emirates who authorised an extremely large transfer after a telephone call in what he believed was the voice of a company director he knew, supported by emails referencing a plausible acquisition. The lesson was not that the manager was careless. It was that every element of the verification available to him — a familiar voice, a consistent email thread, a business context that made sense — was reproducible. Any regional treasury policy that still permits telephone confirmation of a large transfer, with no separate channel and no pre-agreed challenge, is running a control that has already failed publicly within this jurisdiction. The second is that a quiet regional defence has just disappeared. For a decade, phishing in Arabic was recognisably bad — machine-translated, wrongly registered, mixing formal and colloquial forms in ways a native reader spotted immediately — and staff learned to treat poor Arabic as a warning sign. Generative models write fluent, register-appropriate Arabic, and they handle the bilingual convention of regional business correspondence, where a formal Arabic greeting opens an English message, without the awkwardness that used to give attackers away. Awareness material in this region needs rewriting more urgently than elsewhere, precisely because the local tell was stronger and people relied on it more. The third is trade documentation. This is a trading region, and enormous value moves on documents that circulate by email between traders, forwarders, banks and customs brokers — proforma invoices, delivery orders, bills of lading, certificates of origin, bank confirmations. Their authenticity cue is visual: a letterhead, a stamp, a signature, a scan that looks like every other scan. Producing convincing versions of those artefacts is now trivial, and the people receiving them are processing dozens a day under time pressure. The defence is not better inspection of the image. It is to verify document facts against a source that is not the document — confirming release instructions directly with the line or the bank, checking the consignee against the customs declaration, and refusing to act on a scanned instruction that changes a beneficiary or a delivery party without independent confirmation.
The objection worth taking seriously
The strongest objection is that this is anticipatory panic. There is very little confirmed loss attributable to generative tooling. The most damaging campaign of the last month, which reached well over a hundred organisations, used ordinary text messages and convincing fake login pages — no synthetic media required. Criminals adopt what is cheap and reliable, and commodity phishing kits already work fine against organisations whose real problem is unenforced multi-factor authentication and weak payment controls. Meanwhile, a whole product category is being marketed against a threat nobody can currently measure. All of that is fair, and the last point deserves more scepticism than it usually receives. But notice that nothing recommended above is an AI-specific purchase. Callback verification, dual authorisation on banking changes, cooling periods for new payees, behavioural monitoring and better reporting metrics are controls any competent organisation should already have, and that is the test of a sound recommendation in an uncertain threat environment: it holds regardless of whether the prediction lands. The one thing that must change immediately is the training content, because at present we are actively teaching people a detection method we know to be false. And the asymmetry is unattractive — these controls cost very little, while one successful high-value instruction costs a great deal.
Common Questions
Can we detect machine-generated text?
Not reliably, and the reliability will fall as models improve. Design controls that do not require the answer.
Is voice verification still usable at all?
As one factor among several, with a pre-agreed challenge and a callback to a number you hold. Never as the sole authorisation for a payment or a credential reset.
What is the single highest-value change?
Dual-authorised, callback-verified bank detail changes. It is unglamorous, it takes a week to implement, and it blocks the most expensive fraud category by a wide margin.
What should we expect over the next twelve months?
Expect capable image and voice models to spread fast now that weights are being published openly rather than gated behind an API. Expect voice cloning to appear in help-desk credential resets, which is a cheaper target than a treasury transfer. Expect detection products to be marketed aggressively and to underperform their demonstrations. Expect insurers and auditors to begin asking specifically how payment instructions are verified when the caller sounds correct. And expect at least one widely publicised synthetic video of a named executive, at which point this conversation stops needing to be argued.
AI Threat Defense Strategy — we retire the detection heuristics that no longer work, rebuild payment and reset verification around out-of-band confirmation, and rewrite awareness content for a market where the local warning signs have just disappeared.