The error that cost firm Sh3.33b to AI-generated deepfake

We built machines that think faster than we do. We have not yet built ourselves to doubt more wisely than they can deceive. [iStockphoto]

"Beloved, do not believe every spirit, but test the spirits, whether they are of God." 1 John 4:1

Seeing is no longer believing. Critical thinking is the new currency.

For most of human history, the eye was the final court of appeal. If you saw it, heard it, spoke to it face to face, the case was closed. That courtroom has quietly been dissolved, and most of us are still walking into it expecting the old verdicts to hold.

Picture the world we live in today as a vast field of needles. Somewhere in it lies the one that is real. But scattered around it, indistinguishable to the naked eye, are a thousand forged to look, feel, and gleam exactly the same. Pick the one that won’t prick you.

This week's signal

In January 2024, a finance employee at the Hong Kong office of Arup, the London-based engineering firm behind landmarks such as the Sydney Opera House, received a message that appeared to come from the company's UK-based chief finance officer (CFO).

It requested a confidential transaction. Something felt off, and to his credit, the employee did the responsible thing: he asked to see his CFO confirm it directly, on a video call.

He got exactly that. The CFO appeared. Familiar colleagues joined too. They spoke, answered his questions, and walked him through the process, and, satisfied, he authorised 15 transfers totalling roughly $25.6 million (Sh3.33 billion) to five Hong Kong bank accounts.

Days later, the truth surfaced. Every person on that call had been an AI-generated deepfake, built from publicly available video and audio of the real executives, scraped quietly off the internet.

CNN, which broke the story publicly in May 2024, called it one of the most sophisticated corporate frauds of its kind. The money was never recovered.

The employee had done everything his training told him to do. He was sceptical, he verified, he insisted on face-to-face confirmation, and he still lost.

That should unsettle every leader reading this: the failure was not carelessness. It was trust, weaponised with near-perfect precision.

What it means for business

Every organisation now operates inside a trust architecture built for a world that no longer exists. A face on a screen, a voice on a call, a signature on a document none of these is proof of identity anymore, however convincing they feel in the moment.

Businesses need layered, unglamorous verification: code words for high-value transactions, callback confirmations through independently sourced numbers, and dual sign-off that cannot be short-circuited by urgency or seniority.

Fraud has always exploited hierarchy and the reluctance to question a CFO twice. Artificial Intelligence (AI) has simply made that exploit scalable. The companies that navigate this era well will not be the ones chasing the newest tools, but the ones whose leaders had the foresight to distrust the very screens they built their operations on, before a crisis forced the lesson.

What it means for policy

Governments are still legislating for a world where fraud required a forger's hand, not a forger's algorithm. Kenya, like most nations, has no clear legal framework criminalising deepfake-enabled financial fraud as a distinct offence, and no mandatory disclosure standard for synthetic media in corporate or political communication.

Policymakers must move with urgency: banks need mandated multi-factor verification for large transfers, and law enforcement needs digital forensics capacity to trace AI-generated fraud before the money crosses five accounts and vanishes, not months after.

Laws written at the pace of parliament cannot police crimes that move at the pace of a graphics card.

What it means for people

For the ordinary employee, seeing is no longer believing; the lesson from Arup is humbling. Diligence alone is no longer a shield. The instinct to trust what you see and hear, honed over a lifetime of reliable senses, is precisely what sophisticated fraud now exploits.

The new professional skill is not obedience to protocol, but the discipline to distrust confirmation itself to ask not "does this look real?" but "what would it take for this to be fake, and have I ruled that out?" That habit of mind will increasingly separate those who are protected from those who are prey.

Afterthought

We built machines that think faster than we do. We have not yet built ourselves to doubt more wisely than they can deceive. Seeing is no longer believing, convincingly, in your own CFO's voice.

The people, businesses and governments who thrive in this era will not be the ones who fear AI.

They will be the ones who sharpen their discernment fast enough to keep pace with it. Slow down before any major approval. Verify twice. "Decisions are made on the radar screen, but the future is yours."

-The author is a human-centred strategist and leadership columnist