Tackling Phishing and Cyber Fraud in the Age of AI

Why good governance matters more than ever

AI is changing the cyber threat landscape. Phishing attacks are becoming more convincing, fraud more personalised and new technologies are introducing risks that many organisations are still learning to manage.

But while the threats continue to evolve, the fundamentals of good cyber security remain the same.

This was the central theme of a webinar conversation I joined alongside Alex Thomas, Chief Revenue Officer at Boxphish, hosted by BGF.

Together, we explored how AI is changing the risks emerging within organisations and the threats coming at them, and what businesses can do to strengthen their resilience.

Here are some of the key insights from our discussion.

AI is already in your business. But what about its governance?

AI governance is quickly becoming a business priority, driven by growing customer expectations, emerging regulation and increasing adoption.

According to A-LIGN's 2026 Global Compliance Benchmark Report:

• 4 in 5 organisations now face customer enquiries about AI risk management
• ‍47% expect to be impacted by the EU AIAct‍
• 77% plan to pursue an AI certification within the next 12 months

Yet governance is struggling to keep pace.

This year’s UK Government's Cyber Security Breaches Survey found that 31% of UK businesses are using, adopting or actively considering AI. Of those, 76% have no process in place to manage the associated risks.

And that's before considering the AI tools employees may be using without their organisation's knowledge or approval.

The question isn't simply whether employees have received sufficient training.

It's whether organisations know which AI tools their people are using, what information they're sharing and who is responsible for managing the risks.

Without that visibility, training alone can only go so far.

The AI governance gap is widening, not closing

IBM's Cost of a Data Breach 2026 research, covering 602 breached organisations, highlighted several concerning trends:

• 68% had no policing governing AI use or detecting unsanctioned tools (up from 63% in 2025)
• Just 38% required IT approval before deploying AI tools (down from 45%)
• 92% of AI-related breaches involved inadequate AI access controls

Compromised APIs, exposed applications, cloud misconfigurations and insufficient access controls are familiar security challenges.

AI hasn't fundamentally changed what good cyber security looks like.

Instead, it's made established security and governance practices more important than ever.

Where AI risk is landing: The 63% problem

One point I want to emphasise is that technology alone cannot address every compliance or security risk.

Take ISO 27001.

Of its 93 Annex A controls, just 34 are technological. The remaining 59, or 63%, cover organisational, people and physical security.

This includes policies, employee awareness, acceptable use, access responsibilities and supplier due diligence.

Technology can support these activities but it cannot replace the ownership, judgement and accountability required to manage them effectively.

And it's precisely within these areas that many AI-related risks are emerging.

An organisation might have sophisticated security monitoring, but does it know whether employees are sharing sensitive information with unapproved AI tools? Are suppliers following appropriate security procedures? Can employees recognise increasingly sophisticated impersonation attempts?

The external threat: Phishing remains the biggest risk

Having explored the risks developing inside organisations, Alex turned the conversation towards the external threats AI is amplifying.

The Government's latest Cyber Security Breaches Survey found that 43% of UK businesses experienced a cyber security breach or attack in the previous 12 months.

Among organisations identifying an attack, phishing remained the most common threat, reported by 88% of businesses and 87% of charities.

The common factor? People.

Human behaviour is varied and unpredictable, making it an attractive target for cybercriminals. Attackers exploit trust, urgency and everyday business processes to bypass safeguards that might otherwise be technically sound.

And AI is making those attempts faster, more personalised and increasingly difficult to recognise.

Four real-world examples of human risk

Alex shared four examples demonstrating how attackers exploit people, processes and trusted relationships. Some may be familiar.

1. Co-op: Attackers associated with Scattered Spider reportedly impersonated employees and contacted the IT help desk to reset passwords, exploiting identity verification processes
2. Marks & Spencer: Attackers reportedly impersonated an employee to persuade a third-party IT help desk to reset their password, providing a route into M&S' systems
3. Arup: A finance employee attended a video call featuring what appeared to be the company's CFO and colleagues. In reality, the participants were AI-generated deepfakes, demonstrating how social engineering has moved beyond email
4. Department for Education: A social engineering attack targeting the department's help desk resulted in more than 607,000 records being stolen and published online

These incidents differ in their methods, but demonstrate a common weakness: even sophisticated security arrangements can be undermined when human processes and responsibilities aren't sufficiently protected.

A safeguard nobody owns isn't a safeguard

Security controls are only effective when someone understands them, owns them and knows what to do when something goes wrong.

Yet the figures suggest many organisations lack that accountability.

• Only 19% of organisations in IBM's research had governance and security teams coordinating
• Just 25% of UK businesses have a formal incident response plan
• Only 15% review the risks posed by their immediate suppliers

Think back to Co-op and M&S. Both illustrate how vulnerabilities in third-party processes can have serious consequences.

Having MFA, payment verification procedures and incident response plans is important. But without clear ownership, their effectiveness is limited.

My view is simple: if you can't explain a control without the tool in front of you, you don't have a control. You have a subscription.

Our customer Net-Defence reduced the manual effort behind its compliance programme by 50%. Not by removing human judgement, but by making responsibilities and ongoing compliance activities easier to manage.

Good governance isn't about adding unnecessary processes. It's about ensuring the right people have the visibility and ownership to act when it matters.

AI is accelerating human risk

Alex also highlighted how quickly AI-enabled fraud is developing.

Figures referenced in a recent Home Office call for evidence showed a 469% increase in reports of suspected AI-enabled fraud between 2023 and 2025, with investment fraud the most commonly reported category.

The familiar warning signs of phishing are becoming less reliable.

AI enables attackers to:

• Hyper-personalise phishing: Using publicly available information to create highly relevant messages
• Generate convincing communications: Removing the spelling mistakes and generic messaging traditionally associated with phishing
• Impersonate trusted individuals: Using voice and video deepfakes to imitate colleagues, suppliers and senior leaders
• Scale attacks rapidly: Automating research, personalisation and deployment

As attacks become more convincing, employees need more than an ability to spot suspicious emails. They need clear processes to verify requests and respond appropriately.

Protecting your people: Awareness needs to become everyday practice

Cyber security awareness shouldn't be limited to annual training.

Instead, organisations should build consistent habits through regular training, realistic phishing simulations and a culture where employees feel comfortable reporting suspicious activity without fear of blame.

These habits should be supported by practical safeguards, including MFA, secondary payment verification and a rehearsed incident response process.

Even well-trained employees make mistakes. The objective is to ensure one mistake doesn't escalate into a serious incident.

The capacity problem: Governance teams are already stretched

Introducing new governance requirements is easier said than done when teams are already managing significant compliance workloads.

AI governance is adding another responsibility to teams already balancing multiple frameworks, audits and operational priorities.

But the answer shouldn't be another isolated compliance programme.

Organisations should look at how existing policies, controls and evidence can support emerging requirements.

Our customer Advantex, for example, reduced audit prep time by 30–40% by bringing its compliance activities together in one system.

The underlying work didn't disappear. It simply stopped being duplicated.

Three practical steps to strengthen your AI governance

Of course, platforms like Hicomply are here to help manage the complexity that often comes with managing multiple compliance programmes. But getting started doesn't need significant investment. Here are some immediate (and free!) actions to consider.

1. Ask your team which AI tools they've used recently.

Have an open conversation about the tools being used across departments, including those that haven't been formally approved. Understanding what's happening is the first step towards managing the risk.

2. Review your acceptable use policy.

Does it address AI? Does it explain what information employees can share, which tools are approved and what safeguards are expected? If your policy predates widespread AI adoption, it's time for a review.

3. Assign clear ownership for AI risk.

Establish who is responsible for overseeing AI governance,reviewing risks and ensuring appropriate controls are in place. Accountability needs to sit with a named individual, supported by relevant teams across the business.

None of these actions requires a platform. But all three help establish the foundations for managing AI responsibly.

The fundamentals haven't changed. The urgency has.

My biggest takeaway from the session is that organisations shouldn't view AI as an entirely separate cyber security challenge.

Yes, it's accelerating existing threats and introducing new risks. But the response still comes back to the fundamentals: understanding your exposure, establishing effective controls, educating your people and making accountability clear.

AI can strengthen cyber defence and reduce the administrative burden of compliance. But automation should support human judgement, not replace it.

Ultimately, stronger cyber resilience requires a whole-business approach to governance, not simply another technology investment.

Looking to strengthen your approach to AI governance?

Explore our Ultimate ISO42001 Checklist to understand the foundations of an effective AI management system, or discover how Hicomply helps organisations bring governance, risk management and compliance together.

A big thank you to Alex and the Boxphish team for sharing their expertise, to BGF for bringing us together for an incredibly important and insightful conversation.