What's changed in AI risk? AI risk management

AI risk is not new – we first considered this landscape with the advent of GPT two years ago. At the time, NIST had only recently published the first draft of its Risk Management Framework, which was then released in July 2024. The OpenAI System Cards were starting to unveil what red teamers found when testing their models without guardrails. The world was starting to wake up to the risks posed by AI jailbreaks.

Since then, much has changed, and some hasn’t. Following his latest post on AI legislation and compliance, Head of Service Architecture, Tristan Watkins, takes us through a modern view of AI risk for 2025.

Data loss

The simplest generative AI problem to solve may also be the most prevalent and the least specific to AI. In short, if your organisation hasn’t provided its own, private generative AI capability, there is a chance that your employees are sharing organisational data with public generative AI services. If this isn’t controlled well, you must assume it’s one of your most prominent shadow IT problems – some people now call it ‘shadow AI’. The extent to which data are lost to these services hinges on a common confusion about when an AI service does or doesn’t collect your data for training models – but there’s more to it than that.

Some services will use your content for training their models, so it could be leaked to users of those models in future. Some services explicitly choose not to do that. Others allow you to opt out, or will only do this if you’re using a free tier. In short, it depends. However, this is not the only data loss concern to be mindful of.

These data may be present in logs, so the data security of those logs should be considered, and there could be eavesdropping or broader privacy/security concerns (see the DeepSeek section below for just one example). Because this could be any generative AI service, you simply don’t know how much of these risks you’re exposed to.

This is why you need to provide a sanctioned, safe, private alternative. This problem catalysed the creation of our Private ChatGPT solution accelerator. There is also a range of trustworthy Copilots to choose from, where data privacy guarantees are strong, and your own data/AI can stay together.

Unfortunately, some organisations believe that blocking ChatGPT and Gemini will be sufficient to subdue this problem, but there will always be new services (like DeepSeek). IT teams need to satisfy this user demand internally to disincentivise users from finding their own solutions.

Readiness to bring AI generation to your own data

Coming into 2024, my big hope was that interest in connecting generative AI capabilities with an organisation’s own data would finally make authorisation work focal. Despite some helpful new technology, sadly, this hasn’t shifted much.

Authorisation remains the unloved child of the security world. A diligent IT department might solve its own authorisation problems in its administrative planes, but so much of permission management has been delegated to business departments, which has normalised an accountability void.

In most cases the delegated owners have neither the incentive, skill, nor understanding to properly chip away at overly permissive content problems. In many cases they exacerbate them. This disowned content can in turn be surfaced in search-driven AI solutions, which is the dominant pattern for bringing generative AI in touch with your own data. Generative AI appetite is the natural impetus to solve fundamental content permission problems, but we don’t see that translating into changed priorities often enough.

In the last two years, we’ve been given new technology to unearth permission problems, or sensitive information, but few approaches orientate on intersection of the two. This is where real risks are found, and where we have created our own tooling to fill this gap.

There are also great technologies like those in Entra Identity Governance, which handle management of access in a more mature way, but they typically only reach organisations with funding for premium licences.

In our view, overly permissive sensitive content is the big unsolved generative AI risk that few organisations are actively working on. This unglamorous permissions discovery and remediation work is not the kind of fundamental you want standing in the way of generative AI adoption, especially when you hope to introduce it to control Shadow AI. Yet, organisations must tackle this problem, even if they choose to launch at risk. The detail behind sizing and tackling these risks is quite a nuanced and deep topic., which we cover in greater depth in our Copilot Readiness whitepaper.

Data sovereignty

Data residency or sovereignty are messy topics. Many organisations get themselves in a twist about data sovereignty despite international agreements covering their concerns. In some cases, legal contracts may entangle an organisation in data sovereignty problems, but many legal teams have spent years removing those requirements. All told, most organisations should know whether they have these needs or not. In some cases, it may even be a matter of preference.

Sovereignty vs regionalisation in Azure OpenAI

Data sovereignty requirements can have a secondary impact in Azure OpenAI model availability. If you want Open AI models hosted in the UK, they are exclusive to Azure. This is where choice of available models, sovereignty vs regionalisation, and spending commitments vs pay-as-you-go options, require some considered thought.

For instance, we see some organisations sacrificing UK hosting preferences for EU “DataZone Provisioned-Managed” models (and the pay-as-you-go options that become available there). As it stands right now, UK-hosted model availability without up-front spending commitments is significantly behind DataZone and Global options, and spending commitments may start at a threshold beyond most budgets, so some flexible thinking may be required. This topic is too large to capture concisely here, but this additional reading is authoritative:

DeepSeek

With the launch of its R1 model, DeepSeek has grabbed international headlines, impacted stock markets, and, as a Chinese service, has instantly became politicised at a time when the TikTok ban is already focal in America. The model itself has achieved strong benchmark performance (purportedly using OpenAI’s Deliberative Alignment training technique, although they may have also distilled from OpenAI). The SaaS service comes at a low cost in exchange for fewer safety assurances. But what are these trade-offs specifically?

According to the DeepSeek Privacy Policy, we can see some fairly invasive collection practices, such as personal information, user input (text, audio, uploaded files, or other content), device/network information such as ‘keystroke patterns or rhythms’, web beacons and MAIDs – all of which can be used for R&D. What are Web Beacons and MAIDs?

Web Beacons: “very small images or small pieces of data embedded in images, also known as ‘pixel tags’ or ‘clear GIFs’, that can recognize Cookies, the time and date a page is viewed, a description of the page where the pixel tag is placed, and similar information from your computer or device.”

Mobile Advertising Identifiers (MAIDs): “Advertisers, measurement, and other partners share information with us about you and the actions you have taken outside of the Service, such as your activities on other websites and apps or in stores, including the products or services you purchased, online or in person. These partners also share information with us, such as mobile identifiers for advertising, hashed email addresses and phone numbers, and cookie identifiers, which we use to help match you and your actions outside of the Service.”

In other words, DeepSeek gathers a deep behavioural profile from what’s already known through other services (this is what Web Beacons and MAIDs give them), enriches this with what you provide, and then ‘share information collected through your use of the Service with our advertising or analytics partners’.

We need to remember that even with the first AI chatbot interactions 60 years ago, people were quickly trusting their chat partner, unveiling deeply secret, personal information. The richness of this harvested generative AI interaction is an advertising goldmine, but unfortunately, once sold, the advertising broker market is so ungoverned that the data are on the loose.

Even though a work profile is different than a personal profile, this is still a troublesome organisational risk, and there is nothing to say a user’s personal and corporate data haven’t been corelated by ad brokers. MAIDs may seem the most innocuous thing I’m highlighting here, but are potentially the most frightening, including your physical location history/patterns. On top of all this, we’ve also now seen problems with how data are safeguarded at DeepSeek.

Soon before this article was published, NowSecure released further research about the DeepSeek iOS app, which is also very concerning.

None of this is specific to China (and these concerns may be equally true of other services), but some organisations may be especially sensitive to these practices if the service provider is Chinese. Remember, this is just one example of your Shadow AI risks, and one example of where data collection meets sovereignty.

It’s worth noting that DeepSeek publishes its models through Open Source AI platforms, and it is now possible to use their R1 model more safely in Azure, but from a Shadow AI and data sovereignty perspective, it’s much more likely users would simply use their dangerous SaaS platform.

Sovereign AI investment

For organisations (especially in government) who have workloads that must be deployed on sovereign AI in the UK, there is some good news in the new AI Opportunities Action Plan, which includes provision to create a new UK Sovereign AI unit and allocation of strategically allocated sovereign compute. This is unlikely to touch much in the way of BAU AI usage, but where there are large opportunities and expensive research demands, this may become pivotal.

Agents

Agency is the AI theme of 2025. Agency simply means that something has the ability to take action. A Language Model (LM) can transform text input into a text response, and multi-modal models can take audio, visual or text input and transform that into audio, visual or text output – but an LM can’t take action on its own. That requires an application which takes instructions from both a user and the LM. For instance, an LM might translate user text into code, which gets executed by the application. This is agentic AI.

Agentic AI carries a special class of AI risks, since those risks depend on what constraints have been imposed on the application. For instance, Code Interpreter in the OpenAI Assistants API is agentic, but it can only write and execute code in a tightly controlled sandbox.

You will also find AI projects and solutions that leave these problems unsolved. They typically come with big warnings, suggest they are only fit for experimentation, etc. However, many users will dismiss this caution and may do things like allow an LM to control their web browser, which is categorically not a good idea. Consider what might happen if you have cryptocurrency or password manager extensions installed. Not all LMs have sufficient safety controls, and we really must remain cognisant that this is the sort of unconstrained risk that is extremely uncomfortable at many levels.

Copilot Studio

Although it’s been possible to build agentic generative AI solutions for years, the democratisation of AI solution building represents a distinct risk when these solutions might take action on an LM’s instruction.

Constrained agency is where some of the biggest value can be realised, and some of the biggest risks might be introduced. While that can be controlled well by builders capable of engaging with these risks, that will not be true of all users, so Copilot Studio usage warrants its own governance.

Operator and competitors

While Copilot Studio opens specific, narrow forms of agency, the new research preview of OpenAI’s Operator is general and broad. It allows GPT-4o to use its visual capabilities to operate a browser on behalf of a user.

There are guardrails, and the service interrupts action to bring a human into the loop when necessary (for instance to enter a password or payment data) – but this is still giving an LM far too much unrestricted functionality and service access. While foundation models have come a long way with safety, we really need at least the sort of safety improvements found in the o3 models to begin to feel comfortable with these risks. Although you could argue OpenAI is only introducing a service that some of its competitors already offer, and it’s still in a limited research preview, this new service is an uncharacteristically uncomfortable offering.

Multi-modal models

New modes of generative AI such as convincing audio, images and video carry their own possible rewards and risks. The most obvious of these risks pertain to deep fakes, but these modes also accentuate tensions with creatives over intellectual property rights, where there is much legal activity.

Potential infringement of these rights (intentionally or otherwise) can introduce organisational liabilities, and there are acute problems with bias in visual input/output, which are not handled as well by all models. Additionally, giving a model visual access to your screen can open unforeseen consequences – for instance if secrets are accidentally shared.

Users need to be educated about these risks. There is also a unique concern with Shadow AI for visual generation, since those models come at a higher cost, and aren’t as widely deployed. These emerging modes must be present in a risk management plan.

Improved capability in SLMs and Open Source models

Small Language Models (SLMs) such as Microsoft’s Phi-4 have become much more capable in recent years, and many open-source models (including DeepSeek’s R1 and Meta’s Llama models) come in smaller versions that sacrifice some capability to enable use on a wider range of devices, including those that don’t have powerful GPUs or NPUs. Some of the models can also be modified to remove guardrails, or are less safe to begin with.

Clearly, the ability to run models locally on a personal computer or a mobile device opens compelling scenarios, but some of these risks can be hard to control if fundamental protections aren’t as strong as they should be. And adversarial use of these models is clearly very hard to control. Given the technical complexity of running models locally today, this problem won’t be widespread, but we can expect local models to become simpler to use and more common over the next year. This can be another form of Shadow AI.

Weaponisation of cheaper and smaller models

One thing that isn’t getting much discussion outside of the security world is potential weaponisation of a powerful, cheaper model such as DeepSeek’s R1, or increasingly capable SLMs that can be run locally.

Many legitimate penetration testing technologies have been weaponised by attackers routinely over the years. Now, we see Initial Access Brokers and Ransomware as a Service providers are using generative AI with penetration testing tooling to improve vulnerability discovery times, automate reconnaissance, and modify exploitation techniques to avoid detection.

As with most things in cyber security, the solution can become a weapon, and the weapon can become a solution. As AI becomes more prevalent for attacks, we can expect defensive technology to orientate on those signals, and to use generative AI defensively to accelerate detective and protective outcomes.

Azure AI content safety and prompt shields

It’s not all new risk! Microsoft has pushed harder on Content Safety than any other organisation. In this layer atop Microsoft’s AI services, we now have detections for:

Harm categories: Hate and Fairness, Sexual, Violence and Self-Harm
Protected Material (Text or Code)
Jailbreaks
Indirect prompt injection
Groundedness

We also have Custom Categories, blocklists and support for streaming content (in preview). Collectively these content safety features supplement the guardrails built into models to offer a good (if not bullet-proof) level of protection – certainly far beyond what we had two years ago. These protective capabilities are broadly aligned with the types of risk spelled out in the NIST AI RMF and the OWASP Top 10, and aren’t limited only to the Azure OpenAI models (as we see with Azure’s new support for DeepSeek R1).

Liabilities from use of unsafe AI

Keep in mind that much of this risk landscape reduces to how you control what your users can use, can’t use, and what you encourage them to use. Rejecting all generative AI usage will only create a Shadow AI problem, carrying greater risks and potential liabilities than any compromises you might accept, since use of those unsanctioned services is completely uncontrolled.

Taking ownership of these new, rapidly evolving capabilities may feel uncomfortable, and bringing generative AI to your own data may shine a light on imperfect IT fundamentals. But this can be turned into an opportunity to gain support to work on those problems, and to gain control on your own terms. This is why our most focal risks are the same as they were two years ago.

The rest of this landscape may have introduced new problems, but controlling Shadow AI and standing up a private generative AI solution with well-managed data carries huge rewards. The unglamorous permissions remediation work will improve security fundamentals beyond AI.