Artificial intelligence is reshaping how organisations innovate and operate, driving automation, efficiency, and faster insights. As adoption accelerates, risks around security, privacy, governance, and trust are increasing. Leading bodies such as NIST, the ICO, ENISA, and the OECD stress the need for a lifecycle-based approach to safeguarding data in the AI era.
Security and privacy must be embedded from strategy and design through data management, model development, deployment, and ongoing monitoring. Traditional cyber controls are no longer sufficient. Organisations must address AI‑specific risks including data leakage, model manipulation, prompt injection, adversarial attacks, re-identification, excessive data collection, biased outputs, limited explainability, and misuse of autonomous systems.
AI adoption grows, so do the risks associate with security, privacy, governance, and trust. Guidance from NIST, the Information Commissioner’s Office (ICO), ENISA, and the OECD consistently emphasizes that organisations need a stronger, lifecycle-based approach to secure and protect data while adapting to AI.
In the age of AI, security and privacy can no longer be treated as downstream compliance checks or standalone technical controls. They must be embedded from the outset to strategy, design, data practices, model development, deployment, monitoring, and governance. This means moving beyond traditional cyber controls and adopting a broader approach that addresses AI-specific risks such as data leakage, model manipulation, prompt injection, adversarial attacks, re-identification, over-collection of personal data, biased outputs, lack of explainability, and misuse of autonomous or agentic systems.
NIST highlights that trustworthy AI should be secure, resilient, accountable, transparent, privacy-enhanced, and managed throughout its lifecycle, while the ICO stresses accountability, fairness, lawfulness, and transparency in how AI uses personal data.
A strong security and privacy posture in AI begins with robust governance. Organisations need clear ownership, defined accountability, risk-based policies, and oversight mechanisms that ensure AI systems are developed and used responsibly. This includes establishing AI governance forums, classifying AI use cases by risk, defining acceptable data use, setting security guardrails for internal and third-party AI tools, and ensuring that legal, security, privacy, compliance, and business stakeholders are involved in decision-making. The NIST AI Risk Management Framework and OECD.AI both point to governance as the foundation for trustworthy AI adoption.
Data Privacy is especially critical because AI systems depend heavily on data regularly on large volumes of personal data, sensitive, behavioural, or inferred data. In this context, privacy is not just about compliance with data protection laws, it is about maintaining trust, minimising harm, and ensuring that innovation does not come at the expense of individual rights. The ICO makes clear that organisations using AI must address lawfulness, fairness, transparency, purpose limitation, data minimisation, and accountability, while also carrying out robust impact assessments where AI may pose higher risks.
Key Risks and Challenges in AI Security and Privacy
As AI adoption scales, organisations face a wider and more complex risk profile than traditional digital transformation programmes. Common challenges include prompt injection, sensitive information disclosure, data and model poisoning, supply chain vulnerabilities, improper output handling, excessive agency in agentic systems, vector and embedding weaknesses, misinformation or hallucination, and unbounded consumption or denial of service. These issues are now widely recognised in the 2025 OWASP Top 10 for LLM Applications.
At the same time, privacy risks continue to grow through over-collection of personal data, unlawful reuse of data for model training, weak transparency, re-identification, opaque automated decision-making, and inadequate human oversight. Broader AI threat models captured by MITRE ATLAS further show that attacks can occur across the full AI lifecycle, from data acquisition and model development to deployment, integration, and agentic execution.
To manage these risks, organisations should treat AI as part of their core security architecture. This includes securing datasets, validating provenance, protecting model integrity, enforcing strict identity and access management, segmenting environments, using red teaming and adversarial testing, validating outputs, applying content and prompt controls, monitoring abuse, logging decisions, and assessing third-party AI providers for supply chain risk. ENISA and NIST both support a layered, lifecycle-based approach that combines foundational cybersecurity with AI-specific safeguards.
Solutions to Address AI Security and Privacy Risks
Addressing these risks requires organisations to combine governance, technical safeguards, operational discipline, and workforce awareness. A practical response starts with an enterprise AI governance model that defines ownership, risk classification, acceptable use, approval routes, and monitoring expectations for all internal and third-party AI systems.
- Adopt Privacy by Design and by Default: Minimise personal data, justify data use, restrict retention, apply pseudonymisation or anonymisation where possible, and build transparency into user journeys.
- Establish strong AI Governance: Create a cross-functional governance forum, classify use cases by risk, maintain an inventory of AI systems, and define clear accountability for model owners, data owners, and control functions.
- Secure the AI Lifecycle: Protect training and inference data, validate provenance, control access to models and prompts, secure APIs and plugins, segment environments, and enforce least privilege for agentic actions.
- Strengthen Output Controls: Validate outputs before downstream execution, apply filtering and policy checks, constrain high-risk actions & use human decision for sensitive/material decisions.
- Manage Supplier and Supply Chain Risk: Assess third-party models, datasets, and AI service providers for security, privacy, contractual safeguards, resilience & incident response capability.
- Enable Observability and Incident Response: Log prompts, outputs, tool actions, and model behaviour where appropriate, define escalation paths, and extend cyber incident response plans to cover AI-specific scenarios.
- Train Employees and Developers: Build awareness of safe prompting, acceptable use, privacy obligations, data handling, and the risks of overreliance on AI-generated outputs.
This is why organisations need to move from a compliance mindset to a culture mindset. Shaping security and privacy in the age of AI is not solely the responsibility of cyber teams, privacy officers, or legal functions. It requires organisational culture including leadership commitment, clear principles, practical guardrails, workforce awareness, responsible innovation practices, and ongoing oversight.
Industry Best Practices:
Leading organisations are increasingly aligning their AI programmes with recognised frameworks and practical industry guidance. Some of the best practices to adapt and evolve are –
- Embed AI risk management into enterprise governance using NIST AI Risk Management Function.
- Use data protection by design & default from earliest stages of solution design and procurement.
- Apply defence-in-depth across the AI stack, including infrastructure, data pipelines, models, embeddings, prompts, APIs, and agents.
- Maintain current inventory of AI systems, associated datasets, third-party dependencies, and business owners.
- Conduct DPIAs, risk assessment, and threat modelling for high-risk AI use cases before deployment
- Test continuously using adversarial techniques, red teaming, abuse case simulations, and control effectiveness reviews.
- Establish clear policies for acceptable AI use, data handling, transparency, record-keeping, and incident reporting.
- Monitor for drift, misuse, privacy leakage & control failure to update and safeguards as risks evolve.
- Create a culture of trustworthy AI by training leaders, builders, and users on security, privacy, bias, and responsible use.
Looking ahead, the organisations that succeed with AI will be those that build resilience, responsibility, and trust at scale. Security and privacy must evolve alongside AI capabilities becoming more proactive, measurable, and deeply integrated into governance, product development, third-party oversight, and business strategy. The challenge is not only to use AI powerfully, but to use it safely, lawfully, and credibly. In that sense, shaping security and privacy in the age of AI is not merely a technical requirement, it is a leadership imperative and a defining factor in sustainable AI adoption.
Kavitha Srinivasulu is a senior cyber risk and resilience executive with over 22 years of global leadership experience advising Boards and Executive Committees across Financial Services, Healthcare, Retail, Technology, and regulated industries. Delivered and led large-scale, regulator-driven cybersecurity, AI-driven, PCI, and SOC transformations for Tier-1 banks, global healthcare organisations, and highly regulated enterprises operating across the UK, EU, USA, APAC, and ANZ. Trusted advisor to Boards, C-suite, regulators, and global enterprises, consistently delivering resilient, compliant, and scalable cyber operating models.
Disclaimer:
The views and opinions expressed by Kavitha in this article are solely her own and do not represent the views of her company or her customers.