This article is a companion to the deck used in its presentation: our expectations of software, how OWASP remains our anchor for risk, what AI brings to the picture, and how teams should look to embrace AI while not abdicating accountability. Let’s get the ball rolling!
Software Expectations
When we think about production software, we often evaluate it based on three key qualities:
- Scalable: It should grow seamlessly with users, data, and teams, avoiding any shortcuts that could lead to issues down the line.
- Maintainable: Clear boundaries and ownership are essential, making it easy to review changes and understand the code.
- Reliable: The software must behave predictably, even under stress or failure.
These qualities are interconnected; a weakness in one area can lead to failures in another. For instance, if a team is stuck with unmaintainable code, they’ll struggle to identify bugs and ensure the product is reliable.
Similarly, a system that performs well under normal conditions but falters during peak usage is ultimately unreliable.
The Gold Standard: Five 9s
Achieving 99.999% availability means only about five minutes of unplanned downtime each year, a benchmark many teams strive for when defining reliability.
- Five 9s isn’t just about having redundant servers; it’s about minimising bad changes, detecting issues quickly, and recovering efficiently.
- Security incidents count as downtime: breaches, authentication outages, and emergency patches all consume valuable resources.
- AI can increase the volume of changes; without proper checks, this speed can lead to costly mistakes.
Very few teams actually reach true five-9s, and the investment required to get there is significant. The real question is whether pursuing that last fraction of a percent is worth it.
If most users already enjoy a fast and reliable experience, should we focus on perfecting every edge case, or should we prioritise improving the overall user journey and addressing the factors that erode trust? While five 9s is a great goal, product and risk teams must determine where the returns diminish.
Security: A Pillar of Reliability
Reliability and security are intertwined; they can either support or undermine one another. A reliable system that neglects security can be compromised, leading to downtime. On the flip side, brittle security measures can create their own outages. When authorisation fails, it can either expose sensitive data or lock out legitimate users.
To achieve durable reliability, security must be integrated into the design process. This includes defining who can act on what, managing untrusted inputs, safeguarding secrets, and ensuring thorough reviews of changes, especially those generated by AI.
Teams that view security as a guiding principle for behaviour in adversarial conditions are more likely to deliver software that remains operational for the right reasons.
Measuring a Secured System: Establishing a Benchmark
The Open Web Application Security Project (OWASP) provides invaluable guidance on application security, helping developers, architects, and reviewers communicate effectively about risks.
The OWASP Top 10:2025 outlines the most critical web application risks, offering a shared language for teams to address vulnerabilities.
How AI-Backed Development Has Changed the Game
AI introduces new challenges, such as:
- Hallucinated dependencies: Fake or misleading package suggestions.
- Outdated dependencies with vulnerabilities: Code that relies on libraries with known issues.
- Indirect prompt injection: Manipulation through README files or web pages.
- Agent runtime sandboxing: Agents operating with full development privileges unless properly contained.
- Test fabrication: Passing tests that haven’t been genuinely verified.
AI has not eliminated OWASP risks; it can amplify them, especially as delivery speeds increase.
Using AI Effectively While Maintaining Accountability
To harness AI for secure delivery, it must be integrated into existing standards rather than treated as an afterthought.
- Anchor on OWASP guidelines and establish clear team rules.
- Define a workflow: Classify data, prompt with security requirements, and maintain rigorous CI gates.
- Monitor AI access: Limit what tools can access sensitive information.
- Require human oversight for critical actions.
- Leverage AI to draft tests and scenarios, but always verify them.
Accountability is key. Every AI-assisted change should have a designated human owner who can explain and justify it. Keeping a human in the loop for significant actions ensures that we don’t lose sight of responsibility in our fast-evolving landscape.
The Gold Standard, five 9s
And if the user is king, and for most users today we deliver a small, fast, reliable experience, then is it better work or better service for that user to work towards the ideal of "catch 'em all" perfection of every percentile and edge case compared with delivering a better median journey and fixing those things that are truly chipping away at trust?
Five 9s is a good north star, but the product teams and risk teams have to know when diminishing returns start to kick in.
If you're the king, and 95%+ of your users have a neat and reliable user experience today, do you spend your time perfecting your app for every single percentile and edge case, or do you "catch 'em all" and work on improving the median journey and fixing things that are really damaging trust?
Five 9s is a good north star, but the product teams and risk teams have to know when diminishing returns start to kick in.
If you're the king, and 95%+ of your users have a neat and reliable user experience today, do you spend your time perfecting your app for every single percentile and edge case, or do you "catch 'em all" and work on improving the median journey and fixing things that are really damaging trust?
Security is fundamental to reliability
Reliability and security are not separate backlogs, they reinforce or undermine each other. A reliable system that ignores security can be taken offline by abuse, credential theft, or a supply chain compromise; that is downtime, by another name.
Conversely, security controls that are brittle or untested become their own source of outages, locked accounts, failed auth, and emergency rollbacks. When authorisation fails open, data leaks; when it fails closed under load, legitimate users cannot work.
Injection and misconfiguration do not only “break security”; they corrupt behaviour and corrupt trust in predictable outcomes.
Observability, testing, and design discipline still matter but they only produce durable reliability when security is part of the same design: who may act on what, what untrusted input can reach, how secrets and dependencies are guarded, and how changes (including AI-generated diffs) are reviewed before merge.
Teams that treat security as the guardrail for correct behaviour under adversarial conditions ship software that stays up for the right reasons.
How do we measure a secured system? We need a benchmark
The Open Web Application Security Project (OWASP) is a nonprofit foundation that publishes community-driven guidance on application security. Its Top 10 lists the most critical risk categories so developers, architects, and reviewers can speak the same language instead of reinventing checklists per project.
OWASP Top 10: 2025 gives that shared language for the most critical web application risks.
A01 Broken Access Control Users act outside their intended permissions.
A02 Security Misconfiguration Unsafe defaults, exposed admin, missing hardening.
A03 Software Supply Chain Failures Compromised dependencies, build, or distribution.
A04 Cryptographic Failures Weak or missing crypto for sensitive data in transit or at rest.
A05 Injection Untrusted data executed as code or queries (SQL, XSS, etc.).
A06 Insecure Design Missing threat modeling and security in the design phase.
A07 Authentication Failures Broken login, session, or credential recovery.
A08 Software & Data Integrity Failures Updates, CI/CD, or data trusted without verification.
A09 Logging & Alerting Failures Attacks and abuse go unseen or unresponded to.
A10 Mishandling of Exceptional Conditions Errors and edge cases leak data or leave the system unsafe.
How AI-backed development changed the game
From OWASP’s Secure Coding with AI cheat sheet main sections in one line each:
1. Hallucinated dependencies: fake or typosquatted packages in suggestions.
2. Outdated dependencies with CVEs: “working” code on vulnerable libraries.
3. Indirect prompt injection: READMEs, issues, and web pages steer the agent.
4. MCP and tool security: overpowered or malicious tools and servers.
5. Agent runtime sandboxing: agents run with full dev power unless contained.
6. Rules files & persistent steering: hidden instructions in repo config files.
7. Out-of-scope edits: changes outside the task buried in large diffs.
8. Test fabrication & deletion: fake passes or removed failing security tests.
9. Prompt context leakage: secrets and IP sent to model providers.
10. Prompt-to-code supply chain: untrusted instructions become shipped code.
11. CI/CD agents & confused deputy: PR bots with org secrets weaponised.
12. Markdown, link & unicode injection: deceptive content manipulates agents.
13. Multi-agent propagation: bad context spreads across handoffs.
Different, but not that different
AI-specific risks map back to classic OWASP themes:
A01 Access control; Agent sandboxing; CI/CD agents
A03 Injection Indirect prompt injection; markdown/unicode injection
A04 Insecure design Test fabrication; human accountability
A05 Misconfiguration Rules files; out-of-scope edits
A06 Vulnerable components Hallucinated & outdated dependencies
A07 Auth failures MCP and tool security
A08 Integrity failures Prompt-to-code supply chain
A09 Logging failures Context leakage; CI/CD agents
A10 SSRF Indirect injection; MCP/tool fetches
Same fundamentals, new places they show up while we build with AI.
Has AI made us less security-conscious?
Several patterns show up when delivery accelerates:
Review fatigue more plausible code than reviewers can deeply read.
• Complacency: “it compiles and tests pass” mistaken for secure.
• Overconfidence: research suggests developers with AI assistants often believe their code is safer than it is.
• Speed without ownership: “the AI wrote it” is not an accountability model.
• Vibe coding: accepting output without understanding trust boundaries.
AI did not remove OWASP risks, it can amplify them at higher velocity.
Using AI well, and keeping humans accountable
AI can accelerate secure delivery when it sits inside the same standards as everything else, not beside them.
- Anchor on OWASP Top 10 + Secure Coding with AI + team rules (Cursor rules, DevSecOps guidance).
- Define an AI workflow: classify data → prompt with security requirements → small diffs → same CI gates (SAST, secrets, SCA).
- Monitor AI access: approved tools, no production secrets in prompts, scoped agent tokens, egress limits.
- Require human confirmation for shell, push, migrate, deploy.
- Use AI to draft tests and threat scenarios, then verify them, not rubber-stamp them.
Accountability is what makes those practices stick. Every AI-assisted change needs a named human owner who can explain and defend it. Keep a human in the loop for irreversible or high-impact actions.
Never waive human review on auth, crypto, IAM, CI/CD, or data-access paths because “the agent did it.” Monitor what tools can read, run, and exfiltrate. The model does not accept liability; the team that merges does.
Summary
As we navigate the complexities of AI-driven software development, it is essential to prioritise security alongside reliability and maintainability.
By integrating OWASP guidelines and fostering a culture of accountability, teams can harness the power of AI while safeguarding their systems against emerging vulnerabilities. The journey towards secure software is ongoing, and staying informed is crucial.
For more insights and resources on enhancing your software security practices, we invite you to visit the Trinity Software Center. Explore our comprehensive materials and join a community dedicated to advancing secure software development. Your journey towards excellence in software security starts here!