Chatbots and RAG
What can it be made to say?
- What I look at
- Which sensitive data it can reach, and whether its guardrails hold when someone works to get around them.
- Guided by
- OWASP Top 10 for LLM Applications
What I do
AI System Testing
Most teams shipping AI features know they need testing done. Far fewer are sure what that testing should actually cover. That's fine, because working it out is the first part of the job.
Where it starts
Discovery comes first.
Every engagement starts with a working session with your team, including the architect or engineer who built the thing. Together we work out what you've actually shipped.
- Data
What data can it reach?
- Actions
What actions can it take?
- Boundaries
Where are the boundaries meant to be?
I use that session to shape the test, and your security team often comes out of it with a clearer picture too.
What I test
Two questions, depending on what you built.
A chatbot and an agent fail in different ways, so the question I'm trying to answer changes with what you've built.
Agentic systems
What can it be made to do?
- What I look at
- MCP servers, agent harnesses, and multi-agent workflows. I treat each touch point as its own attack surface.
- Guided by
- OWASP Top 10 for Agentic Applications, alongside the LLM list
One service
A lot of systems turn out to be a bit of both. A chatbot that can look up an order or open a ticket is already taking actions, and that usually comes out in discovery. Keeping this as one service means I can adjust the test to whatever we find.
Where I test
Agentic systems are non-deterministic. The same input can produce different actions on different runs, and you can't promise in advance which one you'll get.
So I test against non-production environments by default. If an agent genuinely can't take destructive action, I'm happy to talk about an exception.
What you walk away with
The report
You get the same report format I use for application pentests.
- An executive summary your leadership will read.
- Findings with a real discussion of risk and steps to reproduce.
- Remediation guidance your developers can act on.
- A narrative of how the testing actually went.
Shipping something with a model inside it? Tell me what it does, and we'll work out what the test should cover.
Tell me what you're solving