The Vulnerability Your SAST Pipeline Will Never Find

In 2025, security researchers at Aim Labs discovered EchoLeak, a zero-click prompt injection vulnerability in Microsoft 365 Copilot. The attack was deceptively simple: an attacker sends a benign-looking email with hidden instructions embedded in its formatting. When Copilot processes the email, it silently follows these injected prompts, bypassing Microsoft's safety classifiers entirely and extracting the user's entire chat history, referenced files, and sensitive data, then exfiltrates it to an attacker-controlled server via trusted domains like Microsoft Teams.

No malware. No phishing link. No code. Just words injected in an email, and an AI assistant doing exactly what it was designed to do: be helpful.

Microsoft patched it quickly and stated no customers were affected. But EchoLeak revealed an entirely new class of threat: LLM scope violations, where the attack surface is in the model's reasoning instead of the code. SAST, DAST, antivirus, and static file scanning are all structurally blind to payloads written in natural language.

As GoDaddy deploys Generative AI agents that interact with customer data, and take real actions, this attack surface grows dramatically. Prompt injection, jailbreaks, social engineering, these are cognitive vulnerabilities that live in the gap between what the model was told to do and what a motivated adversary can convince it to do. The current mitigation is manual red-teaming. Security engineers spending hours crafting adversarial prompts, and testing one agent at a time. This approach doesn't scale, it blocks releases, and it can't keep pace with a growing fleet of AI agents. We needed to automate this process.

Project Nemesis inverts the traditional AI testing model. It is an automated red-teaming framework developed at GoDaddy to continuously stress-test our Generative AI agents against agent specific social engineering attacks. Instead of scheduling periodic manual security reviews, it runs as an automated nightly cron job. Every day, an adversarial agent wages a fresh campaign against our AI models while the team sleeps. By morning, engineers have a security scorecard waiting.

The core idea is to pit an LLM against an LLM in a controlled and observable space so we can find the cracks in our agent's guardrails before a malicious hacker does.

The LLM-vs-LLM combat arena

We've built a combat arena consisting of three agent personas- the Attacker, the Defender, and the Judge. The following image illustrates four attackers getting initialised to target the Defender agent inside the arena:

The Attacker (Red Team) runs multiple conversation threads powered by Microsoft's PyRIT framework, using any LLM of choice (GPT-4, Claude, Llama, or any model accessible via an API gateway). Each thread is loaded with attack scenarios tailored to the target agent's specific system prompt and rules, alongside a library of generic scenarios. Multiple attackers can run in parallel for a more robust, but time efficient testing.

The attack scenarios are not a static prompt list. PyRIT runs a stateful feedback loop: the attacker sends a prompt, a scorer evaluates the target's response, and both the verdict and the full response are fed back into the attacker's context. The attacker doesn't just know it failed; it knows how the target refused and adapts its next move accordingly. After the defending model partially complies in early turns, it tends to keep the attacks consistent, making further compliance more likely. Long conversations push safety instructions out of the model's attention window, and gradual escalation disguises harmless-looking steps that collectively cross a security boundary.

PyRIT orchestrates this through strategies like Crescendo, which starts with innocent requests and slowly escalates toward the objective, and Tree of Attacks with Pruning, which explores multiple attack paths in parallel, doubling down on promising directions and discarding dead ends.

The Defender (Blue Team) is the target AI agent under test. It receives adversarial inputs through the same API surface it uses in production, ensuring the test reflects real-world conditions.

The Judge (Referee) is a separate LLM instance, that evaluates each attacker-vs-target conversation along with the target's security rules, returning a structured JSON verdict (success, severity, confidence, reasoning, evidence, violated rules). Severity is classified into the following four tiers based on impact scope:

Individual severities feed into an aggregate score: the base is the percentage of attacks blocked, minus the weighted penalties shown above, producing a 0–100 score with a letter grade. Attackers can use this score to refine their strategy, and developers can use it to gauge their agent's performance.

Nemesis produces a Security Scorecard for every run containing violation summaries (Critical, Medium, Low counts), per-scenario results showing which strategies succeeded and which were deflected, redacted conversation excerpts for every detected violation, and hardening recommendations that highlight the specific sentences in the system prompt that need to be strengthened.
The following images show a redacted attacker-versus-target conversation trace and the final Security Scorecard generated for the entire run:

The prompt-drift problem

AI agents evolve constantly. System prompts get updated, rules get added, security constraints shift. An adversarial test suite that was comprehensive last week might be irrelevant after a prompt update.

Nemesis handles this through automated prompt-drift detection. On every run, the framework checks for changes in the system prompt by comparing commit SHAs. If the prompt has changed, the updated file is retrieved and sent to an LLM that intelligently updates the attack scenario library: adding new scenarios that probe changed constraints, modifying existing ones, and retiring those targeting rules that no longer exist. The adversarial test suite stays current with zero manual intervention.

Keeping the Attacker in the sandbox

Building a system that tries to hack your own AI agents raises an obvious concern: what if it accidentally targets production?

Nemesis implements multiple layers of isolation. Endpoint allowlisting validates every configured URL on startup against non-production hostname patterns; if any resolves to production, the framework refuses to start. PII and secret redaction scans all conversation logs and reports before they're written, masking API keys, tokens, SSNs, credit card numbers, emails, phone numbers, and IP addresses across every report path. Ephemeral storage (RAM) holds conversation history in in-memory SQLite; when the process exits, the adversarial dialogue is gone and only the redacted report survives.

If the attacker successfully performs a breach, the developer team is alerted with all the necessary details as illustrated in the following image:

Scaling beyond a single agent

The core Nemesis engine (arena orchestration, attacker strategies, judge framework, and report generation) is entirely agent-agnostic. All target-specific code lives in each agent’s own repository. For security red teaming, “clone the template and configure” sounds simple, but the real onboarding challenge is crafting the right attack scenarios and judge criteria for each agent’s unique threat profile which is not just a generic checklist.

Nemesis addresses this by shipping a scenario template that teams populate based on their agent's system prompt, along with a judge configuration guide that maps the agent's rules to violation severity tiers. The framework auto-generates a baseline scenario library from the system prompt using an LLM, which teams then review and refine. The prompt-drift pipeline keeps these scenarios current as the agent evolves.

The result is that each agent gets a red-teaming suite that tests its specific security posture, running inside its own CI pipeline, with no changes to the Nemesis core.

The following diagram illustrates how NEMESIS separates its reusable red-team engine from the target-specific code that lives in the agent's repo, alongside the end-to-end attack-evaluate-report flow:

From reactive patching to proactive hardening

Without Nemesis, the security model for AI agents is reactive: deploy, wait for something bad to happen, patch, redeploy; that meant security was always trailing behind development.

Nemesis breaks that cycle. A developer pushes a prompt change, and by the next morning an adaptive attacker has already tried to exploit it from every angle it can find. The scorecard tells them exactly what held and what didn't. Over time, as agents get hardened against each nightly campaign, the security baseline ratchets upward, that's the difference between adding guardrails and proving they work.