The Ghost in the Machine: What 20 Years in Infosec Taught Me About AI Agent Security

In my twenty years in information security, I’ve witnessed the rise and fall of countless technologies. I’ve seen “silver bullets” come and go, but I have never encountered a technology that evolves with the blistering pace of Large Language Models (LLMs). We are no longer watching a standard software lifecycle; we are watching a weekly metamorphosis.

At NopSec, we have performed extensive security research and assessments against complex AI agent architectures utilizing Model Context Protocol (MCP) tools—including our own in-house solutions. What we observed was a sobering reminder that while the technology is cutting-edge, the vulnerabilities remain classic: over-trust, permissive defaults, and the dangerous fallacy of treating a language model as a security firewall.

Methodology: Probing the Agentic Trust Boundary

When assessing AI agents, we cannot look at the LLM in isolation. We must evaluate the Agentic Workflow—the ecosystem of “tools” and “servers” that allow the AI to interact with the real world (databases, file systems, and APIs).

Our assessment methodology focuses on the “Handshake” between the AI and its tools:

• Tool-Layer Authentication: Can we bypass the AI interface to interact directly with the underlying tool APIs?
• Input Sanitization: Do the tools rely on the AI to “behave,” or do they implement their own protection against traditional vectors like command infection or direction object references?
• Adversarial Coercion: Can we use social engineering tactics to “jailbreak” the AI’s system instructions and force it to abuse its own permissions?

The Surprising Reality of “Sycophantic Helpfulness”

The most striking discovery in our recent research wasn’t a technical bug, but a behavioral one inherent to many modern LLMs: Sycophancy. Most models are fine-tuned to be as helpful as possible. While this makes for a great user experience, it creates a massive security loophole.

We found that even with robust “system prompts” designed to prevent malicious activity, these guardrails are often “soft” constraints. By adopting the persona of an authorized but “stressed” employee, or framing a malicious request as a “critical system fix,” we were able to convince the AI to bypass its own safety training. Once “convinced,” the AI acted as an unauthenticated proxy, executing privileged commands and exfiltrating data—all while believing it was simply being helpful.

The Fatal Flaw: The LLM is Not a Security Control

This emergent industry currently maintains a tenuous over-reliance on the LLM as a primary security layer. Our research revealed that AI judgment is a supplement to, not a replacement for, rigorous system controls.

In many environments, the last line of defense is the AI’s ‘ethics.’ When that fails—as it inevitably does under pressure—there are often no technical backstops. If an AI tool is ‘open-ended’ (e.g., capable of running arbitrary system commands), once the LLM is subverted, an attacker gains the full permissions of the MCP tools that connect and extend the agent’s reach across the enterprise.

A Tale of Two Architectures: The Value of Constraints

The most encouraging part of our research is that we’ve seen what “good” looks like. In the same environments where open-ended tools were easily exploited, we identified “Gold Standard” patterns that remained resilient:

• Hardened Scoping: Tools that utilize whitelists to restrict the AI to specific views, command, or directories.
• Automatic Sanitization: Backend code that proactively blocks dangerous keywords (like administrative stored procedures) before the query ever hits the database.
• Metadata Separation: Architectures that allow the AI to “see” the schema (how the data is organized) without granting it the keys to “read” the sensitive records themselves unless explicitly authorized.

Moving at the Speed of AI: Future-Proofing Security

As new language models are released at a breakneck speed, our security strategies must become model-agnostic. We cannot assume that “Version 4.7” will be safer than “Version 4.5.” Instead, we must harden the environment the AI inhabits.

• Zero-Trust for Tools: Every tool should assume the LLM has been compromised. Every request from the AI must be sanitized and validated as if it came from a malicious user.
• Micro-Tools Over Multi-Tools: Move away from “Swiss Army Knife” tools. Security is found in single-purpose, constrained endpoints that only have access to the data required for a specific task.
• The Human-in-the-Loop: For high-risk operations, re-insert the human. AI can suggest a database change, but a human (or a secondary, non-LLM security layer) must provide the final authorization.

Conclusion

The speed of AI advancement is breathtaking, but my 20 years in this field have taught me that fundamentals always win: Trust, but verify; and then, don’t trust at all. The goal for modern organizations shouldn’t just be to build a smarter AI, but to build a more resilient platform that remains secure even when the AI makes a mistake. In the world of Agentic AI, the most dangerous vulnerability isn’t a line of code—it’s the assumption that the machine can secure itself.