Can AI Audit Your Firewall Rules? What an LLM Agent Catches in a 104-Rule Set

Firewall rule bases rot quietly. A ruleset that started clean accumulates exceptions, temporary “just for now” permits, and duplicates until no human can confidently say what it actually allows. That makes firewall configuration review an appealing target for AI assistance: the work is tedious, pattern-heavy, and unforgiving of small mistakes. A recent experiment put this idea to the test by pointing an LLM-driven agent at a deliberately flawed next-generation firewall (NGFW) policy and measuring what it caught. The results are a useful reality check on where AI-assisted auditing helps today and where it quietly fails.

The test setup

The exercise used a real NGFW policy exported from the FORWARD chain as a CSV file of roughly 34 KB containing 104 rules. The configuration modeled six network zones — an office LAN, a server LAN, a DMZ, a guest Wi-Fi segment, a management network, and a remote branch — and each rule specified the usual attributes: protocol (TCP/UDP/ICMP), source and destination objects, interfaces, ports, schedules, IPS/DPI inspection flags, and an action (accept or drop).

The policy was deliberately seeded with realistic problems across severity tiers, so the experimenters knew the ground truth in advance:

• Two critical issues. One was a broad drop-any-to-any rule placed ahead of the default-deny, shadowing later rules and suppressing logs on blocked traffic. The other was a chain of guest-network rules opening RDP, SQL, SSH, HTTP, HTTPS, and DNS straight into internal servers.
• Five high-severity issues, including unrestricted guest-to-office access, DMZ servers reaching internal administration ports, and services published on HTTP (80) and SMTP (25) with no IPS/DPI inspection.
• A spread of medium and low items: overly permissive policies and duplicate rules.

The most interesting trap was a set of dangerous accept rules masked by earlier drop statements. Because of ordering, those permits were inert — but they were “land mines”: reorder the policy, and the holes go live. Spotting them requires reasoning about rule precedence, not just reading each line in isolation.

Two failure modes: tool-calling vs. analysis

The original ambition was a fully autonomous agent — connect to the firewall over its API, pull the rules, analyze them, and write a report. That ambition exposed the first hard lesson: agent capability and reasoning quality are different dimensions.

A weaker model never got past step one. It couldn’t reliably chain the tool calls needed to authenticate, query the API, and act on the results. Autonomous operation depends on dependable tool-calling, and the model simply didn’t have it. To salvage a fair comparison, the testers removed the most fragile layer — live tool use — and handed both models the pre-extracted CSV. That isolated analytical ability from agentic plumbing.

What the capable model found

Given the CSV, a frontier-class model produced a clean, severity-tiered report that closely matched the seeded ground truth:

Severity	Findings
Critical	2
High	5
Medium	8
Low	6
Info	2

It flagged the premature drop-any rule as a “silent killer,” explaining the downstream effect: it renders the following rules inactive and stops blocked traffic from being logged, creating blind spots for incident investigation. More impressively, it understood the masked-permit problem. It correctly noted that the guest-to-server, DMZ-to-server, and guest-to-office permits were currently neutralized by upstream drops, but described them as delayed-action mines that any future reordering would arm — exactly the reasoning a human reviewer would apply.

It then went beyond detection into remediation: publish HTTP through a reverse proxy with web application firewall protection, route mail through a filtering relay, and tighten the order-dependent rules so security no longer hinges on accidental sequencing. The output read like an actionable plan, not just a list of complaints.

What the weaker model reported

The weaker model produced confident nonsense. Its summary claimed it analyzed 105 rules, identified 4,083 redundant rules, found zero unsafe rules, and saw no missing rules. It concluded that nothing in the configuration permitted arbitrary access to resources.

Two things went wrong. The 4,083 “redundancies” came from naive combinatorial enumeration — counting nearly every pair of rules with overlapping sources or destinations as duplicates. In reality, only one genuine duplicate existed, meaning roughly 4,000 findings were false positives. Worse, it missed every seeded critical and high issue — even rules that carried explicit “CONFIG ERROR” annotations in the data.

This is the dangerous outcome. A confident false negative is worse than an honest failure. A model that says “I can’t do this” prompts a human to step in. A model that says “your firewall is secure” when it is riddled with exploitable holes actively manufactures false assurance.

Cost is about useful output, not tokens

A counterintuitive footnote: the model that failed burned several times more tokens than the one that succeeded — on the order of ~700K tokens versus ~85–110K (dropping toward ~40K with prompt caching) for a result that was both correct and concise. The lesson for anyone budgeting AI workloads is to measure the cost of a useful result, not the per-token price. A cheap token that produces a wrong audit is the most expensive token you can buy.

Key takeaways

• AI can do real firewall auditing — with the right model. A capable LLM found all seeded critical and high-severity issues, reasoned about rule ordering, and produced a remediation plan.
• Tool-calling reliability and analytical depth are separate hurdles. A model can be too weak to drive an API yet still be trusted to think — or fail at both. Test them independently.
• Beware confident false negatives. “All rules are secure” on a flawed policy is more harmful than an explicit inability to complete the task.
• Ordering bugs need reasoning, not pattern-matching. Masked permits that activate on reorder are invisible to line-by-line scanning.
• Judge AI agents on cost-per-useful-result. Token volume is a poor proxy for value; a failing model can be the more expensive one.

AI-assisted configuration review is no longer hypothetical, but it is not a checkbox you can outsource blindly. Treat the agent’s output as a knowledgeable first pass that still needs an expert reviewer — and validate any model on a seeded, known-answer ruleset before you trust it with the real one.