Hunting for Security Bugs with AI Agents: A Full Walkthrough
In my previous article, I introduced Hound, an open-source code auditing tool that models the cognitive and organizational processes of human experts (see the paper for details). In this follow-up, we’ll walk through a complete end-to-end audit using Hound in practice. Hopefully, this will help developers catching security bugs before code goes into production, or security researchers to earn some bounties. Use it responsibly.
Hound is a code security analysis agent that uses autonomously generated, adaptive knowledge graphs to model the components and relationships of the target system. These graphs (like mental models) let LLMs reason about different aspects at multiple levels of abstraction while “zooming in” on the precise code snippets needed at any point during the reasoning process. They also allow the agent to refine its understanding of the codebase over time and revise its opinions on previous findings as new information comes in.
Although Hound was originally developed for smart contract auditing, it turns out that its analysis pipeline generalizes well across other programming languages. And since Rust seems to be on everyone’s mind these days, I decided to tackle a Rust project for this run.
To get going, I asked Grok to recommend Rust applications that might contain vulnerabilities. One of its recommendations was rustic-server — a REST server that’s part of the Rustic backup system. With only ~30 stars and an explicit “this is early dev software” warning, this seems like a perfect candidate!
With that decided, let’s gear up and unleash Hound on rustic_server, walking through each step of the journey in detail.
Installation and Configuration
To follow along, you’ll need API keys for at least one LLM API. The configuration described below has worked well for me, but you can experiment with mixing and matching different models in various ways, including OpenAI, Anthropic, xAI, Google and Deepseek.
Install Hound: If you haven’t already, grab Hound’s code.
git clone https://github.com/muellerberndt/hound.git
cd hound
pip install -r requirements.txtAPI Keys: Export your keys as environment variables (never hard‑code them or store them in files committed to source control):
export OPENAI_API_KEY="sk-…your OpenAI key…"
export GOOGLE_API_KEY="…your Google (Gemini) key…"Copy the example config and open it for editing:
cd hound
cp config.yaml.example config.yaml
# Now open hound/config.yaml in your editorThis config.yaml is where you tell Hound which models to use for each role. There are five main roles:
- Graph builder — builds the knowledge graphs from code. Needs the largest possible context window to ingest and structure the code. Gemini (~1M input tokens) is ideal here. GPT‑4.1 also produces good results but is very expensive.
- Scout (junior agent) — explores code and annotates graphs. The scout relies on the strategist for advice. Can run on a smaller, cheaper model. GPT-5-nano or GPT-5-mini works well here.
- Strategist (senior agent) — plans the audit, generates vulnerability hypotheses and guides the scout. Use your strongest model (GPT‑5, Opus 4.1, or whatever else is best at the moment).
- Finalizer (QA) — reviews and verifies hypotheses. Requires a strong reasoning model (GPT‑5 or Opus 4.1).
- Reporter — (optional) generates the final report. Can use a mid‑tier model for summarization and report generation. I normally use GPT-4o.
Example:
models:
graph:
provider: gemini
model: gemini-2.5-pro
max_context: 1000000
scout:
provider: openai
model: gpt-5-mini
reasoning_effort: low
strategist:
provider: openai
model: gpt-5
plan_reasoning_effort: medium
hypothesize_reasoning_effort: high
finalize:
provider: openai
model: gpt-5
reasoning_effort: high
reporting:
provider: openai
model: gpt-4oStep 1: Building Graph Models
In this phase, Hound builds relational models of the target scope. They are called “aspect graphs”.
Let’s start with the limitations: Hound does not perform well on very large codebases. Graph construction is constrained by the model’s context window, and results deteriorate once the codebase exceeds ~80k lines. At 10–20% above the limit you may still obtain usable graphs, but beyond that the builder will resort to random sampling and quality drops sharply. For large systems, it’s best to isolate subsystems and audit them independently.
Use a high-context model like Gemini for this step, since it must ingest and structure a significant volume of code. The graphs it produces form the foundation of the entire audit. If the graphs are weak, all later reasoning will suffer. Reserve your strongest reasoning model (e.g. GPT-5) for deeper phases like strategist and finalizer.
Start by creating a whitelist of target files. You can write this manually (a comma-separated file list) or generate it with a script. Personally, I use a whitelist builder to collect the most relevant files within a given LOC budget.
To begin, download the Rustic server repo and generate a whitelist (this should include all Rust files):
./whitelist_builder.py --input sources/rustic_server \
--output whitelists/rustic_server.txt \
--print-summary --verbose --limit-loc 50000 --enable-llNow build the graphs in auto mode. By default, this will model 5 aspects, which should take a few minutes.
./hound.py project create rustic_server
./hound.py graph build rustic_server --auto \
--files "$(tr -d '\n' < ../whitelists/rustic_server.txt)"After the command returns, you can list the generated graphs and export them to a HTML file:
./hound.py graph ls rustic_server
Name Nodes Edges Updated Focus
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
RepositoryDataFlow 70 136 2025-09-06T08:56:51 The lifecycle of data within a repository (e.g., `data`, `keys`, `index`, `snaps
CLICommandWorkflow 32 56 2025-09-06T08:46:34 The operational logic of the `auth` subcommand, which provides an out-of-band ad
AuthorizationMap 41 83 2025-09-06T08:18:05 The fine-grained access control logic centered around the `acl.rs` module. This
SystemArchitecture 46 91 2025-09-06T08:10:04 A high-level overview of the `rustic-server` components, their interactions, and
ConfigurationInfluence 44 83 2025-09-06T08:02:28 How configuration settings, from `rustic_server.toml` and command-line overrides
./hound.py graph export rustic_server --openYou should see something like this:
The agent discovers useful aspect graphs automatically. You can also tell it to build a graph of your choice. For a REST server like Rustic, a taint-style data flow graph would be nice to have. We create it with:
./hound.py graph build rustic_server \
--graph-spec "Data flows and validation from taint sources to sinks" \
--iterations 4
This graph helps the agent reason about input validation. By examining the generated graphs, we can see that they model data flows from taint sources — such as URL paths — to vulnerability sinks like file system operations. Crucially, validation functions (for example, check_name) are also included in these flows.
During analysis, the agent can drill into specific slices of the graph. For instance, to verify whether input validation is performed during URL processing, the agent can simply load the code snippets corresponding to the relevant nodes along the data-flow path.
Another important feature is the tracking of assumptions and observations. As the agent explores the codebase, it annotates the graphs with information it discovers (some of which is visible in the example above). This process produces a richer mental model: when zooming out, the agent can detect mismatches between expected and actual behavior. Identifying these mismatches is crucial for uncovering bugs in the application-specific logic of the target system.
Different use-cases require different models. Examples:
- DeFi app: model asset flows across contracts.
- Compiler/parser: model state transitions in parsing logic.
- ZKP library: model group operations, elliptic curve pairings, or polynomial transformations in the proving system (I didn’t try this so no idea if it actually works).
The graphs for Rustic look good now, so let’s move on to the actual analysis:
Step 2: Running the Audit
The default way to run an audit is simply:
./hound.py agent audit rustic_serverThe longer it runs, the better — just like a human audit, more time means more coverage and deeper reasoning. You can also bound the runtime:
./hound.py agent audit rustic_server --time-limit 120This will run for two hours. The agent tries to balance prioritizing promising areas with eventually achieving broad coverage of the code. At the moment this strategy can’t be customized.
You can interrupt an audit at any time and resume it later. Sessions can be listed and resumed with the --session option. Hound also reports coverage statistics as it runs, showing how many graph nodes and code slices have been explored. During planning, the strategist typically prioritizes high-impact areas, but is also encouraged to visit previously unseen code. Striking the right balance between depth and breadth is especially important for long audits.
Chatbot and Web UI
To monitor audits interactively, use the --telemetry flag together with the web chatbot UI:
./hound.py agent audit rustic_server --telemetry --session
python chatbot/run.pyOpen the UI in your browser to watch the audit in real time. In the chatbot, you’ll meet Inuzumi, your private audit assistant, who guides you through the investigation in a more conversational way (See the README for setup details.
You can talk to her in real-time using voice to get status updates, tell her to shift the priority of the audit. Note that this is the least developed part of Hound, so don’t expect it to work perfectly.
Steering the Audit
You can also steer investigations with missions or targeted investigations:
./hound.py agent audit rustic_server --mission "Focus on input validation flaws"
./hound.py agent investigate rustic_server "Focus on input validation flaws"This lets you guide the Strategist toward particular concerns while still leveraging the knowledge graphs.
Step 3: Eliminating False Positives
As the agent works, it forms hypotheses (or beliefs) and continuously adjusts their confidence as new evidence appears.
How this works: During an audit, the Strategist creates initial beliefs with an assigned confidence level. As the Scout explores the codebase and gathers evidence, those confidence levels rise or fall. Over time, this process results in a collection of hypotheses, each associated with a different degree of confidence.
At any point, you can review the current hypotheses by running the project hypotheses command or by checking the Findings tab in the web UI. You will see a list similar to this:
The percentages on the left represent the model’s current confidence that a given issue is valid.
Once you decide the audit has progressed far enough, it is time to finalize. During this stage, Hound takes all beliefs above a chosen threshold (≥ 0.5 by default) and hands them to a reasoning model for direct verification against the source code. The model may also pull in additional files or functions as needed to reach a conclusion. Most of the time, this process either confirms or rejects the hypotheses. In some cases, however, the result may remain uncertain — for instance, when the codebase is incomplete or dependencies are missing.
The finalization step does the following:
- Adjusts confidence values based on the deeper reasoning pass
- Updates the status of each hypothesis (confirmed / rejected / uncertain)
- Records a concise justification that cites the relevant code lines considered
You can view the updated hypotheses and their statuses in the Findings tab.
To run the finalization process via the CLI:
./hound.py finalize rustic_serverAt this point, the agent will investigate each issue one by one. For example:
🧠 Reviewer gpt-5 enters the chamber...
Loaded 1 file(s) from source_files
Added 10 file(s) for next iteration
✓ CONFIRMED: Read-only users can create/remove locks due to access downgrade
Justification: The code explicitly downgrades all access checks for the Locks type to Read, which allows users with only read permissions to perform write operations on locks. In src/acl.rs, Acl::is_allowed sets
access_type = AccessType::Read whenever tpe == TpeKind::Locks, regardless of the requested access (Append/Modify). Handlers for creating and deleting files (src/handlers/file_exchange.rs add_file/delete_file) require
AccessType::Append via check_auth_and_acl, but since they pass the Locks type through to is_allowed, the downgrade causes these checks to succeed for read-only users. The web router (src/web.rs) exposes POST/DELETE
/:repo/:tpe/:name, so POST/DELETE /:repo/locks/:name is reachable. There are no compensating guards preventing writes to locks; even the ACL tests (src/acl.rs tests) assert that Modify on Locks is allowed for users with
lesser privileges, confirming the behavior. This permits read-only users to create or remove lock files, enabling denial-of-service (e.g., persistent locks) or unauthorized unlocking.After finalization completes, you can re-run project hypotheses to see the updated confidence levels and statuses.
Step 4 (Optional): Proof-of-Concepts
In general, if you need a proof-of-concept for an issue, it’s always a good idea to let a coding agent do most of the work for you. Hound includes experimental support for generating and managing proof‑of‑concept exploits. The poc command group has three subcommands:
make-prompt– Generate PoC prompts for confirmed vulnerabilities. This produces natural‑language prompts you can feed into a coding model to generate exploit scripts or test cases.import– Import actual PoC files (e.g. scripts you’ve written) and attach them to a hypothesis.list– Show all imported PoCs linked to the project.
I won’t go into detail here since this is just an early experimental feature.
Step 5: Exporting the Report
Finally, export a report:
./hound.py report rustic_serverThis generates an HTML report with the audit details. Since this is still an early version, expect a few rough edges (for example, duplicate elimination in Hound is currently weak).
At the beginning of the report you’ll find an executive summary in the style you’d expect from a professional audit firm. (For some reason, GPT-4o hallucinated a line about the Rustic team “paying prompt attention to the discovered vulnerabilities,” which is a little hilarious).
Let’s look at the first finding:
One of the HTTP methods that can escape the webroot is DELETE method can escape outside of the webroot.
Checking the results
The first finding is a path traversal bug that allows a user to read and delete data they shouldn’t. Let’s create a minimal PoC very to test if this actually. First, we create a config.toml:
[server]
listen = "127.0.0.1:8000"
[storage]
data-dir = "/tmp/data/webroot"
[auth]
disable-auth = true
[acl]
disable-acl = true
append-only = false
[tls]
disable-tls = trueThen, we create the required directories and start the server:
mkdir -p /tmp/data/webroot
rustic-server serve --config config.toml & SERVER_PID=$!Let’s attempt to exploit this bug:
curl --path-as-is -sS -u '..:x' -X DELETE http://127.0.0.1:8000/../Looking into /tmp/data, we see that the webroot directory was deleted. Interestingly, if we put the webroot directly under /tmp, we instead get a permission-denied error:
curl --path-as-is -sS -u '..:x' -X DELETE http://127.0.0.1:8000/../
error removing repository folder: "Could not remove repository: Permission denied (os error 13)"This indicates that the server at least attempts to remove the parent directory of the webroot itself, but the operation fails due to filesystem privileges. Further testing suggests it’s unlikely that directories further up the hierarchy can be deleted.
From what we’ve seen so far, the impact of this issue appears to be minor — it certainly wouldn’t qualify as critical in a bug bounty program. We could consider reporting it to Rustic via GitHub at some point, but there’s no rush. The key takeaway is that this is a real, reproducible issue, and Hound correctly described it, complete with precise code locations spanning multiple files.
I also skimmed through the remaining findings, and they looked reasonable enough. The results should be reproducible by anyone running Hound on the same repository.
Conclusion
In this walkthrough, we went end-to-end with Hound: generating a whitelist of target files, building aspect graphs, running the audit, reviewing hypotheses, and finally exporting a report. Along the way, we saw how the agent models system components and their relationships and ties them back to exact code snippets.
Hound is still early and has clear limitations. Severity ratings can be inflated, long-term audit planning is fairly basic, and large codebases remain a challenge. Because little tuning and benchmarking has been done so far, there’s plenty of room for improvement. Enhancing graph construction, refining audit planning, and optimizing the trade-off between coverage and cost are just some of the most important next steps.
I believe, however, that the core principles behind Hound already bring AI audits close to optimal. Its dynamic, graph-based, iterative, multi-prompt, belief-refinement approach has the potential not only to elevate code security reviews, but also to boost LLM reasoning in entirely different domains — from scientific research to medical diagnosis. I’m excited to see how it evolves as the underlying models continue to grow stronger.
Postscript for Security Scanner Vendors
When I released Mythril back in 2017, several commercial vendors wrapped it into online services to sell “online security scans”. This time around, if you intend to integrate Hound into such a service, you will need to either purchase a license or acquire the intellectual property rights (see LICENSE.txt).
There are no restrictions on other uses, including security research, bug bounty work, audit contests, and professional audits.
