How Bastyn Scan finds bugs in AI agent code

Bastyn is a static scanner for AI agent repositories. You point it at a folder, it reads the files, and it reports security problems. It doesn't run your code and it doesn't call a model. Everything it knows comes from reading text.

This article explains how the engine works, from the outside in. The first section is the whole pipeline. The second opens the part that reads source code. The third covers the checkers for everything that isn't source code.

1. Walk. The engine lists every file under the path, in sorted order. Anything it leaves out on purpose is written down as "skipped" instead of vanishing.
2. Route. Each file is classified by name and type, without being opened. A Python file goes to the rule engine. mcp.json goes to the MCP checker. A Dockerfile goes to the infrastructure checker. A file nobody claims, like an image, is never read.
3. Analyse. Each file is handled on its own, across all CPU cores. When the results come back they are put in file order before anything reads them, so the order of the report doesn't depend on which thread finished first. Given the same files, scanner version and settings, with --offline enabled, the output is identical on every run. An agent can re-run the scan to check its own fix and get a comparable answer.
4. CVE lookup. Dependencies found in manifests like package.json and requirements.txt are checked against OSV.dev, a public database of known vulnerabilities. This is the only step that uses the network, and --offline skips it. It is also the one input the scanner doesn't control: the database changes over time, so the same repository can gain a finding with no change to the code. The report records the lookup status for this reason.
5. Report. Duplicates are removed, findings are sorted, and everything is split into two kinds: defects and observations. The text, JSON and SARIF outputs all print from the same report object.

A defect is something the scanner can point to in the code and call wrong. The dataflow rules hold to that standard. A few text checkers also report defects on weaker evidence, and the Skill files section says which. An observation is something worth knowing that the scanner won't call a defect. Most are missing controls, such as "this agent has no rate limit". The repository can't show whether that absence is a problem, so the scanner doesn't claim it is. A few are non-security validity findings, such as a skill file missing a required field. Observations are hidden unless you ask for them.

The project's design rule is "say less, be right". It is a design goal, not a measured result, and the measurements are modest. The release gate runs the scanner over a test corpus the authors wrote themselves. On it, the scanner finds all 99 cases designated as expected detections. But the same people wrote the fixtures and the rules, so 99 out of 99 works as a regression alarm. It says little about real repositories.

The gate also tracks the failures. Each known gap is one specific file and line where a defect exists and the scanner misses it, written down with the reason. There are 17 of those, plus 8 that only a network lookup could catch. The gate runs offline, so those 8 are excluded from the figures below. In the offline evaluation, the scanner detects 99 positive cases and misses 17, giving recall of 99/116, about 85%. Of the 103 defects it reported, 99 were expected and 4 are documented false positives, giving precision of about 96%. These figures come from scanner version 0.2.2 at commit 3b3bedc. Recall and precision on real-world code are unmeasured.

The dataflow rules favor narrow, traceable findings, and when one of them can't be made precise, it is left out. Some text checkers use weaker phrase matching, covered below. A file that can't be parsed is listed as skipped, not counted as scanned, because a scan that quietly covered less than it claimed is worse than one that failed.

Most agent code is Python, TypeScript or JavaScript, so most of the work happens in the rule engine. It has three stages.

Searching source text for eval( would flag comments and strings that mention it. So the engine first turns each file into a syntax tree using tree-sitter, the same kind of structure a compiler builds. In that tree a function call is a call node with a name and arguments, and a comment is a comment node that no rule confuses with code.

The engine then walks the tree once. At each node, it asks every rule whether the node looks like a match.

Rules are written in YAML, and the core of a rule is a code pattern with blanks. This is a simplified version of the rule for model output executed as code:

- id: BAS-LLM10-001
  title: Model output executed as code
  kind: defect
  severity: critical
  language: python
  any:
    - eval($ARG)      # $ARG matches any argument and remembers it
    - exec($ARG)
  none:
    - eval("$LIT")    # fixed strings are excluded from this rule

The matching is done by ast-grep, which compares a pattern to the tree. $ARG is a metavariable, a placeholder that matches any expression. Rules can narrow themselves with none (not if it looks like this), inside (only inside this), and a few other conditions. The engine ships 53 rules across four YAML files.

Shape alone isn't enough. eval(x) is only a hole if an attacker can influence x. In an agent, that usually means x came from a model's reply, because a model can be steered by any text it reads. The code's shape doesn't say where x came from.

The first version of this rule checked the variable's name. It fired if the argument was called something like response, completion or output. The project then measured that guess: rename the variable to suggestion or expression and the scanner missed the bug. The name-based rule caught 0 of 10 realistic renamings of the same bug.

So rules can now carry a flow: clause, and the engine follows the value instead of reading its name. For each file it builds a small dataflow graph: this variable was assigned from that call, which was passed into this function.

reply = client.chat.completions.create(...)   # source: a model call
code = reply.choices[0].message.content       # the value flows along
eval(code)                                    # sink: executed. Defect.

The names don't matter. What matters is that code traces back to a call the engine recognises as a model call. The engine carries a catalogue of sources (model calls) and sinks (eval, os.system, pickle.loads and similar). It matches on the API being called, never on what the author named things.

Two refinements keep it honest:

• Guards. For these rules, the engine accepts certain checks that restrict a value to a fixed set of literals, and it drops the finding when one of them dominates the sink: every path that reaches the sink has passed through it. A check that sits nearby doesn't count. The engine matches the check to the sink by variable name, within one function. It does not invalidate that match when the variable is reassigned after the check, so a finding can be suppressed even when the value reaching the sink is no longer constrained. This is a known bug (issue 39), not a design choice. The guard checker recognises three forms, and each needs a fixed set of literal values (a tuple, list or set written in the file):

  • the value is itself drawn from such a set;
  • if value not in (...) followed by a branch that ends in raise, return, break, continue or a process exit;
  • if value in (...) with the sink inside the branch.

  Everything else counts as unguarded and the finding stays. if x not in registry.known() proves nothing the engine can check, so it doesn't count. Neither does assert x in (...), because python -O strips assertions. The narrowness is deliberate. A wrongly accepted guard hides a real vulnerability, which is worse than the false alarm it removes.

• One hop of wrapping. If eval is hidden in a helper function in the same file, the engine follows the value into the helper.

The tracer has hard limits. It handles Python only, one file at a time, and follows a function call one level deep. When it can't prove where a value came from, for example because it was assigned differently in two branches, it answers "unknown". Unknown never produces a defect.

That means the scanner misses some real bugs. It accepts that on purpose. When it does speak, the finding should be right.

The dataflow graph is built lazily. It is only built for a file after a rule has already found a candidate match that needs flow analysis, so most files never pay for it. And large files are scheduled first across the cores. One 20 MB bundled file can otherwise leave most cores idle while a single core grinds through it.

Agent repositories hold more than source code. Configuration, instruction and environment files go to specialized checkers that inspect their structure or text directly. Their rule IDs use their own prefixes, like BAS-MCP-*, instead of the BAS-LLM* style.

SKILL.md is the manifest that agent hosts like Claude Code and Cursor use to describe a skill, a packaged set of instructions an agent can load. It plays two roles at once. The host trusts it as a manifest, and the model reads its text directly: the description field is shown to the model so it can decide whether to use the skill.

That makes it an attack surface. In ordinary code, a comment saying "ignore previous instructions" is just a comment. In a skill file, that sentence is addressed to a model. The checker makes three checks:

• BAS-SKILL-001: the header is missing name or description. This is a spec-validity check and is reported as an observation, not a security claim.
• BAS-SKILL-002: classic prompt-injection phrasing in the skill's text, such as "ignore previous instructions". Maps to LLM01, Prompt Injection, in the OWASP Top 10 for GenAI (August 2026 edition). Reported as a defect at high confidence.
• BAS-SKILL-003: language telling the agent to skip confirmation before acting, such as "do not ask for confirmation". Maps to LLM03, Excessive Agency, in the same 2026 edition. That edition renumbered the list: Excessive Agency was LLM06 in the 2025 edition, and LLM03 there was Supply Chain. Reported as a defect at medium confidence.

The checker finds the file by name, wherever it sits in the repository, because a host can load a skill from outside the usual folder.

These two checks are weaker than the dataflow rules. They match fixed phrases in raw text and don't know what the text means. A skill that documents prompt-injection detection can contain "ignore previous instructions" as an example and trigger BAS-SKILL-002. A skill that legitimately describes acting autonomously within a narrow task can trigger BAS-SKILL-003, which is why that one is reported at medium confidence. The project accepts this trade-off because the phrases are a cheap, clear signal. Read these findings as "a human should look at this line", not as proof of an attack.

An empty report means the scanner found nothing within its limits. It doesn't mean the repository is safe. Those limits are:

• dataflow tracing covers Python only, one file at a time, one call deep;
• a value it can't trace produces silence, so some real bugs are missed by design;
• a guard is matched to the sink by variable name, so code that reassigns the name after the check can still be treated as guarded;
• missing controls are observations and are hidden by default;
• files it couldn't parse or deliberately left out are listed as skipped, and the count is in the summary;
• dependency checks only cover versions pinned exactly, and only as of the day the database was queried.

Check the skipped list and the summary before reading "no findings" as good news.

The engine aims for precise findings, but the strength of the evidence varies by checker. The dataflow rules trace a value from a source to a sink. The text checkers match phrases and shapes, and their findings need a human to read them. A file that can't be parsed is listed as skipped. A value it can't trace produces silence. A missing control is an observation. With the same scanner version and settings in offline mode, repeat scans provide a stable basis for comparing findings after code changes. Whether the scanner is right as often as it aims to be is a question for measurement on real repositories, and the corpus numbers above are a start.

Try it on your code 

Bastyn Scan has been run against thousands of repositories so far. But to keep improving the scanner and the next round of assessments, we need more people to scan their repos. If you build agents, you can help:

• Run it on your own agent code. It takes a minute and needs no account. Get it from github.com/bastyn-labs/bastyn-scan.
• Tell us what it got wrong. False alarms, missed vulnerabilities and rules that do not fit your framework are the most useful input we can get. Open an issue at github.com/bastyn-labs/bastyn-scan/issues.
• Share your results with the community. Post what it found, and what it missed, in the same issue tracker so others can compare notes and we can fold the findings into the next round.