Performance¶
Making a slow scan faster¶
A real scan is dominated by the gates, not by gandalf. Thirty-odd gates each
shell out to a linter, a scanner, or a docker run; that is tens of seconds to
minutes, and no amount of tuning in gandalf's own code touches it. The lever is
the gate set and the order it runs in.
First: find out where the time actually goes¶
Every line carries the elapsed time it was written at, so the log reads as a timeline: each stage as it starts, the order the gates were scheduled in, each gate's start and its duration, and every external command with its timeout and exit code. The run ends with the five slowest gates.
A scan that runs out of time¶
A run with no --deadline is bounded only by whatever is waiting on it. In the
editor that is gandalf.scan.timeoutSeconds, and a kill at the timeout used to
end the scan with nothing: no scorecard, and — because the cache was written
once at the end — nothing banked either, so the next scan re-ran the thirty
gates that had already finished and was killed in the same place.
Both halves are fixed, and they compose: give the run a --deadline and it
stops itself with time to write the report, and every gate that finished is in
the cache the moment it finishes, so the scan after it starts from there. The
extension passes one automatically, derived from its own timeout.
Reach for --debug especially when a scan doesn't finish. The timings in the report
and the extension's Gandalf: Show Gate Timings can only describe gates that
got to the end; a gate still running when the editor's timeout fires appears in
neither, and the --debug log is the only place it is named — look for a gate
<name>: start with no matching completion.
Then: the levers, roughly in order of what they save¶
| Lever | What it does |
|---|---|
--no-llm |
Drops the LLM summary and the judge gates (grill_me, codebase_architecture, well_architected, compliance, the skill_* gates). Each is a full model round trip; with nothing listening at GANDALF_LLM_URL each is a connect timeout plus its retries instead. For an editor or pre-commit run this is usually the largest single saving. |
skip in .gandalf.toml |
The honest answer for a gate that costs minutes and tells you something you already know from CI — codeql, ci_act, a full test-suite gate. In the repository, where it can be reviewed. |
--cache |
A gate whose files did not change is not re-run at all. Advisory gates (trivy, osv, …) still expire after six hours, because a new CVE is a new answer for identical bytes. Each result is written as it lands, so a run that is killed still leaves everything that finished — the next scan starts where this one stopped. A gate that could not run is never cached; "tool missing" and "timed out" are not answers to keep. |
--deadline N |
Bounds the run as a whole, which a per-gate timeout cannot: one gate may make fifty tool calls, so "120s each" is not 120s. Past the deadline a tool call is not started, so the queue drains in milliseconds and the scan ends with a scorecard — the gates that did not get their turn are marked did not run rather than counted. Off by default; the editor extension sets it just under its own kill. |
--exclude / .gandalfignore |
Vendored trees and generated code are usually most of what a full-tree scanner reads. |
[gandalf.timeouts] |
Bounds the worst case per gate rather than removing it: semgrep = 60 degrades that gate to amber at a minute instead of letting it own the run. |
--concurrency N |
Trades wall-clock for a responsive machine. Rarely a speed-up on its own — see below. |
One scan, not one per gate¶
trivy fs walks the whole repository, and two gates want what one walk
produces: trivy reads the vulnerabilities, secrets and misconfigurations,
licenses reads the licences that same scan already collected. It is made once
now and awaited by whoever else needs it — one walk instead of two, and no two
trivy processes contending over the one vulnerability database the tools image
shares between them.
Why the order gates run in matters¶
Gates run concurrently but bounded, so there is a queue, and whichever gate
starts last decides when the run ends. Discovery order is alphabetical by module
filename, which put bandit first and trivy near the back: the queue drained
into a five-minute scanner that nothing could overlap with.
Gates are now submitted heaviest first — longest-processing-time-first, the
standard scheduling answer — so the expensive ones start immediately and the
quick ones fill in behind them. The estimate is, in order of preference: what
the gate cost last time (recorded per gate in the --cache file, so the second
run on a repository schedules by measurement), a cost attribute a gate
declares for itself, then a coarse built-in prior. See gandalf/schedule.py.
The practical consequence is that lowering --concurrency is much less
punishing than it was: with --concurrency 2 the two heaviest gates are the two
that start, rather than two arbitrary linters while trivy waits.
What the benchmark measures — and what it doesn't¶
Everything below is the other cost: the work gandalf and the extension do themselves, between the gates finishing and the report appearing. It matters because it is the part that runs on the editor's UI thread, and the part that scales with the number of findings rather than with the number of gates. It is not where a slow scan's minutes are.
Running it¶
make bench # table + redraws the chart above
python3 scripts/bench.py --json # machine-readable, to diff over time
Stdlib only, no dependencies. The extension half needs node and an
npm install in extensions/vscode; without them that section is skipped and
the gandalf half still runs.
Each figure is the fastest of several runs, not the mean: every source of noise on a developer machine makes a run slower and none make it faster, so the minimum is the closest thing to the cost of the work itself. Absolute numbers are machine-specific — the reference run below is a 16-core Linux box on Python 3.13 — but the ratios travel.
| Operation | Before | After | |
|---|---|---|---|
| Rebuild the board, 40 streamed gates | — | 570 ms | |
| Reconcile 20k findings | — | 136 ms | |
| Cache key, 2k files + a 32 MiB blob | 108 ms | 100 ms | |
| Exclusion filter, 25k paths | — | 79 ms | |
| Normalize 20k findings (extension) | — | 65 ms | |
| Detect languages, 400-file tree | 1.74 ms | 0.24 ms | 7.4x faster |
| Sort 20k findings (extension) | — | 1.66 ms | |
| Cache key, peak heap | 32.0 MiB | 0.27 MiB | 118x smaller |
| Write JSON report, 20k findings | 8.47 MiB | 0.05 MiB | 166x smaller |
Where the wins came from¶
Peak memory, twice. write_text(json.dumps(...)) renders the whole report
into a string and then writes it, so the process holds the document twice over
at the moment it lands; json.dump streams it into the file. Separately, the
cache key hashed each file with read_bytes(), which puts that file's entire
contents on the heap — one 32 MiB tracked blob was a 32 MiB spike. file_digest
reads in fixed-size blocks.
One listing, not two. scope.languages() ran its own git ls-files
moments before the gates ran another one. It now reads the cached listing —
which also fixed a bug, since plain ls-files quotes non-ASCII paths and the
old whitespace split() mangled them into nothing a language matcher
recognised.
Counted, not timed. Three more changes don't appear above because their win is a count rather than a duration, and counts are asserted in the test suite instead: the findings pane walks the board once per repaint rather than two or three times, diagnostics are published in one bulk write rather than one per file, and the report webview no longer re-renders a multi-megabyte document into a hidden tab on every save.
Two things that measured the wrong way¶
Kept here because the obvious "optimization" is wrong in both cases, and the next person will otherwise try it again.
Hoisting an Intl.Collator out of the sort comparator made it 5x slower —
33 ms against 6 ms sorting 20k paths. V8 already fast-paths
String.prototype.localeCompare for the default locale; extracting .compare
loses that fast path, and numeric: true disables it outright. The comparator
in parse.ts carries a note saying so.
file_digest is about 20% slower than read_bytes() on a tree of nothing but
small source files — one syscall beats a read loop. It was kept anyway,
because the moment a repo contains one large tracked file the result inverts
hard: on the mixed tree benchmarked above it is both faster and 118x smaller.
Bounded memory loses slightly on the tidy case and wins enormously on the untidy
one.
Both were caught by keeping the old implementation next to the new one in the benchmark, so a claimed speedup has to survive being measured.
The CI half¶
Timings don't belong in CI, so the regressions that matter are pinned as invariants that hold on any machine, at any load:
tests/test_perf.py— the ignore patterns are compiled once per tree walk, the tracked listing is read from git once per workdir, a scoped run never enumerates the tree at all.extensions/vscode/src/test/perf.test.ts— one path resolution per distinct file, one board walk per repaint, one bulk diagnostic write per publish.
Each ends with a deliberately loose complexity tripwire (25k paths, 20k findings) sized so that only a genuine change of algorithmic shape can trip it.