garak LLM Scanner: Test Models for Prompt Injection

What the garak LLM scanner does

garak is an open-source vulnerability scanner for large language models, developed by NVIDIA. Think of it as nmap or Nessus for a model: instead of probing ports, it sends batches of adversarial prompts and checks whether the model’s replies show a weakness such as a successful prompt injection, a jailbreak, leaked training data or toxic output.

If you build anything on top of an LLM, running the garak LLM scanner against it before you ship gives you a repeatable, scriptable baseline rather than a gut feeling.

Only scan models you own or are explicitly authorised to test. Probing someone else’s hosted model can breach its terms of use, and every probe against a paid API costs you tokens.

Install garak

garak is a Python package. Install it into its own environment so its dependencies do not collide with anything else:

python3 -m venv garak-env
source garak-env/bin/activate
python -m pip install -U garak

Or use pipx install garak if you prefer a global command. Confirm it works by listing the available probes:

python -m garak --list_probes

The four moving parts

• Generators are the target: a Hugging Face model, an OpenAI-compatible API, a local runtime and so on.
• Probes are the attacks: each one sends a family of prompts, such as encoded instructions, DAN-style jailbreaks or prompt injection.
• Detectors read the replies and decide whether the attack worked.
• Buffs optionally mutate the prompts, for example by paraphrasing them, to widen coverage.

Run your first scan

A quick, free test against a small local Hugging Face model is a good way to see the output without spending anything:

python -m garak --model_type huggingface --model_name gpt2 --probes encoding

To test a hosted model through an OpenAI-compatible API, set your key and pick the probes you care about, such as prompt injection:

export OPENAI_API_KEY="your-key-here"
python -m garak --model_type openai --model_name gpt-4o-mini --probes promptinject

Start with one probe family at a time. A full default run sends a very large number of prompts, which is slow and, on a paid API, expensive.

Read the results

garak prints a line per probe and detector as it works, ending in a pass or fail count. A fail means the attack succeeded, which is the finding you care about. It also writes a report log in JSON Lines format and an HTML summary into its output directory, and the end of the run tells you where.

• Look at which probe families fail most. A model that falls to encoding tricks but resists plain jailbreaks needs different mitigations from one that fails the other way round.
• Treat a pass as ‘not broken by these prompts’, not as proof the model is safe.
• Re-run after any change to your system prompt, guardrails or model version so you can compare.

Where this fits in a wider test plan

Cost and safety tips

A scan against a hosted model is billed per token, so cap the scope. Pick one probe family, use the cheapest model that is representative of what you ship, and check your provider’s usage dashboard after the first run. Log files from a scan contain the prompts and the model’s replies, which may include offensive or sensitive text, so store them like any other test artefact.

Run scans in a staging environment, never against a production assistant that real users depend on.

garak tests the model. It does not test the application around it, such as tool permissions, retrieval sources or output handling. Pair it with a manual review, using the prompt injection pentest checklist, and look at the defences covered in LLM prompt injection defense tools.