formalmethods.dev

§ 04 · take it with you

The doing-guide.

The operating manual: write a property, pick the instrument by question, read a verdict without lying to yourself. One tools/call away for agents.

SKILLS.md — doing formal methods with the lab

The one idea

Testing with formal methods = write the obligation once, let machines generate and judge the cases.

obligation (LTL)  →  cases (runs)  →  verdict (holds / violated / incomplete)

Everything below is a recipe for one of those three steps.

Step 1 — Write the property

LTL grammar (as this lab parses it):

p ::= atom | true | false | !p | p && p | p || p | p -> p | p <-> p
    | X p | G p | F p | p U p | p W p | p R p      (atoms: [A-Za-z_][A-Za-z0-9_]*)

Precedence loosest→tightest: <->, ->, ||, &&, then U W R (left-assoc), then ! X G F.

Property patterns and when to reach for them:

You want to sayWriteKind
nothing bad ever happensG !badsafety
something good eventually happensF good / G F goodliveness
every request is answeredG (request -> F response)response
grant only when readyG (grant -> ready)invariant
after grant, ready next stepG (grant -> X ready)step
p holds until q takes overp U q (must happen) / p W q (may never)precedence
alternate foreverG (a -> X (!a U b))rhythm

Rules of thumb:

  • One obligation per formula. If you wrote && between two independent thoughts, you will get two tools' worth of value by checking them separately.
  • Say when, not how. F response — not retry then response. The model owns "how"; the property owns "when".
  • If you find yourself writing G (!a && !b && ...), ask whether G !(a || b || ...) or an auxiliary atom reads better. Future-you reads these.

Step 2 — Pick the instrument

The four tools answer four different questions. Pick by question, not by habit:

QuestionTool / instrumentCost
Does this exact run satisfy the property?check_ltl (lab A)cheap, finite semantics (LTLf)
Does every possible run of the model satisfy it?model_check (lab B)exhaustive over finite models
Can random runs break it?falsify_property (lab C)cheap, shrinks what it finds
Which states/transitions did my tests never touch?analyze_coverage (lab D)cheap, negative metric

Decision recipe:

  • You have a log/test trace and a property → check_ltl. Truth table tells you which step broke it.
  • You have a model (states, transitions, labels) → model_check. Counterexample = a ready-made failing test.
  • Model too big to check? Bound the depth (model_check with depth) — bounded model checking: violations are real, "incomplete" is honest.
  • No model, just a spec you distrust → falsify_property over the atoms. A surviving campaign is evidence, not proof.
  • Wrote tests against a model → analyze_coverage with the traces. It proves gaps, never correctness.
  • Don't know the vocabulary → explain_concept (no args: index; with concept: chapter).
  • Need the whole manual → read_skills_guide (returns this very file, verbatim).
  • Coming from C / Python / Rust → read the matching chapter under Approaches (/approaches): native code, the artifact, and which tool works it.

Step 3 — Read the verdict honestly

Every tool labels exactly what it proved. Quote it verbatim; never upgrade:

  • `holds` — only from exhaustive enumeration (depth = every state, no truncation). Means: this model cannot violate it. As strong as the model is faithful.
  • `violated` — sound even when truncated: the counterexample lasso is a genuine failing run. Turn it into a regression test.
  • `incomplete` — depth or path budget stopped the search. It means unknown, not passing.
  • `not falsified` — random traces stayed lucky. Evidence, not proof.

A failing fuzzer campaign plus a holds model check on the same formula is the dream result: the machine found nothing you can't explain, and the exhaustive search found nothing at all.

Step 4 — Build the model (when you need one)

Kripke structure, as JSON:

{
  "atoms": ["ready", "coin", "dispense"],
  "initial": ["idle"],
  "states": [
    { "id": "idle",  "atoms": ["ready"],    "next": ["paid"] },
    { "id": "paid",  "atoms": ["coin"],     "next": ["vend"] },
    { "id": "vend",  "atoms": ["dispense"], "next": ["idle"] }
  ]
}
  • atoms = the vocabulary; label each state with the ones true there.
  • next = may-transitions: nondeterminism is a feature — checking the structure checks every resolution (that's the adversary).
  • A state with no next stutters forever — that's how F x catches hangs.
  • Strip everything irrelevant: no variables, no timers, only what the property mentions. If the model is big, it's wrong.

Calling over MCP

POST /api/mcp — stateless JSON-RPC 2.0 (Streamable HTTP). No session handshake:

curl -s https://formalmethods.dev/api/mcp \
  -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call",
       "params":{"name":"check_ltl",
                 "arguments":{"formula":"G (request -> F response)",
                              "trace":[["request"],["timeout"]]}}}'
  • initialize and tools/list are answered but nothing is remembered.
  • Bad args → JSON-RPC error -32602; a tool that ran and failed → result.isError: true.
  • Read result.structuredContent (agents) or result.content[0].text (humans).
  • Seed everything: falsify_property returns the seed it used — record it in the bug report.

Recipes

Spec-first feature: write property → model_check against the model → take counterexamples as tests → analyze_coverage → ship the model + property + generated traces in the repo.

Triage a flaky bug: write the property you believe → falsify_property with a fixed seed → shrink output is your minimal reproduction → check_ltl on the shrunken trace shows exactly which step sins.

Check a log/CI run: same property, check_ltl over the trace of events — runtime verification without a monitor.

Hunt liveness bugs: G F something + a model with a stuttering dead end. If the checker says violated with a lasso, the loop in the counterexample is the hang.

Honesty checklist (before you report a result)

  • Named the semantics: LTLf (finite) or infinite (lasso)?
  • Quoted the tool's verdict verbatim — no upgrades?
  • Seeded and recorded every random campaign?
  • Turned every counterexample into a kept test?
  • Labelled bounded/unfalsified results as evidence, not proof?
If you caught yourself saying "it passes" about an incomplete or not falsified result — stop, rerun, say what actually happened. The lab is built so you never have to lie about a verdict; use that.