formalmethods.dev

§ 5.2 · Python · pytest + Hypothesis

Python

the property is a hypothesis; the runs are generated; the failure is shrunk.

Hypothesis inverts the usual test: you declare what a run may look like (st.lists(...)), and it invents thousands of them. The body of the test is the property — what must hold for every run.

Its signature move is shrinking: when a random case fails, it delta-debugs the input down to the smallest case that still fails — the same loop our fuzzer runs over traces.

in Python

def test_...(run): assert holds(run) — one property, every generated run

as LTL · expected violatedG (coin -> F dispense)
test_vend.py
# test_vend.py — the property is a hypothesis; runs are generated.
from hypothesis import given, strategies as st

events = st.sampled_from(["coin", "dispense", "timeout"])

@given(st.lists(events, max_size=12))
def test_coin_is_answered(run):
    owed = 0
    for e in run:
        if e == "coin":
            owed += 1
        elif e == "dispense":
            owed -= 1
        assert owed >= 0        # safety: no coin answered twice
    # "eventually answered" cannot be asserted here —
    # that half of the property belongs to the model checker.
the artifact · A failing run, after Hypothesis shrank it: one coin inserted, never answered. Replay the shrunken trace through check_ltl to see exactly which position sins — and through falsify_property to have the machine shrink it for you.
012pos
coin
timeout
exposed over mcpcheck_ltl
view the JSON-RPC body · POST to /api/mcp
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "check_ltl",
    "arguments": {
      "formula": "G (coin -> F dispense)",
      "trace": [
        [
          "coin"
        ],
        [
          "coin"
        ],
        [
          "timeout"
        ]
      ]
    }
  }
}
how the lab helps · skills guide + mcp against Python
read_skills_guide

The guide's triage a flaky bug recipe is the Python workflow written down: property → seeded falsification → shrunk reproduction → check_ltl on the shrunken trace. Agents follow it step by step.

falsify_property

A Hypothesis failing example is one draw of many. Agents re-run the property through falsify_property, record the returned seed, and the bug report carries a one-number reproduction.

analyze_coverage

Which of the model's transitions did the pytest suite never drive? Agents map test executions to state-id traces and let analyze_coverage list the gaps — Hypothesis targets aim there next.

Shrinking is Hypothesis's killer feature — ours is seeded (falsify_property returns the seed it used), so a shrunk counterexample is reproducible in a bug report.

Coverage-guided generation (Hypothesis targets) is the industrial version of the fuzzer in lab C.

field guide · Property-based testing & falsification · Coverage as a negative metric