Which Tests Cover What

TraceCov can tell you which tests cover each part of your API. It uses labels: names attached to the requests your tests send. The pytest plugin labels every request with the name of the test that sent it, so you get this with no extra setup.

Professional Edition

Labels require TraceCov Professional. The Community edition ignores them, shows a TracecovEditionWarning, and rejects --tracecov-fail-under-label at startup.

See coverage per test

Set up the pytest plugin, then write the JSON report:

$ pytest --tracecov-format=json

The report has a labels section with one entry per test, named by its pytest node id:

$ jq '.labels | keys' tracecov-report.json
[
  "tests/test_orders.py::test_list[10]",
  "tests/test_orders.py::test_refund"
]

Each entry holds that test's coverage, in the same shape as the report's summary. Each parametrized case is a separate test, and so is each Schemathesis test that runs under pytest.

Group tests under your own labels

Add labels with the tracecov marker. Put it on a test, a class, or a whole module:

import pytest

pytestmark = pytest.mark.tracecov(labels=["team:orders"])


@pytest.mark.tracecov(labels=["PAYMENT-42"])
def test_refund(api_client):
    api_client.post("/orders/1/refund")

test_refund now has three labels: its node id, team:orders and PAYMENT-42. The team:orders entry in the report covers every test in the module.

Fail CI when a label's coverage drops

--tracecov-fail-under-label LABEL:DIMENSION=N fails the run when a label covers less than N percent of DIMENSION:

$ pytest --tracecov-fail-under-label team:orders:operations=100

DIMENSION is operations, parameters, keywords, examples or responses. Repeat the option for more rules, or list them in pyproject.toml:

[tool.pytest.ini_options]
tracecov_fail_under_label = [
    "team:orders:operations=100",
    "team:orders:parameters=80",
]

A label with no recorded requests fails its rule, so a typo in a label name cannot pass. Failures print next to the global ones:

------------------------- tracecov threshold failures --------------------------
team:orders: Parameters       50.0% / 80% required
PAYMENT-7: no traffic recorded under this label

Ask which labels covered an element

In Python, covered_by() lists the labels that covered one element of the spec:

coverage.covered_by(
    "GET",
    "/orders",
    location="query",
    name="limit",
    keyword="/minimum",
)
# [
#     {"label": "PAYMENT-42", "reached": "invalid"},
#     {"label": "team:orders", "reached": "both"},
# ]

reached says what the label's requests sent: values that pass the keyword ("valid"), values that break it ("invalid"), or both.

To ask about a whole operation, pass only the method and path. To ask about a response, pass status, plus keyword for a keyword in its body. Add media_type when the response declares more than one.

label_summary("team:orders") returns one label's coverage, or None if the label recorded nothing. label_summaries() returns all of them. check_thresholds(label=...) checks one label, like the pytest option.

Label requests outside pytest

In your own code, wrap requests in tracecov.labeled(). A nested block adds its labels to the outer ones:

import requests
import tracecov

coverage = tracecov.CoverageMap.from_path("openapi.json")
session = coverage.requests.track_session(requests.Session())

with tracecov.labeled("team:orders"):
    session.get("https://api.example.com/orders?limit=10")
    with tracecov.labeled("PAYMENT-42"):
        # Labelled team:orders and PAYMENT-42
        session.get("https://api.example.com/orders?limit=0")

To label a single recording, pass labels to record():

coverage.record(
    tracecov.HttpRequest(method="GET", url="https://api.example.com/orders"),
    tracecov.HttpResponse(status_code=200, elapsed=0.1),
    labels=["smoke"],
)

Through tracecov proxy, send the X-TraceCov-Labels header with comma-separated labels. The proxy removes it before forwarding:

curl -H "X-TraceCov-Labels: team:orders, PAYMENT-42" \
     http://localhost:9000/orders

Choose label names

A label is any string. A kind:value prefix keeps different kinds of labels apart:

Kind	Example
Owning team	`team:orders`
Feature	`feature:checkout`
Ticket	`PAYMENT-42`
Suite	`suite:smoke`

Colons are fine anywhere. Avoid commas: the proxy header splits on them.

Troubleshooting

A label is missing from the report, or label_summary() returns None:

The request came from a shared fixture. A fixture with a scope wider than function serves many tests, so its requests carry no test label.
The request came from a thread you started. Labels do not pass into new threads. Run the thread's work through contextvars.copy_context().run. Asyncio tasks keep labels.
You run the Community edition. Look for TracecovEditionWarning in the pytest warnings.
Labels are off. Check for --tracecov-no-labels or tracecov_no_labels = true.
The traffic was imported. --traffic-output files and HAR, Postman or VCR imports carry no labels.

Limits

Labels appear in the JSON report and the Python API. The HTML, Markdown and text reports do not show them.
A label records whether it covered each element, not how many times. The overall totals keep the counts.
Labels never change the overall totals: every request counts, labelled or not.
A map holds up to 100,000 distinct labels. TraceCov drops new labels past that; label_overflow() counts each one dropped, once per request that carried it.