Coverage Depth

Martín-López, Segura and Ruiz-Cortés define ten coverage criteria for RESTful APIs and stack them into eight Test Coverage Levels, TCL0 through TCL7 (A-TEST 2019). It is the reference framework for answering "how thoroughly was this API tested?" without reading the source of the API. Restats, the tool built to compute it, implements eight of the ten criteria - response body properties and operation flows are the two it omits - and its level calculation returns at most 6.

TraceCov measures every criterion except operation flows, and adds one the model does not define: coverage per JSON Schema keyword.

The ladder

Each level subsumes the ones below it.

Level	The model requires	In TraceCov
TCL1	Every path requested	An operation's hits; a path is reached once any operation on it is. There is no separate path percentage.
TCL2	Every operation requested	The operations summary.
TCL3	Every declared request and response content type exercised	One row per declared media type on both sides, each with its own hits.
TCL4	Every parameter used, and both a success and an error response per operation	The parameters summary; the success/error split reads off the per-operation response rows.
TCL5	Every value of every boolean and enum parameter, and every documented status code	Enum values individually, one row per value in the HTML schema view; the responses summary counts every documented status, including 4XX wildcards and default.
TCL6	Every property of every response body obtained	The response_keywords summary: every keyword of the response schema, marked reached or not. An optional property the server never returns leaves its keywords unreached.
TCL7	Sequences of operations - create, then read, update and delete the resource	Not measured. On the roadmap.

Past the top of the ladder

The model's finest-grained input criterion is the parameter value, and only for booleans and enums. Under it, {"type": "string", "minLength": 8, "pattern": "^[a-z]+$"} is fully covered by one request carrying any string at all.

TraceCov's unit is the keyword: minLength and pattern are each tracked, and each is tracked twice - once for input that satisfies it, once for input that violates it. A suite that only ever sends well-formed data reaches the same TCL as one that probes every boundary, and a different keyword percentage. See Keywords.

The response side is bidirectional. The model asks whether a property was obtained; TraceCov also validates the body against the response schema, so a maxLength the server broke is reported as violating rather than merely covered.

Why a keyword denominator holds up

Every criterion in the model has a denominator the specification hands you: count the paths, the operations, the parameters, the enum values, the documented status codes. A per-keyword denominator invites the obvious objection - many keywords cannot be exercised both ways, so 100% would be unreachable and the percentage meaningless. minLength: 0 admits no string that fails it. {"not": {}} admits no value that passes.

TraceCov classifies each keyword as Normal, Positive or Negative and asks only for the cases that exist: both for Normal, valid only for Positive, invalid only for Negative (Satisfiability). The denominator holds only achievable demands, which keeps 100% reachable and the number comparable between schemas.

References

• Alberto Martín-López, Sergio Segura, Antonio Ruiz-Cortés. Test Coverage Criteria for RESTful Web APIs. A-TEST 2019. doi:10.1145/3340433.3342822
• Davide Corradini, Amedeo Zampieri, Michele Pasqua, Mariano Ceccato. Restats: A Test Coverage Tool for RESTful APIs. ICSME 2021. doi:10.1109/ICSME52107.2021.00063