Reconstructing OpenAPI Specifications from Captured Network Traces

Software engineers often encounter proprietary systems where the internal communication protocols are a complete black box. This lack of documentation creates significant hurdles for security auditing, third-party integration, and legacy system maintenance. Relying on guesswork to understand these interfaces leads to fragile implementations and undiscovered security vulnerabilities.

API reverse engineering is the systematic process of reconstructing the functional blueprint of an application programming interface from its observable behavior. By treating the API as an unknown state machine, we can map out its transitions, data structures, and constraints through controlled observation. This approach transforms raw network noise into a structured contract that developers can rely upon for future development.

The primary objective is to move from reactive troubleshooting to proactive design by creating a definitive source of truth. Structured documentation allows for the generation of client SDKs, the implementation of automated testing suites, and the identification of unintended data exposures. It essentially bridges the gap between what a system is perceived to do and what it actually does in production.

An undocumented API is a security liability that behaves exactly like a hidden backdoor until its surface area is fully mapped and understood.

The foundation of any reverse engineering effort is a high-quality capture of representative network traffic. Tools like Mitmproxy or Burp Suite act as man-in-the-middle proxies that intercept HTTP requests and responses between the client and the server. This allows you to inspect headers, query parameters, and request bodies in their raw format before they are processed by the application logic.

However, raw traffic often contains sensitive information such as session tokens, personal identifiers, and environment-specific keys. Before this data can be used to generate documentation, it must be sanitized to remove private details while preserving the structural integrity of the payloads. This sanitization process ensures that the resulting documentation is safe to share across development teams and does not leak security credentials.

1from mitmproxy import http
2import json
3
4def response(flow: http.HTTPFlow) -> None:
5    # Define a set of sensitive keys to redact from JSON responses
6    sensitive_keys = {"session_id", "auth_token", "api_key", "email"}
7    
8    if flow.response.headers.get("content-type") == "application/json":
9        try:
10            data = json.loads(flow.response.get_text())
11            # Recursively redact values to maintain schema structure
12            sanitized_data = redact_sensitive_info(data, sensitive_keys)
13            flow.response.set_text(json.dumps(sanitized_data))
14        except json.JSONDecodeError:
15            pass
16
17def redact_sensitive_info(data, keys):
18    if isinstance(data, dict):
19        return {k: "[REDACTED]" if k in keys else redact_sensitive_info(v, keys) for k, v in data.items()}
20    elif isinstance(data, list):
21        return [redact_sensitive_info(item, keys) for item in data]
22    return data

The logic above demonstrates how to hook into the proxy lifecycle to modify data on the fly. By replacing sensitive values with placeholders, we maintain the key names and data types which are essential for schema generation. This approach allows the downstream documentation tools to correctly identify that a specific field is a string or a number without exposing the actual production values.

Once you have a collection of cleaned traffic logs, the next step is to aggregate these individual transactions into a cohesive API specification. This involves grouping requests by their endpoint paths and identifying the common patterns in their request and response cycles. Manual conversion is tedious and error-prone, so automation is highly recommended for any API with more than a few endpoints.

Utility tools can parse common proxy formats like HAR files or Mitmproxy flow files and convert them into OpenAPI definitions. These tools look for variations in the URL paths to identify dynamic parameters, such as user IDs or resource slugs. For example, requests to users/123 and users/456 are automatically collapsed into a single path template like users/{id}.

• Export traffic from the proxy tool in a standard format like HAR or JSON flow logs.
• Use a conversion utility to map HTTP methods and status codes to OpenAPI operations.
• Normalize URL paths by identifying recurring patterns that signify path parameters.
• Incorporate response headers into the schema to document caching behavior and custom metadata.
• Validate the generated YAML or JSON file against the OpenAPI specification standards.

The resulting schema is an initial draft that reflects the observed reality of the API. It provides a skeleton that includes the endpoints, methods, and basic data structures found in the traffic. While this draft is functional, it often requires manual refinement to account for edge cases and optional fields that were not captured during the initial observation period.

A generated schema is only as good as the traffic it was built from. If your capture session only included successful requests, your documentation will lack information about error codes and validation messages. To build a robust specification, you must intentionally trigger error states by sending malformed requests and observing how the API responds.

Refinement also involves moving beyond basic data types to define more specific constraints. For instance, a capture might show that a priority field is an integer, but manual analysis might reveal it only accepts values between one and five. Adding these constraints to your OpenAPI definition makes the documentation significantly more useful for developers who need to implement valid clients.

1paths:
2  /v1/orders/{order_id}:
3    get:
4      summary: Retrieve order details
5      parameters:
6        - name: order_id
7          in: path
8          required: true
9          schema:
10            type: string
11            format: uuid # Manually refined from generic string
12      responses:
13        '200':
14          description: Success
15          content:
16            application/json:
17              schema:
18                $ref: '#/components/schemas/Order'
19        '404':
20          description: Order not found
21          content:
22            application/json:
23              schema:
24                $ref: '#/components/schemas/ErrorResponse'

By manually refining the schema as shown in the example, you transform a generic log into a high-fidelity technical contract. Specifying formats like UUID or providing detailed error response models allows for better validation and improved developer experience. This manual step is where the institutional knowledge of the system is codified into the documentation.

Beyond documentation, reverse engineering APIs serves as a powerful method for security auditing. Once you have a full map of the API, you can look for logical flaws such as Broken Object Level Authorization (BOLA). This occurs when an API allows a user to access resources belonging to another user by simply changing an ID in the request URL.

A structured schema allows you to run automated security scanners against the API with much higher coverage. These scanners use the schema to understand the expected inputs and then systematically inject payloads designed to trigger SQL injection, cross-site scripting, or buffer overflows. Without a schema, these tools are often limited to basic fuzzing which might miss deep-seated logic vulnerabilities.

Finally, reverse engineering provides insights into the data leakage risks of a system. You might discover that certain endpoints return much more data than the client actually displays, such as internal database IDs, timestamps, or even user metadata. Identifying these verbose responses is the first step toward implementing proper data filtering at the API gateway or application layer.