.. _crash-report-spec-chapter: ======================================= Specification: Submitting crash reports ======================================= .. contents:: :local: Summary ======= The Breakpad library includes an `HTTP upload tool `_. However, Mozilla doesn't use that tool for its products. Instead, we have our own set of crash reporting clients. This specification covers the HTTP POST request and response for submitting crash reports to the Socorro collector. History ======= History: * 2020-01-15: Initial writing * 2020-01-17: Add specifying annotations as a single JSON-encoded value. * 2021-05-26: Add notes about where we stray from the Breakpad crash reporting client. * 2021-12-20: Generalize to cover HTTP POST request and response for the Socorro collector. Crash reporter client submission request ======================================== Request method -------------- Crash reports are submitted by HTTP POST to the Socorro collector crash submission URL. Request headers --------------- The HTTP POST request *must* include the following headers: ``Content-Type`` The content type header of the crash report must be either ``multipart/form-data`` or ``multipart/mixed`` and it must specify the multipart boundary. ``Content-Length`` The content length of the crash report must be set. The value is the length of the body. The HTTP POST request *can* include the following headers: ``User-Agent`` (optional) A user agent string which includes the product name, version, and "crash reporter". .. Note:: Including a user agent makes it easier for us to help you debug crash reporter issues. ``Content-Encoding`` (optional) If the HTTP body is gzipped, then the ``Content-Encoding`` must be set to ``gzip``. Generally, it's good to compress bodies since it reduces the size of the body, reduces the time it takes to upload, and also increases the size of the crash reports you can send. Request body ------------ The HTTP POST payload contains the crash report which is comprised of crash annotations and minidumps. .. Note:: If the ``Content-Encoding`` header is set to ``gzip``, the body must be gzipped. Crash annotations ~~~~~~~~~~~~~~~~~ The body consists of a sequence of multipart fields. Each field is either an annotation or a binary like a minidump. .. seealso:: RFC for multipart/form-data and multipart/mixed: https://tools.ietf.org/html/rfc7578 Crash annotations can be provided in one of two ways: 1. annotations as individual key/value pairs, OR 2. annotations as a single JSON-encoded value .. Note:: You can provide crash annotations as EITHER key/value pairs, OR a JSON-encoded value for all annotations. If your crash report includes both, it will be rejected with HTTP 400 and:: Discarded=malformed_has_json_and_kv Annotations as key/value pairs in form-data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. The ``Content-Disposition`` must be ``form-data``. 2. The ``Content-Disposition`` must specify a ``name``. This is the annotation name. The value must consist of ASCII characters. 3. The value of this field is the annotation value. It is always a string. Example:: Content-Disposition: form-data; name="DOMFissionEnabled" 1 Annotations as single JSON-encoded value ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. The ``Content-Disposition`` must be ``form-data``. 2. The ``Content-Disposition`` must specify a ``name``. The value should be ``extra``. 3. The ``Content-Type`` must be ``application/json``. 4. The value of this field must a JSON-encoded object of all of the crash report annotations. All annotation keys and values must be strings. If the annotation value is an object, it must be JSON-encoded. Because the annotation value is JSON-encoded object and it is itself in a JSON-encoded object, quotes must be escaped in the final field value. 5. There must be only one "extra" field in the payload. Example:: Content-Disposition: form-data; name="extra" Content-Type: application/json {"ProductName":"Firefox","Version":"1.0","TelemetryEnvironment":"{\"build\":{\"applicationName\":\"Firefox\",\"version\":\"72.0.1\",\"vendor\":\"Mozilla\"}}"} .. versionadded:: 2020-01-17 This was added in `bug 1420363 `_. That work landed in December 2019 and is in Firefox 73. Minidumps and other binary data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. The ``Content-Disposition`` must be ``form-data``. 2. The ``Content-Disposition`` must specify a ``name``. The name must consist of ASCII characters. Examples of names: * ``memory_report`` * ``upload_file_minidump`` * ``upload_file_minidump_browser`` * ``upload_file_minidump_content`` 3. The ``Content-Disposition`` may specify a ``filename``. Examples of filenames: * ``6da3499e-f6ae-22d6-1e1fdac8-16464a16.dmp`` 4. For minidumps, the ``Content-Type`` must be ``application/octet-stream``. For memory reports, the ``Content-Type`` must be ``application/gzip``. 5. The value of this field is binary data. Example:: Content-Disposition: form-data; name="upload-file-minidump"; filename="6da3499e-f6ae-22d6-1e1fdac8-16464a16.dmp" Content-Type: application/octet-stream BINARYDATA .. Note:: The Socorro processor treats the file with the name ``upload_file_minidump`` as the minidump of the crashing process. It extracts information from it and that's what shows up on Crash Stats. If you're writing your own crash reporter client, you should make sure to set the name for the minidump of the crashing process as ``upload_file_minidump``. Collector response ================== The collector throttles crash reports and returns a response to the crash reporter client in the HTTP response. HTTP 200 Crash report submission was successful. Accepted: If the crash report is accepted by the collector, then the collector must return an HTTP status code of 200 with a body specifying the crash id:: "CrashID" "=" CRASHID For example:: CrashID=bp-d101d046-638f-42e0-902d-bd245c200115 .. Note:: It's possible for a crash report to be accepted by the collector, but be malformed in some way. For example, if one of the annotation values was ``null``. The processor has rules that will fix these issues and add processor notes for what it fixed. Rejected: If the crash report is rejected by the collector, then the collector must return an HTTP status code of 200 with a body specifying the rejection rule:: "Discarded" "=" RULE For example:: Discarded=rule_has_hangid_and_browser Rejection rules are specified in the collector's throttler. They change periodically. Some rejection rules are hard-rejections and the collector will never accept the crash report. Some rejection rules are soft-rejections from sampling and the collector may accept that crash report again in the future. The crash reporter client may submit the crash report again. .. seealso:: Code for throttler: https://github.com/mozilla-services/antenna/blob/main/antenna/throttler.py HTTP 400 If the crash report is malformed, then the collector must return an HTTP status code of 400 with a body specifying the malformed reason:: "Discarded" "=" REASON For example:: Discarded=malformed_no_annotations Non-exhaustive list of reasons the crash report could be malformed: ``malformed_no_content_type`` The crash report HTTP POST has no content type in the HTTP headers. ``malformed_wrong_content_type`` The crash report HTTP POST content type header exists, bug it's not set to ``multipart/form-data`` or ``multipart/mixed``. ``malformed_no_boundary`` The content type doesn't include a boundary value, so it can't be parsed as ``multipart``. ``malformed_bad_gzip`` The ``Content-Encoding`` header is set to ``gzip``, but the body isn't in gzip format or there's a parsing error. ``malformed_invalid_json`` The payload included a part named ``extra``, but the value wasn't valid JSON. ``malformed_invalid_json_value`` The payload included a part named ``extra``, but the value wasn't an object--it was some other type (str, int, ...) ``malformed_invalid_annotation_value`` The payload included a ``text/plain`` part which had a value that wasn't utf-8. ``malformed_has_json_and_kv`` The crash report encodes annotations as ``form-data`` fields as well as in an extra JSON-encoded object. It should have either one or the other--not both. ``malformed_no_annotations`` The crash report has been parsed, but there were no annotations in it. ``malformed_invalid_payload_structure`` The payload was malformed in some way: missing EOL sequences between parts, missing part boundaries, malformed boundary, malformed end sequence, etc. The crash reporter client shouldn't try to send a malformed crash report again. HTTP 413 The HTTP POST body is too large and exceeds the maximum body size. The crash reporter client shouldn't try to send this crash report again. HTTP 500 This is an internal server error. It's possible this is a bug in the collector. If so, an error report gets sent and maintainers will see it. It's possible this problem is ephemeral and will go away after some time. The crash reporter client may sleep for a bit and retry sending the crash report. HTTP 502 Bad gateway. It's possible this problem is ephemeral and will go away after some time. It's possible that this is a bug in the crash reporting client. The crash reporter client may sleep for a bit and retry sending the crash report. HTTP 503 Service unavailable. It's possible this problem is ephemeral and will go away after some time. The crash reporter client may sleep for a bit and retry sending the crash report. Example of crash report HTTP POST ================================= Example with HTTP headers and body:: POST /submit HTTP/1.1 Host: xyz.example.com User-Agent: Breakpad/1.0 (Linux) Accept: */* Content-Length: 1021 Content-Type: multipart/form-data; boundary=------------------------c4ae5238f12b6c82 --------------------------c4ae5238f12b6c82 Content-Disposition: form-data; name="Add-ons" ubufox%40ubuntu.com:3.2,%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D:48.0,loop%40mozilla.org:1.4.3,e10srollout%40mozilla.org:1.0,firefox%40getpocket.com:1.0.4,langpack-en-GB%40firefox.mozilla.org:48.0,langpack-en-ZA%40firefox.mozilla.org:48.0 --------------------------c4ae5238f12b6c82 Content-Disposition: form-data; name="DOMFissionEnabled" 1 --------------------------c4ae5238f12b6c82 Content-Disposition: form-data; name="BuildID" 20160728203720 --------------------------c4ae5238f12b6c82 Content-Disposition: form-data; name="upload_file_minidump"; filename="6da3499e-f6ae-22d6-1e1fdac8-16464a16.dmp" Content-Type: application/octet-stream 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 --------------------------c4ae5238f12b6c82-- Example with HTTP headers and body using JSON-encoded value for annotations:: POST /submit HTTP/1.1 Host: xyz.example.com User-Agent: Breakpad/1.0 (Linux) Accept: */* Content-Length: 659 Content-Type: multipart/form-data; boundary=------------------------c4ae5238f12b6c82 --------------------------c4ae5238f12b6c82 Content-Disposition: form-data; name="extra" Content-Type: application/json {"ProductName":"Firefox","Version":"1.0","BuildID":"20160728203720"} --------------------------c4ae5238f12b6c82 Content-Disposition: form-data; name="upload_file_minidump"; filename="6da3499e-f6ae-22d6-1e1fdac8-16464a16.dmp" Content-Type: application/octet-stream 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 --------------------------c4ae5238f12b6c82-- Differences from Breakpad upload tool ===================================== The Breakpad library comes with an upload tool. That tool lets you upload crash annotations and dumps as an HTTP POST to a collector. It does not support the following things in this specification: 1. returning crash id on successful submission 2. returning rejection code on rejected crash report 3. crash annotations as a single JSON-encoded value How to debug problems with submitting crash reports =================================================== We all hang out in `#crashreporting matrix channel `_. Here are some notes for issues you might be having: **I'm getting back an HTTP 404** The URL you're using is wrong. Verify the url. If that doesn't work, reach out to us. **I'm getting back an HTTP 413** The crash report request body is too large. If you aren't compressing it with gzip, try that. If you are, then reach out to us but you're probably going to need to remove something from the crash report to reduce the size. **I'm getting back a rejection** Check the response body for the rejection code and look it up in the throttling rules: https://github.com/mozilla-services/antenna/blob/main/antenna/throttler.py If that doesn't help, reach out to us. **I'm getting back an HTTP 500** Reach out to us because something is wrong with our server. **The crash report submitted, but there's very little data on Crash Stats** Verify that the name (not the filename) is set to "upload_file_minidump". The Socorro processor treats that specific minidump as the one for the crashed process and does additional processing for it. You can see a list of all the dumps that were sent in the Debug tab of the report view on Crash Stats. Verify that you're sending all the crash annotations you're intending to send. You can see a list of all the dump names and crash annotations in the ``crash_report_keys`` field of the processed crash. If this is a new crash annotation or one that's not explicitly marked as public, the crash annotation will be treated as protected data. If you don't have access to protected data, you will not be able to see it on Crash Stats. See our protected data access policy: https://crash-stats.allizom.org/documentation/protected_data_access/ **None of these are helping me** Ask yourself these questions and see if they help you at all: 1. When the crash reporter client submits the crash report to Socorro, what is the status code that it gets back? What is the HTTP response body? 2. If you successfully submit a crash report, search for the crash id on Crash Stats. Are there processor notes indicating problems? If nothing here helps please reach out to us in the `#crashreporting matrix channel `_.