Breakpad is a set of client and server components that implement a crash reporting system.
Mozilla currently uses a heavily patched Breakpad library.
Mozilla is working on replacing Breakpad and other tools with new ones written in Rust.
- crash annotation¶
A crash annotation is a key/value pair with some bit of information.
Crash annotations are one of two kinds of data in a Breakpad-style crash report the other being minidumps.
In a crash report, all values are strings, but they can be decoded as a variety of data types like strings, integers, floats, and JSON-encoded structures.
MozCrashReason=Shutdown hanging at step AppShutdownConfirmed. Something is blocking th e main-thread.
- crash id¶
A unique identifier for the crash report. The collector generates the crash id and returns it to the crash reporter in the HTTP response. The crash reporter may keep it so that users have a list of crash reports they submitted with links to the the crash report on Crash Stats.
Crash ids look like this:
de1bb258-cbbf-4589-a673-34f800160918 ^^^^^^^ ||____| | yymmdd | throttle result
Historically, Antenna used the throttle result to encode the throttle result so that we knew which crash reports were collected and accepted for processing and which ones were accepted and stored, but not accepted for processing. Antenna no longer accepts crash reports for storing, but not for processing, so this character is now always
Because of this structure, you can look at the last 6 characters and know when the crash report was collected.
The crash id is also referred to as “uuid”. The collector stores the crash id in the “uuid” key in the raw crash.
(Deprecated) The crash id also used to be referred to as “ooid”, but that term is deprecated and we should remove its usage everywhere.
- crash report¶
Socorro doesn’t accept all incoming crash reports. The collector has a throttler which rejects some crash reports.
Crash reports get rejected for a variety of reasons:
The crash report is malformed in some fundamental way.
The crash report is from a cohort we get many crash reports from and we don’t need them all. For example, we only accept 10% of Firefox desktop, release channel, Windows crash reports.
The crash report contains something we cannot accept. There have been bugs in the crash reporter in the past where it sent crash reports the user did not consent to. We reject crash reports that have the markers of these bugs.
- Specification: Submitting crash reports
Specification covering the structure of a crash report.
- crash reporter¶
When a product crashes, the crash reporter kicks in, captures information about the state of the crashed process, product, and system, and assembles a crash report.
- crash signature¶
The Socorro processor generates a crash signature for every crash report.
Crash signatures help us group similar crashes glossing over differences in operating system versions, platforms, architectures, drivers, video cards, web sites, etc.
Roughly, a signature consists of some flags followed by “interesting” symbols from the stack.
Examples of signatures:
OOM | small shutdownhang | nsThreadManager::SpinEventLoopUntilInternal mozilla::dom::ServiceWorkerRegistrar::GetShutdownPhase <style::stylesheets::rules_iterator::RulesIterator<C> as core::iter::traits::iterator::Iterator>::next
- Signature Generation
Documentation on Socorro’s signature generation.
A minidump is a file format for storing information about a crashed process. It contains CPU information, register contents, stacks for the crashed thread and other threads, some interesting parts of the heap, list of loaded modules, list of unloaded modules, etc.
Minidumps are smaller than core dumps which makes them handy for crash reporting.
Minidumps are created and manipulated using the Breakpad library and the rust-minidump tools.
- Minidump Files (Microsoft)
Documentation on minidump file format.
- Breakpad: minidump file format
Breakpad documentation on minidump file format.
- Breakpad: processing minidumps
Breakpad documentation on processing minidumps.
Type definitions, parsing, and analysis for the minidump file format.
- processed crash¶
The Socorro processor takes crash annotations and minidumps, runs them through the processing pipeline, and generates a processed crash.
The processed crash contains normalized and validated data derived from the original crash report.
- protected data¶
Socorro categorizes crash report data in two ways:
Public data is anything in Category 1 (technical data) and Category 2 (interaction data).
Protected data is anything more sensitive than that.
By default, all data in the raw and processed crash is considered protected. In order for it to be marked as public, we require a data review and for it to be explicitly marked as public in the relevant schema.
- raw crash¶
The Socorro collector parses the HTTP POST payload into a set of crash annotations and minidumps. It collects the crash annotations along with some metadata generated at collection in a raw crash structure. It saves this to AWS S3.
The Socorro processor takes the raw crash and minidumps and passes them through the processing pipeline to generate a processed crash.
The collector tries to save the crash annotation data in the raw crash as it received it. There are some exceptions:
Some crash annotations are no longer allowed to be collected. The collector will drop these before creating the raw crash. For example, we no longer collect the
Annotations that raise some kind of parse error are dropped.
When this happens, a note will be added to the “collector notes” which can be seen in the Debug tab of the report view in Crash Stats.
Socorro can reprocess crash reports. Reprocessing involves starting with the original crash data and running it through the processing pipeline again.
Sometimes we reprocess crash reports after we’ve made changes to signature generation so that the crash reports pick up new crash signatures.
Sometimes we reprocess crash reports after we’ve uploaded missing symbols files so minidump processing has symbols files to work with. This results in improved symbolicated stacks and new crash signatures.
Sometimes we reprocess crash reports that were affected by a bug we’ve just fixed in the processor.
A command-line minidump processor used by the Socorro processor to parse a minidump and generate a JSON-encoded digest of the minidump with symbolicated stacks, modules, hardware information, and other things.