During processing of a crash, Socorro creates a signature using the signature generation module. Signature generation typically starts with a string based on the stack of the crashing thread. Various rules are applied and after everything is done, we have a Socorro crash signature.
The signature generation code is here:
The lists for configuring the C signature generation class are here:
To request a change to signature generation:
Write up a bug in the Socorro product and please include the following:
explanation of what the problem you want to solve is
urls of examples of crashes that have the problem you’re trying to solve
expected signatures for those crashes
We need this to make sure we can help you make the right changes.
Examples of bugs:
If you’ve made changes to signature generation before or you’re confident in the change you’re making, you can make changes directly using the GitHub interface:
If you want to test your changes or experiment with them, then you’ll need to set up a local development environment and make the changes with a GitHub pull request.
See Local dev environment setup for setting up a local development environment.
Read through the rest of this chapter which describes how signature generation works, what files are involved, and how to test changes.
Make sure the PR has a corresponding bug in Bugzilla and references the bug in the commit summary.
This is important because signature generation is tricky and we need the historical data for what changes we made, for whom, why, and how it affected signature generation.
Verify there are no typos in the change.
We have a unit test that verifies there are no syntax errors in those files, but that (obviously) doesn’t cover typos.
Run the pull request changes through signature generation using the command line interface in your local dev environment. See Signature generation module.
Verify with the author that the changes occur as intended.
Merge the PR and verify the example crashes on -stage.
The easiest way to do that is to use Super Search and search for a signature. The most common change is an addition to the prefix list, in which case you want to search for the frame signature that was added, and verify that in recent signatures there is something following it.
If you don’t want to wait for new crash reports to arrive, you can find an existing one and send it to reprocessing. That can be done on the report/index page directly, or via the admin panel.
Note that after a signature change has been pushed to production, you might want to reprocess the affected signatures.
This Python module covers crash signature generation.
This module defines a command line interface for signature generation. Given a crash id, it pulls the raw and processed data from Socorro -prod, generates a signature using the code in this module, and then tells you the original signature and the newly generated one.
This can be used for testing signature generation changes, regression testing, and astounding your friends at parties.
You need to run this inside a Socorro environment. For example, you could run this in the processor Docker container. You can start a container like that like this:
$ make shell
Once you’re in your Socorro environment, you can run signature generation. You can pass it crash ids via the command line as arguments:
socorro-cmd signature CRASHID [CRASHID...]
It can also take crash ids from stdin.
getting crash ids from the file
$ cat crashids.txt | socorro-cmd signature
getting crash ids from another command:
$ socorro-cmd fetch_crashids --num=10 | socorro-cmd signature
spitting output in CSV format to more easily analyze results for generating signatures for multiple crashes:
$ cat crashids.txt | socorro-cmd signature --format=csv
For more argument help, see:
$ socorro-cmd signature --help
This folder contains lists that are used to configure the C signature generation
.txt file contains a list of signatures or regex matching
signatures, that are used at various steps of our algorithm. Regular expressions
use the Python syntax.
When generating a C signature, 5 steps are involved.
We walk the crashing thread’s stack, looking for things that would match the Signature Sentinels. The first matching element, if any, becomes the top of the sub-stack we’ll consider going forward.
We walk the stack, ignoring everything that matches the Irrelevant Signatures. We consider the first non-matching element the top of the new sub-stack.
We rewrite dll frame signatures to be the module only and merge consecutive ones.
We accumulate signatures that match the Prefix Signatures, until something doesn’t match.
We normalize each signature we accumulated. Signatures that match the Signatures With Line Numbers have their associated code line number added to them, like this:
The generated signature is a concatenation of all the accumulated signatures,
separated with a pipe sign (
|), and converted to a regular expression.
Signature generation then uses
.match() to match frames.
Because of that, when changing these lists, make sure you keep the following things in mind:
Make sure you’re using valid regular expression syntax and escape special characters like
There’s no need to add a trailing
.*since signature generation uses
.match()which will match from the beginning of the string.
Try to keep it roughly in alphabetical order so as to make it easier to skim through later.
Signature Sentinels are signatures (not regular expression) that should be used as the top of the stack if present. Everything before the first matching signature will be ignored.
The code iterates through the stack frame, throwing away everything it finds until it encounters a match to this regular expression or the end of the stack. If it finds a match, it passes all the frames after the match to the next step. If it finds no match, it passes the whole list of frames to the next step.
A typical line might be
Irrelevant Signatures are regular expressions of signatures that will be ignored while going through the stack. Anything that matches this list will not be added to the overall signature.
A typical rule might be
Prefix Signatures are regular expressions of signatures that will be combined with the following frame’s signature. Signature generation stops at the first non-matching signature it finds.
A typical rule might be
Note: These are regular expressions. Dollar signs and other regexp characters
need to be escaped with a
The first thing we will ask you to do is to file a bug. We keep track of every change in Socorro via bugs, so it’s important that each commit has one associated to it.
File a bug in the Socorro::Signature component, describe the changes you want to make, and assign it to you.
Then proceed to making those changes and creating a pull request.
If you are a git power user, you probably don’t need us to explain how to do this! :)
If you are not, you’re probably better off using GitHub’s interface. Read on!
First, you need to be logged in to GitHub. Open the file you want to edit, and then click the little pen in the top right corner of the page, the one that says Fork this project and edit the file, or Edit the file in your fork of this project if you already have a fork of it.
That will take you to an editor, where you can write any changes you want. Once you are done editting the file, enter a commit description. We have some conventions, and a bot that will automatically close bugs, so please make your commit message following this pattern: Fixes bug XYZ - Desciption of the change. Once you are ready, click Propose file change.
That will create a branch in your fork of the Socorro project, and take you to the commit you just created. You can verify that the changes you made are correct, and then click Create pull request, and then Create pull request again. Once the pull request is opened, Circle CI will automatically start running our test suite, which includes sanity checks for those signature lists. You can see the status of those tests in the pull request, and click the Details link to see logs in case of a failure.
That’s it! You have proposed a change, we have been notified about it. Someone from the Socorro team will review your changes and merge them if they are appropriate. Thank you for contributing to Socorro!
If you are interested in watching what’s changing in the
in the repository, but don’t care much about what happens in the rest of the
Socorro repo, you can easily set a filter in your email client to do that.
Here’s an example filter you can use today:
to:(firstname.lastname@example.org) ("A socorro/signature/siglists/" OR "M socorro/signature/siglists/" OR "D socorro/signature/siglists")
This is the signature generation pipeline defined at
Generates a signature based on stack frames.
For Java crashes, this generates a basic signature using stack frames.
For C/C++/Rust crashes, this generates a more robust signature using normalized versions of stack frames augmented by the contents of the signature lists.
Rough signature list rules (there are more details in the siglists README):
Walk the frames looking for a “signature sentinel” which becomes the first item in the signature.
Continue walking frames.
If the frame is in the “irrelevant” list, ignore it and continue.
If the frame is in the “prefix” list, add it to the signature and continue.
If the frame isn’t in either list, stop walking frames.
Signature is generated by joining those frames with ” | ” between them.
This rule also generates the proto_signature which is the complete list of normalized frames.
Appends minidump-stackwalker error to signature.
OOM | <size>to signatures for OOM crashes.
See bug #1007530.
Prepends abort message to signature.
See bug #803779.
Replaces signature with async_shutdown_timeout message.
Prepends “shutdownhang” to signature for shutdown hang crashes.
Replaces the signature with IPC channel error.
Appends ipc_message_name to signature.
Stomp on the signature if moz_crash_reason is
parentBuildID != childBuildID.
In the case where the assertion fails, then the parent buildid and the child buildid are different. This causes a lot of strangeness particularly in symbolification, so the signatures end up as junk. Instead, we want to bucket all these together so we replace the signature.
Replaces signature with JIT classification.
Fix whitespace in signatures.
This does the following:
trims leading and trailing whitespace
converts all non-space whitespace characters to space
reduce consecutive spaces to a single space
Truncates signatures down to SIGNATURE_MAX_LENGTH characters.