315 lines
14 KiB
Markdown
315 lines
14 KiB
Markdown
|
|
# Understanding Crash Reports
|
|
|
|
```{eval-rst}
|
|
+--------------------------------------------------------------------+
|
|
| This page is an import from MDN and the contents might be outdated |
|
|
+--------------------------------------------------------------------+
|
|
```
|
|
|
|
If a user experiences a crash they will be prompted to submit a raw
|
|
crash report, which is generated by Breakpad. The raw crash report is
|
|
received by [Socorro](https://github.com/mozilla/socorro) which
|
|
[creates](https://github.com/mozilla/socorro/blob/master/socorro/processor/mozilla_processor_2015.py)
|
|
a processed crash report. The processed crash report is based on the raw
|
|
crash report but also has a signature, classifications, and a number of
|
|
improved fields (e.g. OS, product, version). Many of the fields in both
|
|
the raw crash report and the processed crash report are viewable and
|
|
searchable on [crash-stats](https://crash-stats.mozilla.org/).
|
|
Although there are two distinct crash reports, the raw and the
|
|
processed, people typically talk about a single "crash report" because
|
|
crash-stats mostly presents them in a combined way.
|
|
|
|
Each crash report contains a wealth of data about the crash
|
|
circumstances. Despite this, many crash reports lack sufficient data for
|
|
a developer to understand why the crash occurred. As well as providing a
|
|
general overview, this page aims to highlight parts of a crash report
|
|
that may provide non-obvious insights.
|
|
|
|
Note that most crash report fields are visible, but a few
|
|
privacy-sensitive parts of it are only available to users who are logged
|
|
in and have "minidump access". A relatively small number of users have
|
|
minidump access, and they are required to follow certain rules. For
|
|
access, see the [Protected Data Access docs on Crash Stats](https://crash-stats.mozilla.org/documentation/protected_data_access/).
|
|
|
|
Each crash report has the following tabs: Details, Metadata, Modules,
|
|
Raw Dump, Extensions, and (optional) Correlations.
|
|
|
|
## Details tab
|
|
|
|
The Details tab is the first place to look because it contains the most
|
|
important pieces of information.
|
|
|
|
### Primary fields
|
|
|
|
The first part of the Details tab shows a table containing the most
|
|
important crash report fields. It includes such things as when the
|
|
crash occurred, in which product and version, the crash kind, and
|
|
various details about the OS and configuration of the machine on which
|
|
the crash occurred. The following screenshot shows some of these
|
|
fields.
|
|
|
|
```{image} https://mdn.mozillademos.org/files/13579/Details1.png
|
|
```
|
|
|
|
All fields have a tool-tip. For many fields, the tool-tip describes its
|
|
meaning. For all fields, the tool-tip indicates the key to use when you
|
|
want to do searches involving this field. (The field name is usually but
|
|
not always similar to the search key. E.g. the field "Adapter Device ID"
|
|
has the search key "adapter_device_id".) These descriptions are shown in
|
|
the [SuperSearchFields
|
|
API](https://crash-stats.mozilla.org/api/SuperSearchFields/) and can be
|
|
[modified in super_search_fields.py](https://github.com/mozilla-services/socorro/blob/main/socorro/external/es/super_search_fields.py)
|
|
or by writing up a [bug in Socorro](https://bugzilla.mozilla.org/enter_bug.cgi?format=__standard__&product=Socorro).
|
|
|
|
The fields present in this tab vary depending on the crash kind. Not all
|
|
fields are always present.
|
|
|
|
The "Signature" field is the main identifier or label for a crash report.
|
|
Rather than considering each crash report in isolation, we want to put
|
|
crash reports into clusters so we can deal with groups of them at once.
|
|
An ideal clustering algorithm would put all crash reports with the same
|
|
root cause into a single cluster, and all crash reports with different
|
|
root causes into different clusters. The crash signature is our
|
|
imperfect but still useful attempt at such an algorithm. Most crash
|
|
signatures are based on the crashing stack trace, but some
|
|
special-purpose annotations are used to indicate particular kinds of
|
|
crashes.
|
|
|
|
- `Abort`: A controlled abort, e.g. via `NS_RUNTIMEABORT`.
|
|
(Controlled aborts that occur via `MOZ_CRASH` or
|
|
`MOZ_RELEASE_ASSERT` currently don't get an `Abort` annotation,
|
|
but they do get a "MOZ_CRASH Reason" field.)
|
|
- `OOM | <size>`, where `<size>` is one of `large`, `small`,
|
|
`unknown`: an out-of-memory (OOM) abort. The `<size>` annotation
|
|
is determined by the "OOM Allocation Size" field; if that field is
|
|
missing `<size>` will be `unknown`.
|
|
- `hang`: a hang prior to shutdown.
|
|
- `shutdownhang`: a hang during shutdown.
|
|
- `IPCError-browser`: a problem involving IPC. If the parent Firefox
|
|
process detects that the child process has sent broken or
|
|
unprocessable IPDL data, or is not shutting down in a timely manner,
|
|
it kills the child process with a crash report. These crashes will
|
|
now have a signature that indicates why the process was killed,
|
|
rather than the child stack at the moment.
|
|
|
|
When no special-purpose annotation is present and the signature begins
|
|
with a stack frame, it's usually a vanilla uncontrolled crash. The crash
|
|
cause can be determined from the "Crash Reason" field. Most commonly
|
|
it's a bad memory access. In that case, on Windows you can tell from the
|
|
reason field if the crash occurred while reading, writing or executing
|
|
memory (e.g. `EXCEPTION_VIOLATION_ACCESS_READ` indicates a bad memory
|
|
read). On Mac and Linux the reason will be SIGSEGV or SIGBUS and you
|
|
cannot tell from this field what kind of memory access it was.
|
|
|
|
See [this
|
|
file](https://github.com/mozilla-services/socorro/blob/master/socorro/signature/README.rst)
|
|
for a detailed explanation of the crash report signature generation
|
|
procedure, and for information on how modify this procedure.
|
|
|
|
There are no fields that uniquely identify the user that a crash report
|
|
came from, but if you want to know if multiple crashes come from a
|
|
single user the "Install Time" field is a good choice. Use it in
|
|
conjunction with other fields that don't change, such as those
|
|
describing the OS or graphics card, for additional confidence.
|
|
|
|
For bad memory accesses, the "Crash Address" field can give additional
|
|
indications what went wrong.
|
|
|
|
- 0x0 is probably a null pointer deference[\*].
|
|
- Small addresses like 0x8 can indicate an object access (e.g.
|
|
`this->mFoo`) via a null `this` pointer.
|
|
- Addresses like 0xfffffffffd8 might be stack accesses, depending on
|
|
the platform[\*].
|
|
- Addresses like 0x80cdefd3 might be heap accesses, depending on the
|
|
platform.
|
|
- Addresses may be poisoned: 0xe4 indicates the address comes from
|
|
memory that has been allocated by jemalloc but not yet initialized;
|
|
0xe5 indicates the address comes from memory freed by jemalloc. The
|
|
JS engine also has multiple poison values defined in
|
|
`js/src/jsutil.h`.
|
|
|
|
[\*] Note that due to the way addressing works on x86-64, if the crash
|
|
address is 0x0 for a Linux/macOS crash report, or 0xffffffffffffffff for
|
|
a Windows crash report, it's highly likely that the value is incorrect.
|
|
(There is a [bug
|
|
report](https://bugzilla.mozilla.org/show_bug.cgi?id=1493342) open
|
|
for this problem.) You can sanity-check these crashes by looking at the
|
|
raw dump or minidump in the Raw Dump tab (see below).
|
|
|
|
Note that for non-release builds the "Version" field represents multiple
|
|
different builds since nightly and beta version numbers are reused for
|
|
builds created over a series of days until the version number is bumped.
|
|
(The "Build ID" field can disambiguate.) It's not currently possible to
|
|
[restrict searches to a given version or
|
|
later](https://bugzilla.mozilla.org/show_bug.cgi?id=1401517) (using
|
|
\>= with a build ID and a given release channel may work around this).
|
|
|
|
Some fields, such as "URL" and "Email Address", are privacy-sensitive
|
|
and are only visible to users with minidump access.
|
|
|
|
The Windows-only "Total Virtual Memory" field indicates if the Firefox
|
|
build and OS are 32-bit or 64-bit.
|
|
|
|
- A value of 2 GiB indicates 32-bit Firefox on 32-bit Windows.
|
|
- A value of 3 or 4 GiB indicates 32-bit Firefox on 64-bit Windows
|
|
(a.k.a. "WoW64"). Such a user could switch to 64-bit Firefox.
|
|
- A value much larger than 4 GiB (e.g. 128 TiB) indicates 64-bit
|
|
Firefox. (The "Build Architecture" field should be "amd64" in this
|
|
case.)
|
|
|
|
Some crash reports might contain a memory report. This memory report will
|
|
have been made some time before the crash, at a time when available
|
|
memory was low. In this case, a number of key measurements from the
|
|
memory report are shown in the Details tab, each one having a field name
|
|
starting with "MR:", short for "memory report". The full memory report
|
|
can be obtained in the Raw Dump tab (see below).
|
|
|
|
### Bug-related information
|
|
|
|
The second part of the Details tab shows bug-related information, as the
|
|
following screenshot shows.
|
|
|
|
```{image} https://mdn.mozillademos.org/files/13581/Details2.png
|
|
```
|
|
|
|
The "Report this bug in" links can be used to easily file bug reports.
|
|
Each one links to a Bugzilla bug report creation page that has various
|
|
fields pre-filled, such as the crash signature.
|
|
|
|
The "Related Bugs" section shows related bug reports, as determined by
|
|
the crash signature.
|
|
|
|
### Stack traces
|
|
|
|
The third part of the Details tab shows the stack trace and thread
|
|
number of the crashing thread, as the following screenshot shows.
|
|
|
|
```{image} https://mdn.mozillademos.org/files/13583/Details3.png
|
|
```
|
|
|
|
Each stack frame has a link to the source code, when possible. If a
|
|
crash is new, the regressing changeset can often be identified by
|
|
looking for recent changes in the blame annotations for one or more of
|
|
the top stack frames. Blame annotations are also good for identifying
|
|
who might know about the code in question.
|
|
|
|
Sometimes the highlighted source code is puzzling, e.g. the identified
|
|
line may not touch memory even though the crash is memory-related. This
|
|
can be caused by compiler optimizations. It's often better to look at
|
|
the disassembly (e.g. in a minidump) to understand exactly what code is
|
|
being executed.
|
|
|
|
Stack frame entries take on a variety of forms.
|
|
|
|
- The simplest are functions names, such as `NS_InitXPCOM2`.
|
|
- Name/address pairs such as `nss3.dll@0x1eb720` are within system
|
|
libraries.
|
|
- Names such as `F1398665248_____________________________` ('F'
|
|
followed by many numbers then many underscores) are in Flash.
|
|
- Addresses such as `@0xe1a850ac` may indicate an address that wasn't
|
|
part of any legitimate code. If an address such as this occurs in the
|
|
first stack frame, the crash may be exploitable.
|
|
|
|
Stack traces for other threads can be viewed by clicking on the small
|
|
"Show other threads" link.
|
|
|
|
If the crash report is for a hang, the crashing thread will be the
|
|
"watchdog" thread, which exists purely to detect hangs; its top stack
|
|
frame will be something
|
|
like{literal}`mozilla::`anonymous namespace'::RunWatchdog`. In that
|
|
case you should look at the other threads' stack traces to determine the
|
|
problem; many of them will be waiting on some kind of response, as shown
|
|
by a top stack frame containing a function like
|
|
`NtWaitForSingleObject` or `ZwWaitForMultipleObjects`.
|
|
|
|
## Metadata tab
|
|
|
|
The Metadata tab is similar to the first part of the Details tab,
|
|
containing a table with various fields. These are the fields from the
|
|
raw crash report, ordered alphabetically by field name, but with
|
|
privacy-sensitive fields shown only to users with minidump access. There
|
|
is some overlap with the fields shown in the Details tab.
|
|
|
|
## Modules tab
|
|
|
|
The modules tab shows all the system libraries loaded at the time of the
|
|
crash, as the following screenshot shows.
|
|
|
|
```{image} https://mdn.mozillademos.org/files/13593/Modules1.png
|
|
```
|
|
|
|
On Windows these are mostly DLLs, on Mac they are mostly `.dylib`
|
|
files, and on Linux they are mostly `.so` files.
|
|
|
|
This information is most useful for Windows crashes, because DLLs loaded
|
|
by antivirus software or malware often cause Firefox to crash.
|
|
Correlations between loaded modules and crash signatures can be seen in
|
|
the "Correlations" tab (see below).
|
|
|
|
[This page](https://support.mozilla.org/en-US/kb/helping-crashes)
|
|
says that files lacking version/debug identifier/debug filename are
|
|
likely to be malware.
|
|
|
|
## Raw Dump tab
|
|
|
|
The first part of the Raw Dump tab shows the raw crash report, in JSON
|
|
format. Once again, privacy-sensitive fields are shown only to users
|
|
with minidump access.
|
|
|
|
```{image} https://mdn.mozillademos.org/files/13595/RawDump1.png
|
|
```
|
|
|
|
For users with minidump access, the second part of the Raw Dump tab has
|
|
some links, as the following screenshot shows.
|
|
|
|
```{image} https://mdn.mozillademos.org/files/14047/raw-dump-links.png
|
|
```
|
|
|
|
These links are to the following items.
|
|
|
|
1. A minidump. Minidumps can be extremely useful in understanding a
|
|
crash report; see {ref}`this page <Debugging A Minidump>` for an
|
|
explanation how to use them.
|
|
2. The aforementioned JSON raw crash report.
|
|
3. The memory report contained within the crash report.
|
|
4. The unredacted crash report, which has additional information.
|
|
|
|
## Extensions tab
|
|
|
|
The Extensions tab shows which extensions are installed and enabled.
|
|
|
|
```{image} https://mdn.mozillademos.org/files/13599/Extensions1.png
|
|
```
|
|
|
|
Usually it just shows an ID rather than the proper extension name.
|
|
|
|
Note that several extensions ship by default with Firefox and so will be
|
|
present in almost all crash reports. (The exact set of default
|
|
extensions depends on the release channel.) The least obvious of these
|
|
has an Id of `{972ce4c6-7e08-4474-a285-3208198ce6fd}`, which is the
|
|
default Firefox theme. Some (but not all) of the other extensions
|
|
shipped by default have the following Ids: `webcompat@mozilla.org`,
|
|
`e10srollout@mozilla.org`, `firefox@getpocket.com`,
|
|
`flyweb@mozilla.org`, `loop@mozilla.org`.
|
|
|
|
If an extension only has a hexadecimal identifier, a Google search of
|
|
that identifier is usually enough to identify the extension's name.
|
|
|
|
This information is useful because some crashes are caused by
|
|
extensions. Correlations between extensions and crash signatures can be
|
|
seen in the "Correlations" tab (see below).
|
|
|
|
## Correlations tab
|
|
|
|
This tab is only shown when crash-stats identifies correlations between
|
|
a crash and modules or extensions that are present, which happens
|
|
occasionally.
|
|
|
|
## See also
|
|
|
|
- [A talk about understanding crash
|
|
reports](https://air.mozilla.org/a-talk-about-understanding-crash-reports/),
|
|
by David Baron, from March 2016.
|
|
- {ref}`A guide to searching crash reports`
|