Nightly monitoring of the zucchini rollout revealed a surge in
SERVICE_STILL_APPLYING_ON_FAILURE, READ_ERROR and WRITE_ERROR during
updates. This commit aims to address READ_ERROR and WRITE_ERROR on all
platforms, and to address SERVICE_STILL_APPLYING_ON_FAILURE on Windows.
Addressing SERVICE_STILL_APPLYING_ON_FAILURE on Linux and macOS is done
in the follow-up commit.
SERVICE_STILL_APPLYING_ON_FAILURE means the updater process exited
without writing update.status, likely crashing. There are two potential
sources of crashes in zucchini. First, Chromium CHECK macros call
ImmediateCrash() (int3/ud2), which kills the process before
WriteStatusFile can run. More likely, std::bad_alloc from non-fallible
allocations in zucchini's disassembler (e.g. std::make_unique,
std::deque::push_back when parsing a large PE like xul.dll) would also
crash the process. Both cases can be caught by via SEH on Windows.
READ_ERROR and WRITE_ERROR can mask OOMs from CreateFileMapping /
MapViewOfFile failures (or mmap on POSIX) when zucchini tries to
memory-map large files. Unlike bspatch which uses sequential I/O,
zucchini memory-maps both the patch file and the output file, requiring
significantly more virtual address space.
This commit:
1. Adds an SEH exception filter (FilterZucchiniException) around
zucchini's Load and ApplyUnsafe calls that catches
EXCEPTION_IN_PAGE_ERROR (existing), EXCEPTION_BREAKPOINT /
EXCEPTION_ILLEGAL_INSTRUCTION (CHECK failures), and 0xE06D7363
(std::bad_alloc). All are recoverable exceptions where the process
state is sound. Exceptions indicating corrupt state
(EXCEPTION_ACCESS_VIOLATION, etc.) are left unhandled. This improves
crash recovery and OOM detection on Windows.
2. Introduces kStatusOutOfMemory in zucchini's status codes, mapped to
BSPATCH_MEM_ERROR (12) in the updater. OOMs from the SEH exception
path and from the mapping-failure path (via is_oom() from the
previous commit) both flow through this code, which lets the update
service knows that we are currently under memory pressure.
3. Eagerly resets mPatchFileDecoder in PatchFile::Execute() after Apply
returns, so memory-mapped patch files are released between sequential
patch actions rather than accumulating until the ActionList is
destroyed.
Differential Revision: https://phabricator.services.mozilla.com/D288918
Yannis Juglaret
5ce901aa2d