334 lines
16 KiB
Markdown
334 lines
16 KiB
Markdown
# MAR Files
|
|
|
|
MAR files, short for Mozilla ARchive files, are integral to Firefox Application
|
|
Update, so this page will document what they are and go into their structure in
|
|
some detail.
|
|
|
|
## Overview
|
|
|
|
A MAR file consists of headers, signatures, multiple files including at least
|
|
one manifest, and an index. Only one manifest, of the newest supported version,
|
|
will be used; others will be ignored.
|
|
|
|
The manifest must be a complete file, but others may be patch files. Patch files
|
|
are used to transform the old version of the file to the new version. Currently
|
|
supported patch algorithms are bspatch and Zucchini.
|
|
|
|
Files are optionally compressed. Currently supported compression algorithms are
|
|
bzip2 and xz.
|
|
|
|
## Types of MARs
|
|
|
|
There are two types of MAR files: partial MARs and complete MARs.
|
|
|
|
A complete MAR contains an entire installation. It does not require any
|
|
particular files to be on disk in order to update. Theoretically, it can be used
|
|
to update from any version, but
|
|
[Watershed Updates](../../../../update-infrastructure/index.md#watershed-updates)
|
|
complicate this somewhat.
|
|
|
|
A partial MAR contains only the changes between one version and another. While
|
|
they may contain some files in their entirety, they mostly have binary patches
|
|
that can be used to update an existing file to the new version. The advantage of
|
|
this is that partial MARs can be much smaller than complete MARs. But the
|
|
disadvantage is that a partial MAR must target a specific version of Firefox so
|
|
that we can provide the correct patches to update that version of the files. And
|
|
we just cannot provide a partial targeting every version of Firefox. We
|
|
currently generate partials for the previous four versions released (including
|
|
point versions). When Firefox is more out-of-date than that, a complete MAR must
|
|
be used. Another limitation is that the files on the disk must not have been
|
|
changed. We verify this by including headers in the patch file with the expected
|
|
size and CRC of the existing file. If these do not match the file on disk, the
|
|
update fails and we fall back to using a complete MAR.
|
|
|
|
## Manifest
|
|
|
|
The manifest is a special file included in the MAR that describes the contents
|
|
of the MAR and how to install it. It is identified by its filename:
|
|
`updatev3.manifest`.
|
|
|
|
Empty lines and `#`-prefixed comment lines are allowed. Otherwise, each line of
|
|
the manifest will consist of a method name followed by variable number of quoted
|
|
parameters. These are separated by an arbitrary amount of space (`0x20`) and/or
|
|
tab (`0x09`) characters. The lines must end with a Windows/CRLF newline
|
|
(`0x0D0A`).
|
|
|
|
Here is an example manifest containing each of the possible method types:
|
|
|
|
```
|
|
type "partial"
|
|
add "2/20/20text0"
|
|
add-if "distribution/extensions/extensions0" "distribution/extensions/extensions0/extensions0text0"
|
|
add-if-not "defaults/pref/channel-prefs.js" "defaults/pref/channel-prefs.js"
|
|
patch "0/0exe0.exe.patch" "0/0exe0.exe"
|
|
patch-if "distribution/extensions/extensions1" "distribution/extensions/extensions1/extensions1png1.png.patch" "distribution/extensions/extensions1/extensions1png1.png"
|
|
remove "1/10/10text0"
|
|
rmdir "9/99/"
|
|
rmrfdir "9/98/"
|
|
```
|
|
|
|
`type` is special method that should be `partial` or `complete` depending on
|
|
[what kind of MAR this is](#types-of-mars). If included, it must be the first
|
|
method specified. The rest are implemented by classes deriving from `Action` in
|
|
`/toolkit/mozapps/update/updater/updater.cpp`.
|
|
|
|
## Manipulation
|
|
|
|
The recommended way to manually manipulate (create, extract, sign, etc) MAR
|
|
files is with the tool located
|
|
[here](https://github.com/mozilla-releng/build-mar).
|
|
|
|
## MAR Structure
|
|
|
|
This section was almost directly copied from
|
|
[the wiki](https://wiki.mozilla.org/Software_Update:MAR) in order to consolidate
|
|
our documentation into a place where it is more likely to be updated. This
|
|
document should be considered more up-to-date than that one.
|
|
|
|
### Why the bespoke structure?
|
|
|
|
This question was given a fair amount of consideration. Ultimately, we decided
|
|
to go with a custom file format because using libjar would have required a fair
|
|
bit of hacking. Writing custom code was a simpler option, and it resulted in
|
|
less code (mar_read.c is less than 300 lines of code). Moreover, the update
|
|
system does not need a standard file format. The elements stored in the archive
|
|
are bzip2 compressed binary diffs, generated using a variation of bsdiff. So,
|
|
being able to unpack the archive file using standard tools wouldn't be very
|
|
useful in and of itself.
|
|
|
|
### Byte Ordering
|
|
|
|
All fields are in big-endian format. The signatures are in
|
|
NSS / OpenSSL / big-endian order and not CryptoAPI order. If CryptoAPI is used
|
|
to check a signature, the bytes of the signature must be reversed before
|
|
verifying the signature using CryptVerifySignature.
|
|
|
|
### Constraints
|
|
|
|
To protect against invalid inputs the following constraints are in place:
|
|
|
|
- There are at most 8 signatures.
|
|
- The file size of the MAR file is at most 500MB.
|
|
- No signature is more than 2048 bytes long.
|
|
|
|
### File Layout
|
|
|
|
The following sections, in the given order, make up the structure of a MAR
|
|
file. Some of these sections have been added over time and thus may not be
|
|
present in sufficiently old MAR files. Old parsers will completely ignore these
|
|
sections as they will simply appear to be unaddressed space in the "Files"
|
|
section. Modern updater binaries will, however, reject MAR files without
|
|
signatures.
|
|
|
|
#### Header
|
|
|
|
1. (4 bytes) MAR identifier which must be `MAR1`.
|
|
2. (4 bytes) Byte offset of the file index header relative to the start of the
|
|
file.
|
|
3. (8 bytes) Size in bytes of the entire MAR file.
|
|
4. (4 bytes) Number of signatures (minimum of 0, maximum of 8).
|
|
|
|
#### Signatures
|
|
|
|
This section will be repeated as many times as indicated by the relevant header.
|
|
Note that these signatures must sign all bytes of the MAR file excluding the
|
|
signatures themselves.
|
|
|
|
1. (4 bytes) ID representing the type of signature algorithm.
|
|
2. (4 bytes) Size in bytes of the signature that follows.
|
|
3. The signature itself.
|
|
|
|
#### Additional Sections Header
|
|
|
|
This consists of 4 bytes representing the number of additional sections.
|
|
|
|
#### Additional Sections
|
|
|
|
There will be as many of these sections as indicated in the Additional Sections
|
|
Header, immediately prior.
|
|
|
|
1. (4 bytes) Section size, in bytes. Note that, unlike the signature size, this
|
|
is the size of the section and its metadata (the size and identifier).
|
|
2. (4 bytes) Section identifier.
|
|
3. The rest of the section. Consists of 8 bytes fewer than the section size
|
|
given above.
|
|
|
|
#### Files
|
|
|
|
All file data. Must be addressed by the index, below. The length will be the sum
|
|
of content sizes of all files in the index.
|
|
|
|
#### Index header
|
|
|
|
This consists of 4 bytes representing the size of the index in bytes.
|
|
|
|
#### Index entry
|
|
|
|
This section will be repeated to fill the size indicated by the index header,
|
|
describing all files contained in the MAR.
|
|
|
|
1. (4 bytes) Byte offset of the file, relative to the start of the MAR.
|
|
2. (4 bytes) The content size in bytes.
|
|
3. (4 bytes) File permission bits, in standard unix-style format.
|
|
4. Variable length filename
|
|
5. (1 byte) Null terminator
|
|
|
|
### Additional Sections
|
|
|
|
These are meant to be flexible, so more could potentially be added later.
|
|
|
|
#### Product Information
|
|
|
|
The product information section identifies the product and channel this MAR file
|
|
applies to. It also includes the new version number to avoid downgrades.
|
|
|
|
This section has this structure:
|
|
|
|
1. (4 bytes) The size of the product information block.
|
|
2. (4 bytes) The section ID, `1` in this case.
|
|
3. (<64 bytes) The product and channel (such as from MAR_CHANNEL_ID).
|
|
Example: mozilla-central
|
|
4. (1 byte) Null terminator.
|
|
5. (<32 bytes) The product version string (such as from MOZ_APP_VERSION)
|
|
Examples: 12.0.1.5371, 12.0a1
|
|
6. (1 byte) Null terminator.
|
|
|
|
## Modifying Test MARs
|
|
|
|
Occasionally we need to modify a MAR file that we use in testing. These are some
|
|
modifications that have been needed before and the steps to accomplish them.
|
|
|
|
### Rename Files
|
|
|
|
In this example, we will rename some of the files in `partial_mac.mar`.
|
|
|
|
Regarding signing, note the following useful things: (a) The test MAR signatures
|
|
successfully verify against
|
|
`toolkit/mozapps/update/updater/xpcshellCertificate.der` and (b) according to
|
|
[this comment](https://searchfox.org/mozilla-central/rev/fc00627e34639ef1014e87d9fa24091905e9dc5d/toolkit/mozapps/update/updater/moz.build#41-43),
|
|
that certificate was generated from `mycert` in
|
|
`modules/libmar/tests/unit/data`.
|
|
|
|
1. `./mach build` so that `<obj>/dist/bin/signmar` is available.
|
|
2. Install the MAR manipulation utility, if necessary: `pip install mar`.
|
|
3. Make a temporary working directory:
|
|
`cd /path/to/mozilla/repo && mkdir temp && cd temp`
|
|
4. Extracted the MAR to be changed:
|
|
`mar -J -x ../toolkit/mozapps/update/tests/data/partial_mac.mar`.
|
|
The `-J` specifies a compression type of `xz`. You can instead specify
|
|
`--auto` to automatically detect the compression type (though you may want
|
|
to know the original compression later for recompression). You can check
|
|
the compression type by running
|
|
`mar -T ../toolkit/mozapps/update/tests/data/partial_mac.mar` and looking for
|
|
the `Compression type:` line.
|
|
5. Rename extracted files, as necessary.
|
|
6. Edit `updatev2.manifest` and `updatev3.manifest` to update the changed paths.
|
|
7. Run `mar -T ../toolkit/mozapps/update/tests/data/partial_mac.mar` to get a
|
|
complete list of the files originally in that MAR as well as the
|
|
product/version and channel strings (in this case `xpcshell-test` and `*`
|
|
respectively).
|
|
8. Create the new MAR:
|
|
`mar -J -c partial_mac_unsigned.mar -V '*' -H xpcshell-test <file1> <file2> ...`,
|
|
individually specifying each file path listed in by `mar -T`, substituting
|
|
with renamed paths as necessary.
|
|
9. Sign the MAR:
|
|
`../<obj>/dist/bin/signmar -d ../modules/libmar/tests/unit/data -n mycert -s partial_mac_unsigned.mar partial_mac.mar`.
|
|
10. To verify signing:
|
|
`../<obj>/dist/bin/signmar -D ../toolkit/mozapps/update/updater/xpcshellCertificate.der -v partial_mac.mar`.
|
|
This appears to output nothing on success, but it's probably good to check
|
|
to make sure `echo $?` displays `0`. I also compared the output of
|
|
`mar -T partial_mac.mar` to that of the original.
|
|
11. To verify files: Extract the new MAR with
|
|
`mkdir cmp && cd cmp && mar -J -x ../partial_mac.mar && cd ..` and verify
|
|
the files match the originals.
|
|
12. Overwrite the original MAR with the new one and remove the `temp`
|
|
directory:
|
|
`cd .. && mv -f temp/partial_mac.mar toolkit/mozapps/update/tests/data/partial_mac.mar && rm -rf temp`
|
|
|
|
### Generate Zucchini Partials
|
|
|
|
In this example, we will generate partial MARs using Zucchini rather than
|
|
bspatch.
|
|
|
|
First, note that in `partial.mar`, all exe patches are the same patch (turning
|
|
`complete.exe` into `partial.exe`), and all png patches are the same patch
|
|
(turning `complete.png` into `partial.png`):
|
|
|
|
```
|
|
$ mar -J -x ../partial.mar
|
|
|
|
$ grep -r MBDIFF10 .
|
|
Binary file ./0/00/00png0.png.patch matches
|
|
Binary file ./0/0exe0.exe.patch matches
|
|
Binary file ./distribution/extensions/extensions0/extensions0png0.png.patch matches
|
|
Binary file ./distribution/extensions/extensions0/extensions0png1.png.patch matches
|
|
Binary file ./distribution/extensions/extensions1/extensions1png0.png.patch matches
|
|
Binary file ./distribution/extensions/extensions1/extensions1png1.png.patch matches
|
|
Binary file ./exe0.exe.patch matches
|
|
Binary file ./searchplugins/searchpluginspng0.png.patch matches
|
|
Binary file ./searchplugins/searchpluginspng1.png.patch matches
|
|
|
|
$ md5sum ./0/00/00png0.png.patch ./distribution/extensions/extensions0/extensions0png0.png.patch ./distribution/extensions/extensions0/extensions0png1.png.patch ./distribution/extensions/extensions1/extensions1png0.png.patch ./distribution/extensions/extensions1/extensions1png1.png.patch ./searchplugins/searchpluginspng0.png.patch ./searchplugins/searchpluginspng1.png.patch
|
|
69ab8dc8614e6bb154c029b12ca8e3e9 *./0/00/00png0.png.patch
|
|
69ab8dc8614e6bb154c029b12ca8e3e9 *./distribution/extensions/extensions0/extensions0png0.png.patch
|
|
69ab8dc8614e6bb154c029b12ca8e3e9 *./distribution/extensions/extensions0/extensions0png1.png.patch
|
|
69ab8dc8614e6bb154c029b12ca8e3e9 *./distribution/extensions/extensions1/extensions1png0.png.patch
|
|
69ab8dc8614e6bb154c029b12ca8e3e9 *./distribution/extensions/extensions1/extensions1png1.png.patch
|
|
69ab8dc8614e6bb154c029b12ca8e3e9 *./searchplugins/searchpluginspng0.png.patch
|
|
69ab8dc8614e6bb154c029b12ca8e3e9 *./searchplugins/searchpluginspng1.png.patch
|
|
|
|
$ md5sum ./0/0exe0.exe.patch ./exe0.exe.patch
|
|
bd6e413d3248cfbeff65e104b6c4cd39 *./0/0exe0.exe.patch
|
|
bd6e413d3248cfbeff65e104b6c4cd39 *./exe0.exe.patch
|
|
```
|
|
|
|
Generate the equivalent zucchini patches:
|
|
|
|
```
|
|
$ ./obj-x86_64-pc-windows-msvc/dist/bin/zucchini.exe -gen toolkit/mozapps/update/tests/data/complete.exe toolkit/mozapps/update/tests/data/partial.exe exe0.exe.patch -keep
|
|
$ ./obj-x86_64-pc-windows-msvc/dist/bin/zucchini.exe -gen toolkit/mozapps/update/tests/data/complete.png toolkit/mozapps/update/tests/data/partial.png 00png0.png.patch -keep
|
|
```
|
|
|
|
Replace all patch files on disk by their Zucchini equivalent, then double check
|
|
by rerunning the `md5sum` commands:
|
|
|
|
```
|
|
$ grep -r Zucc .
|
|
Binary file ./0/00/00png0.png.patch matches
|
|
Binary file ./0/0exe0.exe.patch matches
|
|
Binary file ./distribution/extensions/extensions0/extensions0png0.png.patch matches
|
|
Binary file ./distribution/extensions/extensions0/extensions0png1.png.patch matches
|
|
Binary file ./distribution/extensions/extensions1/extensions1png0.png.patch matches
|
|
Binary file ./distribution/extensions/extensions1/extensions1png1.png.patch matches
|
|
Binary file ./exe0.exe.patch matches
|
|
Binary file ./searchplugins/searchpluginspng0.png.patch matches
|
|
Binary file ./searchplugins/searchpluginspng1.png.patch matches
|
|
|
|
$ md5sum ./0/00/00png0.png.patch ./distribution/extensions/extensions0/extensions0png0.png.patch ./distribution/extensions/extensions0/extensions0png1.png.patch ./distribution/extensions/extensions1/extensions1png0.png.patch ./distribution/extensions/extensions1/extensions1png1.png.patch ./searchplugins/searchpluginspng0.png.patch ./searchplugins/searchpluginspng1.png.patch
|
|
958bae1b40904145959ba45f988a7156 *./0/00/00png0.png.patch
|
|
958bae1b40904145959ba45f988a7156 *./distribution/extensions/extensions0/extensions0png0.png.patch
|
|
958bae1b40904145959ba45f988a7156 *./distribution/extensions/extensions0/extensions0png1.png.patch
|
|
958bae1b40904145959ba45f988a7156 *./distribution/extensions/extensions1/extensions1png0.png.patch
|
|
958bae1b40904145959ba45f988a7156 *./distribution/extensions/extensions1/extensions1png1.png.patch
|
|
958bae1b40904145959ba45f988a7156 *./searchplugins/searchpluginspng0.png.patch
|
|
958bae1b40904145959ba45f988a7156 *./searchplugins/searchpluginspng1.png.patch
|
|
|
|
$ md5sum ./0/0exe0.exe.patch ./exe0.exe.patch
|
|
7750e88e0c1d006710abbd625710748b *./0/0exe0.exe.patch
|
|
7750e88e0c1d006710abbd625710748b *./exe0.exe.patch
|
|
```
|
|
|
|
Generate an unsigned mar file. Unfortunately on Windows this doesn't preserve
|
|
the permissions from the original mar. The simplest solution is to do this part
|
|
on a UNIX (file)system.
|
|
|
|
```
|
|
$ mar -J -c ../partial_zucchini_unsigned.mar -H xpcshell-test -V '*' 2/20/20png0.png 0/0exe0.exe.patch 0/00/00text2 distribution/extensions/extensions0/extensions0text0 distribution/extensions/extensions0/extensions0png1.png.patch 0/00/00text0 distribution/extensions/extensions0/extensions0png0.png.patch searchplugins/searchpluginspng1.png.patch distribution/extensions/extensions1/extensions1png0.png.patch distribution/extensions/extensions1/extensions1text0 searchplugins/searchpluginstext0 distribution/extensions/extensions1/extensions1png1.png.patch defaults/pref/channel-prefs.js update-settings.ini updatev2.manifest searchplugins/searchpluginspng0.png.patch precomplete 0/00/00png0.png.patch updatev3.manifest exe0.exe.patch 2/20/20text0
|
|
```
|
|
|
|
Generate a signed mar from the unsigned mar:
|
|
|
|
```
|
|
$ /d/mozilla-source/firefox/obj-x86_64-pc-windows-msvc/dist/bin/signmar.exe -d /d/mozilla-source/firefox/modules/libmar/tests/unit/data/ -n mycert -s ../partial_zucchini_unsigned.mar ../partial_zucchini.mar
|
|
```
|