As part of the devices I researched for my book on Android Internals, I got myself a Nexus 9. Although Google made the filesystem image available, there's no 64-Bit emulator (at least, none that I know of), and most of the interesting stuff is found during runtime. Plus, the device is unlockable, getting root is an easy matter, and the full internals of the system would be interesting to explore.
The version the Nexus 9 presently runs (it updates automatically as you first boot it) is LRX21R - technically 5.0.0._r7, and though the 5.0.1 image has been released, it's surprisingly unavailable for the Nexus 9 as an OTA (having checked for system updates 12/4/14, 20:19PM). One would think Google wouldn't have to do so many silent bugfixes after having delayed for so long.
A detailed teardown of the Nexus 9 and a hardware-perspective was already published in several places around the 'Net, so that's not my intent. Rather, this summarizes findings from snooping around the file system I did to kill time on a long trans-pacific with no better distraction the in-flight entertainment... Plus, this is somewhat of a complement to the article I wrote about disassembling aboot - which was SnapDragon (msm) oriented - whereas here we're dealing with Tegra.
As with all Android devices (as discussed in Chapter 2 of the book), the MMC is partitioned into far more than the mountable partitions of /data, /system and /cache. The Nexus contains no less than 33 partitions, and unlike the msm eMMC (wherein the partitions are given sensible names!) the sdhci-tegra gives the partitions the same (annoying) three letter names found in the NVidia Shield (and, I therefore presume, all Tegra based controllers). These can be seen in the following output, which I've annotated for better readability. The greyed partitions are empty:
Quite a few of the partitions are empty, at least on the WiFi-only model. I'm assuming that the situation might be different on the cellular-enabled version, since the baseband likely has need for some partitions.
The DTB partition was supposed to hold the compiled device tree - which instead is appended to the kernel image. To get the device tree you can use imgtool (though you'd need to strip the 0x100 byte hboot header first).
The MSC partition is the classic Android misc, which is used for the Bootloader Control Block (BCB) - which is updated when you use adb reboot with recovery or bootloader (personal note: I'm surprised PST wasn't merged into this - see below).
The Nexus 9 L image contains the persistent_data_block service (android.service.persistentdata.IPersistentDataBlockService) which is not present in the Emulator (another reason why I needed a real device - you can't write a book about Android based on just the emulator images). This service (defined in the AOSP's frameworks/base/services/core/java/com/android/server/PersistentDataBlockService.java) controls whether or not the fastboot oem unlock command will be allowed by the boot loader. The Settings app calls on the service's setOemUnlockEnabled(boolean enabled); to disable or enable the fastboot oem unlock functionality.
The setting naturally has to be read by hboot/aboot - which means it has to reside on a raw partition. Indeed, it does - on the Persistent partition, labeled PST. The service gets the partition's name from the ro.frp.pst property, which is set in the "ADDITIONAL_BUILD_PROPERTIES" of the /system/build.prop.
The partition is entirely empty if the OEM unlock is disallowed (as it is by default), but if the setting is toggled, a single bit magically appears in the last word:
The boot loader's fastboot oem get_unlock_ability can be used to test the bit (as can fastboot oem unlock, of course).
MFG contains various manufacturing data pertaining to the device. The beginning of the partition has a(n as yet unknown) header, followed by variable=value , padded by a lot of null bytes. Somewhere deep in the partition you can find the device manufacturing date (201410xx or 201411xx), MB and S/N (as per the sticker on the back) but surprisingly not the IMEI or the P/N (unless I missed it).
Mounted File Systems
The /data/ filesystem is mounted with F2FS, which is all the rage in the world of Android nowadays. The mounting is over the UDA partition (..p31), but over the device mapper (i.e. /dev/block/dm-0, and not /dev/block/mmcblk0p31) so as to support the filesystem encryption (discussed in the book and the Andevcon lecture on security).
L finally uses the function file system, a successor to the gadgetFS/ConfigFS which serve the functions of the USB gadget driver, which it can adopt as part of the 3.10 kernel offerings. Another notable addition is pstore, which takes over the deprecated ram console - if the kernel ever panics, the log can be found in this directory. As with the experiment in chapter 2:
I also cover most of the filesystems in the book (Chapter 2), but the following output shows the new ones:
The Boot Loader
Unlike MSM devices, which reads images like aboot directly (in the format specified in the previous article), HTC uses their own loader - HBOOT - with a small custom header on images, which can be found prepended to both the aboot images (LNX and SOS), as well as the boot loader itself (bootloader-flounder-3.43.0.0114.img in the images supplied by Google). The header can be seen in the following output:
It's not entirely clear to the author why there appears to be so much redundancy in the header (e.g. why is 0x100 repeated three times? Header Size? Data Start? What else?). But that's a non issue. Once you figure out the actual data starts at 0x100, the custom header can be stripped, leaving you (in the case of LNX/SOS) with an aboot image, or (for bootloader-flounder-xxxx.img) - a zip file! This becomes evident by the "PK" signature, which is correctly identified by file(1):
And voilà! All the boot loader components. I haven't fully researched the exact order of these, but TOS precedes hboot, and nvtboot.img (which is loaded at 0x40000000, before either) precedes them both (i.e. nvtboot → (+wb0) → tos → hboot). Given time and the NVidia docs I'll try and figure out the odds and ends here.
The nvtboot.img (NVidia Tegra Boot) is directly loaded onto the processor (likely by the ROM?). The image is based at 0x40000000 with variables at 0x60000000, and disassembles pretty easily thanks to verbose assertions (@0x4000C104) and printf()s (@0x4000c014) with strings starting at 0x40019978. It is an ARM32 (no Thumb-2 here) binary, and it doesn't bother with setting all the ARM exception vectors - only _reset, which it uses to jump +4 bytes (over where _undef would be) right into the TegraBoot code - the interesting portion of NvtBootMain starts around 0x40000254. It does the usual stuff one would expect (initialize CPU, board, battery, determine charger mode..), and calls 0x400054EC (a.k.a NvTbootTosInit) to load the "Secure Os" (q.v 0x4000531C) - which is the tos.img.
The tos (Trusted Operating System?) image gets written to the TOS partition. I'm assuming this is for the ARM trust zone. You can pull the same dd trick from aboot, this time with bs=0x0100200 skip=1, to get the exception vectors and image (this time, rebase at 0x4800000). The resulting file is none other than lk (as can be evidenced by strings), although with some modifications.
The lk can load "trusty_apps", which are ELF binaries. Indeed, the ELF header magic (\177ELF) can be found at several offsets:
Again, with the same trick, we have:
readelf, objdump and their ilk will usually ignore any ELF headers but the first one, though you'd need to use dd several times to extract the ELF binaries, as there is clearly more than one. Examining the binaries further corroborates they're meant for TrustZone (heck, one is actually called "trusty"!) but is left outside the scope of this writeup.
HBoot is closed source (as far as I know), but who needs source when you have disassembly? Looking at the file it's clearly evident this is an ARM (32) binary. Yup. Even though it's AArch64:
The "eaXXXXXX" words are recognizable as ARM 32 instructions - "e" is a condition (specifying 'always') and 'a' denotes a branch - which makes sense, because these are the instructions to override the ARM exception vector (as discussed in the aboot article). Loading into a disassembler (e.g. IDA), we have:
And note that "0x70000000", which tells us where we need to rebase the image (also q.v. TheIphoneWiki's iBoot discussion - all ARM bootloaders are essentially similar). All the bootloader strings one would expect (e.g. those used in fastboot, and the unlock warning) are towards the end of the image, so disassembly isn't too difficult (though IDA, alas, leaves much to be desired). If anyone's interested in tips for the actual disassembly and (partial) symbolication, feel free to drop me a line.
On the flash
Although the /fstab.flounder contains a specific entry for "/bootloader", and alleges it's in the EBT labeled partition, the partition appears to be hidden from the by-name representation (at least on the device I got, there's no ..../by-name/EBT). Looking through the raw disk image (mmcblk0), though, the boot loader code can be found at 0x80000 (compare with previous output):
Indeed, this is corroborated by looking through the GPT (also at the beginning of mmcblk0), which clearly contains the EBT label.
hboot, not lk, implements fastboot commands
fastboot oem verify will put the boot loader into a "veified state" (hey - that's HTC, not me :-), which has the side effect of A) wiping data and B) relocking the device..
The Nexus 9 uses zram - compressed memory as swap - a feature surprisingly similar to iOS 7+'s memory compressor. The compressed RAM is visible as a block device (/dev/block/zram0, with a size of 512MB. The swap is activated by init with the directive of swapon_all /fstab.flounder in the /init.flounder.rc. The /fstab.flounder contains the file system entry:
Unsurprisingly, Volantis runs a 3.10.x kernel. This was to be expected - It was high time for Android to catch up with Linux, and a post 3.8 kernel was needed to support AArch64 anyway. I'm guessing Google at least patched the well known futex bug here (i.e. TowelRoot).
And.. The kernel is open sourced, (and we just started our descent! It worked :-) so any further discussion here is moot.
If anyone has a Nexus 9 + 3G/LTE - I'd appreciate the baseband partitions, to see what's different.
Feel free to send questions, corrections or other feedback (if you have any) to me (j)