Notes from a Nexus 9
Jonathan Levin, http://NewAndroidBook.com, 12/05/14
As part of the devices I researched for my book on Android Internals, I got myself a Nexus 9. Although Google made the filesystem image available, there's no 64-Bit emulator (at least, none that I know of), and most of the interesting stuff is found during runtime. Plus, the device is unlockable, getting root is an easy matter, and the full internals of the system would be interesting to explore.
The version the Nexus 9 presently runs (it updates automatically as you first boot it) is LRX21R - technically 5.0.0._r7, and though the 5.0.1 image has been released, it's surprisingly unavailable for the Nexus 9 as an OTA (having checked for system updates 12/4/14, 20:19PM). One would think Google wouldn't have to do so many silent bugfixes after having delayed for so long.
A detailed teardown of the Nexus 9 and a hardware-perspective was already published in several places around the 'Net, so that's not my intent. Rather, this summarizes findings from snooping around the file system I did to kill time on a long trans-pacific with no better distraction the in-flight entertainment... Plus, this is somewhat of a complement to the article I wrote about disassembling aboot - which was SnapDragon (msm) oriented - whereas here we're dealing with Tegra.
Partitions
As with all Android devices (as discussed in Chapter 2 of the book), the MMC is partitioned into far more than the mountable partitions of
root@flounder: /# ls -Ll /dev/block/platform/sdhci-tegra.3/by-name/ brw------- 1 root root 259, 13 Nov 30 23:26 APP -> ..29 /system brw------- 1 root root 259, 14 Nov 30 23:26 CAC -> ..30 /cache brw-rw---- 1 system system 259, 7 Nov 30 23:26 CDR -> ..23 brw------- 1 root root 259, 4 Nov 30 23:26 DIA -> ..20 brw------- 1 root root 179, 5 Nov 30 23:26 DTB -> ..5 (normally) Device Tree (but empty) brw-rw---- 1 system system 259, 5 Nov 30 23:26 EF1 -> ..21 brw-rw---- 1 system system 259, 6 Nov 30 23:26 EF2 -> ..22 brw------- 1 root root 179, 3 Nov 30 23:26 EKS -> ..3 brw------- 1 root root 179, 11 Nov 30 23:26 EXT -> ..11 brw------- 1 root root 179, 12 Nov 30 23:26 FST -> ..12 brw------- 1 root root 259, 17 Nov 30 23:26 GPT -> ..33 GUID Partition Table (backup) brw------- 1 root root 179, 1 Nov 30 23:26 KEY -> ..1 brw------- 1 root root 259, 0 Nov 30 23:26 LNX -> ..16 boot.img (with HTC wrap) brw------- 1 root root 259, 9 Dec 1 01:25 MD1 -> ..25 brw------- 1 root root 259, 10 Nov 30 23:26 MD2 -> ..26 brw------- 1 root root 259, 2 Nov 30 23:26 MFG -> ..18 Manufacturing Data brw------- 1 root root 259, 1 Nov 30 23:26 MSC -> ..17 Misc brw------- 1 root root 179, 10 Nov 30 23:26 NCT -> ..10 brw------- 1 root root 259, 12 Nov 30 23:26 OTA -> ..28 OTA Updates brw------- 1 root root 179, 14 Nov 30 23:26 PG1 -> ..14 brw------- 1 system system 259, 11 Dec 5 01:04 PST -> ..27 Persistent brw-rw---- 1 system system 179, 8 Nov 30 23:26 RCA -> ..8 brw------- 1 root root 179, 6 Nov 30 23:26 RV1 -> ..6 ? brw------- 1 root root 179, 13 Nov 30 23:26 RV2 -> ..13 brw------- 1 root root 259, 16 Nov 30 23:26 RV3 -> ..32 brw------- 1 root root 259, 3 Nov 30 23:26 SER -> ..19 brw------- 1 root root 179, 15 Nov 30 23:26 SOS -> ..15 recovery.img (cute :-) brw------- 1 root root 179, 9 Nov 30 23:26 SP1 -> ..9 brw------- 1 root root 179, 2 Nov 30 23:26 TOS -> ..2 ARM TrustZone? See below brw------- 1 root root 259, 15 Nov 30 23:26 UDA -> ..31 User data (i.e /data) brw------- 1 root root 259, 8 Nov 30 23:26 VNR -> ..24 /vendor brw------- 1 root root 179, 4 Nov 30 23:26 WB0 -> ..4 brw------- 1 root root 179, 7 Nov 30 23:26 WDM ->7
Quite a few of the partitions are empty, at least on the WiFi-only model. I'm assuming that the situation might be different on the cellular-enabled version, since the baseband likely has need for some partitions.
DTB
The DTB partition was supposed to hold the compiled device tree - which instead is appended to the kernel image. To get the device tree you can use imgtool (though you'd need to strip the 0x100 byte hboot header first).
Misc (MSC)
The MSC partition is the classic Android adb reboot with recovery or bootloader (personal note: I'm surprised PST wasn't merged into this - see below).
Persist (PST)
The Nexus 9 L image contains the persistent_data_block service (android.service.persistentdata.IPersistentDataBlockService) which is not present in the Emulator (another reason why I needed a real device - you can't write a book about Android based on just the emulator images). This service (defined in the AOSP's fastboot oem unlock command will be allowed by the boot loader. The Settings app calls on the service's setOemUnlockEnabled(boolean enabled); to disable or enable the fastboot oem unlock functionality.
The setting naturally has to be read by hboot/aboot - which means it has to reside on a raw partition. Indeed, it does - on the Persistent partition, labeled PST. The service gets the partition's name from the ro.frp.pst property, which is set in the "ADDITIONAL_BUILD_PROPERTIES" of the
The partition is entirely empty if the OEM unlock is disallowed (as it is by default), but if the setting is toggled, a single bit magically appears in the last word:
# Before
root@flounder:/ # od -A x -t x4 /dev/block/platform/sdhci-tegra.3/by-name/PST
000000 00000000 00000000 00000000 00000000 # No oem unlock - nada
*
080000
# After
root@flounder:/ # od -A x -t x4 /dev/block/platform/sdhci-tegra.3/by-name/PST
000000 00000000 00000000 00000000 00000000 # Still entirely empty...
* # ...
07fff0 00000000 00000000 00000000 01000000 # ... save 1 bit
080000
The boot loader's fastboot oem get_unlock_ability can be used to test the bit (as can fastboot oem unlock, of course).
MFG
MFG contains various manufacturing data pertaining to the device. The beginning of the partition has a(n as yet unknown) header, followed by variable=value , padded by a lot of null bytes. Somewhere deep in the partition you can find the device manufacturing date (201410xx or 201411xx), MB and S/N (as per the sticker on the back) but surprisingly not the IMEI or the P/N (unless I missed it).
Mounted File Systems
The
L finally uses the function file system, a successor to the gadgetFS/ConfigFS which serve the functions of the USB gadget driver, which it can adopt as part of the 3.10 kernel offerings. Another notable addition is pstore, which takes over the deprecated ram console - if the kernel ever panics, the log can be found in this directory. As with the experiment in chapter 2:
# This will crash your system. Make sure you close apps first..
root@flounder:/ # echo c > /proc/sysrq-trigger
# System will automatically reboot. Reconnect adb, and voila:
root@flounder:/ # ls -l /sys/fs/pstore
# Permissions were set by /init.rc
-r--r----- system log 106835 2014-12-11 07:21 console-ramoops
I also cover most of the filesystems in the book (Chapter 2), but the following output shows the new ones:
root@flounder: /# mount rootfs / rootfs ro,seclabel,relatime 0 0 tmpfs /dev tmpfs rw,seclabel,nosuid,relatime,mode=755 0 0 devpts /dev/pts devpts rw,seclabel,relatime,mode=600 0 0 none /dev/cpuctl cgroup rw,relatime,cpu 0 0 adb /dev/usb-ffs/adb functionfs rw,relatime 0 0 proc /proc proc rw,relatime 0 0 sysfs /sys sysfs rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 none /sys/fs/cgroup tmpfs rw,seclabel,relatime,mode=750,gid=1000 0 0 pstore /sys/fs/pstore pstore rw,relatime 0 0 /sys/kernel/debug /sys/kernel/debug debugfs rw,relatime,mode=755 0 0 none /acct cgroup rw,relatime,cpuacct 0 0 tmpfs /mnt/asec tmpfs rw,seclabel,relatime,mode=755,gid=1000 0 0 tmpfs /mnt/obb tmpfs rw,seclabel,relatime,mode=755,gid=1000 0 0 # The actual mounted partitions are here: /dev/block/platform/sdhci-tegra.3/by-name/APP /system ext4 ro,seclabel,relatime,data=ordered 0 0 /dev/block/platform/sdhci-tegra.3/by-name/VNR /vendor ext4 ro,seclabel,relatime,data=ordered 0 0 /dev/block/platform/sdhci-tegra.3/by-name/CAC /cache ext4 rw,seclabel,nosuid,nodev,noatime,errors=panic,data=ordered 0 0 # /data: Note A) mounting over device mapper (encrypted) and B) f2fs /dev/block/dm-0 /data f2fs rw,seclabel,nosuid,nodev,noatime,background_gc=on,user_xattr,acl,errors=panic,active_logs=6 0 0 # SDCard.. /dev/fuse /mnt/shell/emulated fuse rw,nosuid,nodev,relatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0 tmpfs /storage/emulated tmpfs rw,seclabel,nosuid,nodev,relatime,mode=050,gid=1028 0 0 /dev/fuse /storage/emulated/0 fuse rw,nosuid,nodev,relatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0 /dev/fuse /storage/emulated/0/Android/obb fuse rw,nosuid,nodev,relatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0 /dev/fuse /storage/emulated/legacy fuse rw,nosuid,nodev,relatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0 /dev/fuse /storage/emulated/legacy/Android/obb fuse rw,nosuid,nodev,relatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0
The Boot Loader
Unlike MSM devices, which reads images like aboot directly (in the format specified in the previous article), HTC uses their own loader - HBOOT - with a small custom header on images, which can be found prepended to both the aboot images (LNX and SOS), as well as the boot loader itself (bootloader-flounder-3.43.0.0114.img in the images supplied by Google). The header can be seen in the following output:
morpheus@Forge (~)$ od -A x -t x4 boot.img | more
0000000 12345678 00000000 007a6300 00000100
MAGIC ?? MBZ? File Size Hdr Size?
0000010 00000100 007a6100 007a6200 00000100
Hdr Size? PS - 0x100 Payload size ???
0000020 00000000 00000000 00000000 00000000
*
0000100 52444e41 2144494f 006ef842 10008000
# ANDROID! ... etc..
It's not entirely clear to the author why there appears to be so much redundancy in the header (e.g. why is 0x100 repeated three times? Header Size? Data Start? What else?). But that's a non issue. Once you figure out the actual data starts at 0x100, the custom header can be stripped, leaving you (in the case of LNX/SOS) with an aboot image, or (for bootloader-flounder-xxxx.img) - a zip file! This becomes evident by the "PK" signature, which is correctly identified by file(1):
morpheus@Forge (~/.../volantis-lrx21l) % dd if=bootloader-flounder-3.43.0.0114.img bs=0x100 \
skip=1 of=out
11879+1 records in
11879+1 records out
3041194 bytes transferred in 0.026451 secs (114974151 bytes/sec)
morpheus@Forge (~/...//volantis-lrx21l) % file out
out: Zip archive data, at least v2.0 to extract
morpheus@Forge (~/.../volantis-lrx21l) % unzip -l out
Archive: out
Length Date Time Name
-------- ---- ---- ----
253 10-10-14 17:05 _android-info.txt
25744 08-25-14 09:51 mts_preboot_prod
1588112 08-25-14 09:51 mts_prod
8704 10-16-14 10:46 rawbct.img
1046032 10-16-14 10:46 hboot.img
125072 10-16-14 10:46 nvtboot.img
2048 08-04-14 10:29 nvtbootwb0.bin
2650880 10-16-14 11:38 tos.img
6291456 08-04-14 10:29 sp1.nb0
1048576 08-20-14 10:09 gp1.img
524288 08-20-14 10:09 pt_16g.img
524288 08-20-14 10:09 pt_32g.img
524288 08-20-14 10:09 pt_64g.img
-------- -------
14359741 13 files
And voilà! All the boot loader components. I haven't fully researched the exact order of these, but TOS precedes hboot, and nvtboot.img (which is loaded at 0x40000000, before either) precedes them both (i.e. nvtboot → (+wb0) → tos → hboot). Given time and the NVidia docs I'll try and figure out the odds and ends here.
nvtboot.img
The nvtboot.img (NVidia Tegra Boot) is directly loaded onto the processor (likely by the ROM?). The image is based at printf()s (NvTbootTosInit) to load the "Secure Os" (q.v
tos.img
The tos (Trusted Operating System?) image gets written to the TOS partition. I'm assuming this is for the ARM trust zone. You can pull the same dd trick from aboot, this time with bs=0x0100200 skip=1, to get the exception vectors and image (this time, rebase at 0x4800000). The resulting file is none other than lk (as can be evidenced by strings), although with some modifications.
The lk can load "trusty_apps", which are ELF binaries. Indeed, the ELF header magic (\177ELF) can be found at several offsets:
od -t c -t x4 -A x Partitions/TOS | grep "E" | grep "L" | grep "F" | grep 177
010bdf0 A G E _ M A S K ) \0 177 E L F \0 t # as a string
010e200 177 E L F 001 001 001 \0 \0 \0 \0 \0 \0 \0 \0 \0
0115500 \0 \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 177 E L F
017e810 001 \0 \0 \0 \0 \0 \0 \0 177 E L F 001 001 001 \0
01e3b20 \0 \0 \0 \0 177 E L F 001 001 001 \0 \0 \0 \0 \0
01eae30 177 E L F 001 001 001 \0 \0 \0 \0 \0 \0 \0 \0 \0
0256130 \0 \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 177 E L F
Again, with the same trick, we have:
dd if=Partitions/TOS bs=0x010e200 skip=1 of=toself
1+1 records in
1+1 records out
2039296 bytes transferred in 0.002057 secs (991471818 bytes/sec)
morpheus@Forge (~/Documents/Android/Images/volantis-lrx21l) %file toself 7:58
toself: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, stripped
readelf, objdump and their ilk will usually ignore any ELF headers but the first one, though you'd need to use dd several times to extract the ELF binaries, as there is clearly more than one. Examining the binaries further corroborates they're meant for TrustZone (heck, one is actually called "trusty"!) but is left outside the scope of this writeup.
hboot.img
HBoot is closed source (as far as I know), but who needs source when you have disassembly? Looking at the file it's clearly evident this is an ARM (32) binary. Yup. Even though it's AArch64:
morpheus@Forge (~/.../volantis-lrx21l) % od -t x4 xx/hboot.img| head -5
0000000 ea000007 ea016f8c ea016f8c ea016f8a
0000020 ea016f89 ea016f88 ea016f78 ea016f86
0000040 00000000 e32ff0df e3a00207 e590c804
The "eaXXXXXX" words are recognizable as ARM 32 instructions - "e" is a condition (specifying 'always') and 'a' denotes a branch - which makes sense, because these are the instructions to override the ARM exception vector (as discussed in the aboot article). Loading into a disassembler (e.g. IDA), we have:
ROM:00000000
ROM:00000000 loc_0 ; DATA XREF: sub_472A0+BAw
ROM:00000000 ; sub_472A0+C2uw ...
ROM:00000000 B loc_24
ROM:00000004 ; ---------------------------------------------------------------------------
ROM:00000004 B loc_5BE3C
ROM:00000008 ; ---------------------------------------------------------------------------
ROM:00000008 B locret_5BE40
ROM:0000000C ; ---------------------------------------------------------------------------
ROM:0000000C B loc_5BE3C
ROM:00000010 ; ---------------------------------------------------------------------------
ROM:00000010 B loc_5BE3C
ROM:00000014 ; ---------------------------------------------------------------------------
ROM:00000014 B loc_5BE3C
ROM:00000018 ; ---------------------------------------------------------------------------
ROM:00000018 B loc_5BE00
ROM:0000001C ; ---------------------------------------------------------------------------
ROM:0000001C B loc_5BE3C
ROM:00000020 ; ---------------------------------------------------------------------------
ROM:00000020 ANDEQ R0, R0, R0
ROM:00000024
ROM:00000024 loc_24 ; CODE XREF: ROM:loc_0j
ROM:00000024 MSR CPSR_cxsf, #0xDF ; '¯'
ROM:00000028 MOV R0, #0x70000000
ROM:0000002C LDR R12, [R0,#0x804]
And note that "0x70000000", which tells us where we need to rebase the image (also q.v. TheIphoneWiki's iBoot discussion - all ARM bootloaders are essentially similar). All the bootloader strings one would expect (e.g. those used in fastboot, and the unlock warning) are towards the end of the image, so disassembly isn't too difficult (though IDA, alas, leaves much to be desired). If anyone's interested in tips for the actual disassembly and (partial) symbolication, feel free to drop me a line.
On the flash
Although the
morpheus@Forge (~/.../Nexus9/Partitions) % od -A x -t x4 blk0 | grep ea000007
0080000 ea000007 ea016f8c ea016f8c ea016f8a
0081220 e59f11e8 e3a02ec2 ea000007 e59f11e0
00cd0a0 ea000007 e3a00003 e28d1008 e3a02010
00cdd20 908ff100 ea0000c3 ea000007 ea000060
...
Indeed, this is corroborated by looking through the GPT (also at the beginning of mmcblk0), which clearly contains the EBT label.
fastboot commands
fastboot oem verify will put the boot loader into a "veified state" (hey - that's HTC, not me :-), which has the side effect of A) wiping data and B) relocking the device..
... (bootloader) Start Verify: 0 (bootloader) [hboot query] query 24 is not implemented (bootloader) [hboot query] query 24 is not implemented (bootloader) Start Verify: 0 (bootloader) [hboot query] query 24 is not implemented (bootloader) [hboot query] query 24 is not implemented (bootloader) Device is in veified state!
Swap
The Nexus 9 uses zram - compressed memory as swap - a feature surprisingly similar to iOS 7+'s memory compressor. The compressed RAM is visible as a block device (
/dev/block/zram0 none swap defaults zramsize=533413200
The kernel
Unsurprisingly, Volantis runs a 3.10.x kernel. This was to be expected - It was high time for Android to catch up with Linux, and a post 3.8 kernel was needed to support AArch64 anyway. I'm guessing Google at least patched the well known futex bug here (i.e. TowelRoot).
root@flounder:/ # cat /proc/cpuinfo Processor : NVIDIA Denver 1.0 rev 0 (aarch64) processor : 0 processor : 1 Features : fp asimd aes pmull sha1 sha2 crc32 CPU implementer : 0x4e CPU architecture: AArch64 CPU variant : 0x0 CPU part : 0x000 CPU revision : 0 Hardware : Flounder Revision : 0000 Serial : 0000000000000000 MTS version : 33410787 root@flounder:/ # cat /proc/version Linux version 3.10.40-ga3846f1 (android-build@vpbs1.mtv.corp.google.com) (gcc version 4.9.x-google 20140827 (prerelease) (GCC) ) #1 SMP PREEMPT Thu Nov 6 01:39:45 UTC 2014
And.. The kernel is open sourced, (and we just started our descent! It worked :-) so any further discussion here is moot.
Comments/Requests
- If anyone has a Nexus 9 + 3G/LTE - I'd appreciate the baseband partitions, to see what's different.
- Feel free to send questions, corrections or other feedback (if you have any) to me (j)
- Buy the book ;-) (or better yet, get our training about it!)