Decrypting Android M adopted storage

One of the new features Android M introduces is adoptable storage. This feature allows external storage devices such as SD cards or USB drives to be 'adopted' and used in the same manner as internal storage. What this means in practice is that both apps and their private data can be moved to the adopted storage device. In other words, this is another take on everyone's (except for widget authors...) favorite 2010 feature -- AppsOnSD. There are, of course, a few differences, the major one being that while AppsOnSD (just like app Android 4.1 app encryption) creates per-app encrypted containers, adoptable storage encrypts the whole device. This short post will look at how adoptable storage encryption is implemented, and show how to decrypt and use adopted drives on any Linux machine.

Adopting an USB drive

In order to enable adoptable storage for devices connected via USB you need to execute the following command in the Android shell (presumably, this is not needed if your device has an internal SD card slot; however there are no such devices that run Android M at present):

$ adb shell sm set-force-adoptable true

Now, if you connect a USB drive to the device's micro USB slot (you can also use an USB OTG cable), Android will give you an option to set it up as 'internal' storage, which requires reformatting and encryption. 'Portable' storage is formatted using VFAT, as before.


After the drive is formatted, it shows up under Device storage in the Storage screen of system Settings. You can now migrate media and application data to the newly added drive, but it appears that there is no option in the system UI that allows you to move applications (APKs).


Adopted devices are mounted via Linux's device-mapper under /mnt/expand/ as can be seen below, and can be directly accessed only by system apps.

$ mount
rootfs / rootfs ro,seclabel,relatime 0 0
...
/dev/block/dm-1 /mnt/expand/a16653c3-... ext4 rw,dirsync,seclabel,nosuid,nodev,noatime 0 0
/dev/block/dm-2 /mnt/expand/0fd7f1a0-... ext4 rw,dirsync,seclabel,nosuid,nodev,noatime 0 0

You can safely eject an adopted drive by tapping on it in the Storage screen, and the choosing Eject from the overflow menu. Android will show a persistent notification that prompts you to reinsert the device once it's removed. Alternatively, you also can 'forget' the drive, which removes it from the system, and should presumably delete the associated encryption key (which doesn't seem to be the case in the current preview build).

Inspecting the drive

Once you've ejected the drive, you can connect it to any Linux box in order to inspect it. Somewhat surprisingly, the drive will be automatically mounted on most modern Linux distributions, which suggests that there is at least one readable partition. If you look at the partition table with fdisk or a similar tool, you may see something like this:

# fdisk /dev/sdb
Disk /dev/sdb: 7811 MB, 7811891200 bytes, 15257600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt


 #        Start          End    Size  Type            Name
 1         2048        34815     16M  unknown         android_meta
 2        34816     15257566    7.3G  unknown         android_expand

As you can see, there is a tiny android_meta partition, but the bulk of the device has been assigned to the android_expand partition. Unfortunately, the full source code of Android M is not available, so we cannot be sure how exactly this partition table is created, or what the contents of each partition is. However, we know that most of Android's storage management functionality is implemented in the vold system daemon, so we check if there is any mention of android_expand inside vold with the following command:

$ strings vold|grep -i expand
--change-name=0:android_expand
%s/expand_%s.key
/mnt/expand/%s

Here expand_%s.key suspiciously looks like a key filename template, and we already know that adopted drives are encrypted, so our next step is to look for any similar files in the device's /data partition (you'll need a custom recovery or root access for this). Unsurprisingly, there is a matching file in /data/misc/vold which looks like this:

# ls /data/misc/vold
bench
expand_8838e738a18746b6e435bb0d04c15ccd.key

# ls -l expand_8838e738a18746b6e435bb0d04c15ccd.key

-rw-------  1 root root 16  expand_8838e738a18746b6e435bb0d04c15ccd.key


# od -t x1 expand_8838e738a18746b6e435bb0d04c15ccd.key
0000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
0000020

Decrypting the drive

That's exactly 16 bytes, enough for a 128-bit key. As we know, Android's FDE implementation uses an AES 128-bit key, so it's a good bet that adoptable storage uses a similar (or identical) implementation. Looking at the start and end of our android_expand partition doesn't reveal any readable info, nor is it similar to Android's crypto footer, or LUKS's header. Therefore, we need to guess the encryption mode and/or any related parameters. Looking once again at Android's FDE implementation (which is based on the dm-crypt target of Linux's device-mapper), we see that the encryption mode used is aes-cbc-essiv:sha256. After consulting dm-crypt's mapping table reference, we see that the remaining parameters we need are the IV offset and the starting offset of encrypted data. Since the IV offset is usually zero, and most probably the entire android_expand partition (offset 0) is encrypted, the command we need to map the encrypted partition becomes the following:

# dmsetup create crypt1 --table "0 `blockdev --getsize /dev/sdb2` crypt \
aes-cbc-essiv:sha256 00010203040506070809010a0b0c0d0e0f 0 /dev/sdb2 0"

It completes with error, so we can now try to mount the mapped device, again guessing that the file system is most probably ext4 (or you can inspect the mapped device and find the superblock first, if you want to be extra diligent).

# mount -t ext4 /dev/mapper/crypt1 /mnt/1/
# cd /mnt/1
# find ./ -type d
./
./lost+found
./app
./user
./media
./local
./local/tmp
./misc
./misc/vold
./misc/vold/bench

This reveals a very familiar Android /data layout, and you should see any media and app data you've moved to the adopted device. If you copy any files to the mounted device, they should be visible when you mount the drive again in Android.

Storage manager commands

Back in Android, you can use the sm command (probably short for 'storage manager') we showed in the first section to list disks and volumes as shown below:

$ sm list-disks
disk:8,16
disk:8,0

$ sm list-volumes
emulated:8,2 unmounted null
private mounted null
private:8,18 mounted 0fd7f1a0-2d27-48f9-8702-a484cb894a92
emulated:8,18 mounted null
emulated unmounted null
private:8,2 mounted a16653c3-6f5e-455c-bb03-70c8a74b109e

If you have root access, you can also partition, format, mount, unmount and forget disks/volumes. The full list of supported commands is shown in the following listing.

$ sm
usage: sm list-disks
       sm list-volumes [public|private|emulated|all]
       sm has-adoptable
       sm get-primary-storage-uuid
       sm set-force-adoptable [true|false]

       sm partition DISK [public|private|mixed] [ratio]
       sm mount VOLUME
       sm unmount VOLUME
       sm format VOLUME

       sm forget [UUID|all]

Most features are also available from the system UI, but sm allows you to customize the ratio of the android_meta and android_expand partitions, as well as to create 'mixed' volumes.

Summary

Android M allows for adoptable storage, which is implemented similarly to internal storage FDE -- using dm-crypt with a per-volume, static 128-bit AES key, stored in /data/misc/vold/. Once the key is extracted from the device, adopted storage can be mounted and read/written on any Linux machine. Adoptable storage encryption is done purely in software (at least in the current preview build), so its performance is likely comparable to encrypted internal storage on devices that don't support hardware-accelerated FDE.

Comments

Lukas said…
Very informative, thanks!

Could you possibly elaborate a bit more on how adoptable storage is exposed to applications? The preview SDK docs are unfortunately not very helpful.

- Is an encrypted card usable for media files without migrating just like an unencrypted card, i.e. is migration decoupled from encryption?
- Where is the primary shared storage mounted, and does that change after migrating application data? What does Context.getExternalFilesDirs() return in each case?
- After migration, is the internal (emulated) shared storage still accessible (via file managers and/or the external storage APIs), or is it completely shadowed by the external shared storage?
- Where are APK expansion files (OBBs) stored, before and after migration?
Unknown said…
This comment has been removed by the author.
Unknown said…
Encryption is separate from migration. You can have multiple encrypted volumes mounted and migrate to only one (or none) of them. External storage is implemented the same way as before, i.e. emulated using FUSE on top of the media/ directory of the 'real' volume. It does look you can only have one 'primary external storage' at a time, but not 100% about this without the code.

You can write a simple app to check how the Contexts methods behave
Unknown said…
Is it Possible to decyrpt and save data from the adopted SD card without being able to extract the key from the phone. For instance broken phone trying to save pictures of the kids. Thank you
Sam Liddicott said…
So maybe use dmcrypt and then resize the ext4 file system, then close the crypt session and resize the partition, and then add a new vFAT partition as External SD that Google Music can store on?
Sam Liddicott said…
You had the Holy Grail without realising it.

Many people have been stuck with adopted internal storage, because they then have no SD card to which Google Play Music can store the music; and as Google Play Music won't move to the adopted SD card, all music is stored in the internal memory. For me, I have more music than Apps, so this just makes things worse.

However your reference to "sm" and "mixed" gave me something to work with.

$ adb shell sm list-disks adoptable
disk:179_64
$ sm partition disk:179_64 mixed 75
$


gives me 75% as portable storage.

However I advise a reboot after setting the new music storage location this as Google Music may get the wrong idea about much space is available.

http://blog.sam.liddicott.com/2016/02/android-6-semi-adopted-storage.html
Unknown said…
Will it be possible to use this command to add system storage to let's say the new Galaxy s7 or the G5?
PK Mehta said…
My adopted memory card stopped working. So I pressed 'Forget the card'.
Now I want to use the card as portable in my phone.
Can't seem to format it either as portable card or internal card.
Will this technique help? If yes, then great I'm gonna try it and let you know.
If no please guide me through the way which may help me.
Sam Liddicott said…
It sounds to me like your adopted card stopped working because the card is damaged, and this is why it also doesn't work as portable storage.

I would say that you probably need a new card.
Sam Liddicott said…
It sounds to me like your adopted card stopped working because the card is damaged, and this is why it also doesn't work as portable storage.

I would say that you probably need a new card.
Unknown said…
Hi, I configured my micro sd as internal storage in my moto g3 and I foolishly reseted the phone. So, now I can't access to all my photos because android tells me that I have to format the sd. The problem is that I really need to recover all my information. Is there any way in which I could unlocked the sd?
I'll apreciate your help since I am really desperate and nobody can give me any solution.
Thanks.
Maitreya Hegde said…
Bought a new Samsung 16GB SD Card. When i try to set-it up and format it as internal memory, settings app crashes and later when i open, it says SD card Corrupted. Im running on CM13 CONDOR NIGHTLY. Thanks in advance.
Unknown said…
Thank you sir!!! You saved my life. Android wouldn't mount my microsd card anymore due to some corrupt directories. (I used adb shell and ran dmesg to see what's going on). I was able to backup over 18GB of precious family pictures by decrypting it from my ubuntu desktop. You are awesome!
Emanuel said…
This comment has been removed by the author.
Mirco said…
Doesn't work for me. I got this on my ubunut 14.04:

command:
dmsetup create mmc --table "0 62299103 crypt aes-cbc-essiv_sha256 1hereismykey1 0 /dev/mmcblk0p2 0"

the result is

"device-mapper: reload ioctl on mmc failed: Invalid argument
Command failed"

dmesg says:
[119671.798439] device-mapper: table: 252:3: crypt: Invalid IV mode
[119671.798447] device-mapper: ioctl: error adding target to table


Any suggestions?
thx


Unknown said…
Should be 'aes-cbc-essiv:sha256'. Also I very much doubt that Ubuntu has a `mmcblk0p2` device. Check dmesg for the correct device name.
Mirco said…
Thanks a lot for pointing that out...what a silly typing error.

just for the record:
the filesystem on the encrypted card was "f2fs" in my case.
I am using Marshmellow 6.0.1
Unknown said…
Hi,
Trying to decrypt my SD card so I can expand a partition here, but when I run:

dmsetup create crypt1 --table "0 `blockdev --getsize /dev/mmcblk0p2` crypt \aes-cbc-essiv:sha256 0...f 0 /dev/mmcblk0p2 0"

I get the error:

device-mapper: reload ioctl on crypt1 failed: No such file or directory

and I'm not quite sure what to make of it. Could you point me in the right direction?

Cheers.
Raz said…
This comment has been removed by the author.
subscat said…
hello I have a problem with my sd card, I had configured in android marshmallow like internal storage but I had to reinstall the rom and when I put the micro sd card I get as damaged and asks me to format but have important files , some form of they rescue these files or lost forever
Unknown said…
Did you find any solution? I'm stuck on the same problem here
Chad said…
Hi thanks for your contribution. I would like to consult you something because in my case I have a little more complicated. It has happened the same case as you, but additionally I did a hard reset the cell phone, which, I think I do not have that .key file. Without the key file there is no possibility that can decrypt the file system?
Hertz! said…
Amazing article, very informative. would you know if the implementation has changed in Android N ? Is still possible to extract keys ?

Thanks!
Unknown said…
Hello all,
I have recently upgraded my phone to Android 6.0
Everything worked fine until i formatted my SD card to Internal.
After the conversion, it was working well. I was pretty much satisfied. With that confidence, i moved all my apps from Internal to SD card.
After some minutes my phone got struck, so i restarted by removing and placing the battery in the back.

When it booted up it didn't detect my SD card and kept showing an error that, i need to Forget my SD card.
i googled the condition and found a post explaining the issue that it is Android 6.0's bug, and it's highly recommended to not move apps from Internal to SD card without a reboot, after converting the SD card to Internal.

The post also explained about wiping the SD card, as the only option left, and instructed to fastboot the device.

For some unknown reasons, the default "hold press Power button and press Volume Down" didn't work in my phone and it took me to a meta mode, where nothing is displayed apart from saying that it is a meta mode.

Then i took a wrong turn by repeating the same by pressing the Power and Volume Up button. It took me to factory mode, where i was displayed with 6 options in Chinese.

I can't read Chinese, and one of the option had an English word "eMMC", i thought this was the option to wipe out the SD card and selected it. It restored my phone to factory settings by erasing the Internal memory instead of the SD card.

Now, my phone detects my SD card, but shows it as "Unsupported" under Storage and USB option. And it asks me to Set Up the SD card by formatting it.
There are two options, "Use as portable storage" and "Use as Internal Storage".
If i select the first option, it says SD card is ready. But it doesn't format or fix the errors . And if i go for the 2nd option, it doesn't format and it displays the following error as a Toast.

Couldn't erase SD card
command '46 volume partition disk: 179,128 private' failed with '400 46 Command failed'

This is a Strontium 32GB class 10 SD Card. Kindly help me to format the SD Card and fix it's errors so i can use it again.
Unknown said…
Hi, my adobted card stopped working. Is is possible to reveal the encrypting key on my phone? It is not rooted. For rooting it needs to have an opened bootloader, but if I want to open it the phone is completely factory resettled - and the key deleted?
Unknown said…
Hi
Some months ago, I had to do a hard reset on my phone, since then I have my 32gb sd card encrypted, it was originally set as adopted storage.
After a lot of search, I'm still trying to recover such important files that I have in this sd card, after I fone The hard reset on my phone, is there any way to save the data on The sd? If I lost the encryption key, is there any chance to recover my sd data?
Unknown said…
Can you tell me, please.
Is it possible to recover data from the old card, if I clicked "forget" the disk and replaced it with a new one. Whether the result of this operation, a new encryption key is created and with it it is impossible to do?
Unknown said…
This comment has been removed by the author.
Hinduraja said…
When i m trying command sm string vold ] grep ~i expand i m gettin error string not found
When set adoptable is done to true afyer has adoptable command it again shows false
Unable to set it to true
Please help hard earned money sdcard waste
Tried cmd prompt and all formatters
Adb shell method also fail
efebo_abel said…
is there a way to get the SD data/files after a factory reset and not having the .key file ?
Unknown said…
Hello runner, can you please tell me how you were able to decrypt and backup data from SD card. I am stuck in same kind of situation and I have pressed forget button as I wasn't able to access any other SD card on my phone. Now it is recognizing my SD card which was adopted as internal storage but asks to reformat it to be used as portable media or internal storage again. Please replyme on rakesh19900406@gmail.com
Unknown said…
your blog look nice and informative which I found on internet.thank you for your post.
Unknown said…
Can someone create a video tutorial? detailing the step-by-step process on how to decrypt adoptable storage? Thanks.
Wonderful, and very informative! It helps me understand the internal storage file system and encryption, however my problem is that my phone is destroyed so I'm unable to retrieve my encryption key. Is there a way to find out how the encryption key is derived, or if there is another way to retrieve it?

Thanks!
Hello and thank you for this, I'm finally hopping to get back the photos of my sd card adopted as internal storage on CM13...
But I still get an error when I try the command :
dmsetup create crypt1 --table "0 `blockdev --getsize /dev/mmcblk0` crypt \aes-cbc-essiv:sha256 0180e00...170fe15b73b4f 0 /dev/mmcblk0 0"
device-mapper: reload ioctl on crypt1 failed: No such file or directory
Command failed

in dmesg I get :
[11652.228696] device-mapper: table: 252:0: crypt: Error allocating crypto tfm
[11652.228702] device-mapper: ioctl: error adding target to table

Please help !
correction : ...my CORRUPTED sd card ...
EK said…
I also had the problem that somehow the SD-card with very important pictures/movies was no longer available in the moto G.

I've posted a possible solution in order to retrieve the data (somehow the SD-card got corrupted). I did not have a rooted phone so decryption would take years...
Read how I got it back on:

https://android.stackexchange.com/questions/174799/moto-g3-android-marshmallow-not-detecting-sd-card-anymore-solution
Sepero said…
You sir are brilliant! Please never stop
CVrino said…
Hello, tried the following command line and got error:
sudo dmsetup create crypt1 --table "0 'blockdev --getsize /dev/sdd2' crypt \aes-cbc-essiv:sha256 63d27a5ac67ec961207eda47f34aa99c 0 /dev/sdd2 0"
Invalid format on line 1 of table
Command failed

My dmesg show:
29213.292481] usb 1-1.2: Product: USB Storage
[29213.292483] usb 1-1.2: Manufacturer: Generic
[29213.292485] usb 1-1.2: SerialNumber: 000000000819
[29213.292997] usb-storage 1-1.2:1.0: USB Mass Storage device detected
[29213.293142] scsi10 : usb-storage 1-1.2:1.0
[29214.290181] scsi 10:0:0:0: Direct-Access Generic STORAGE DEVICE 0819 PQ: 0 ANSI: 6
[29214.290596] sd 10:0:0:0: Attached scsi generic sg4 type 0
[29214.457848] sd 10:0:0:0: [sdd] 61734912 512-byte logical blocks: (31.6 GB/29.4 GiB)
[29214.458941] sd 10:0:0:0: [sdd] Write Protect is off
[29214.458948] sd 10:0:0:0: [sdd] Mode Sense: 23 00 00 00
[29214.459941] sd 10:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[29214.497989] sdd: sdd1 sdd2
[29214.501822] sd 10:0:0:0: [sdd] Attached SCSI removable disk

Any idea?
Unknown said…
I get the error below.

dmsetup create sd1 --table "0 `blockdev --getsize /dev/sdb2` crypt aes-cbc-essiv:sha256 bbbbc2188b3e4a7476148a9411545715 0 /dev/sdb2 0"
device-mapper: reload ioctl on sd1 failed: Invalid argument

Any help is appreciated
Unknown said…
Hi I was using a 64 GB card as internal storage on lenovo k4 note. Had given the phone for some repairs and removed the card. However after receiving the phone back which was factory reset the card was not recognized. Even the card was formatted . Have important data on it so please help me recover the data. I am not that computer savvy so plz explain in detail..

Regards

Chinmay
Kevin said…
Thanks for these instructions. I'm hoping to test them out tomorrow. After reading your instructions carefully, I noticed that the encryption key you pass to the dmsetup command has what appears to be an extra "01" byte after the "09" byte that isn't in the od command above?

Is that typo or am I missing something?
Unknown said…
I've found that the adopted sd's files can be easily accessed from twrp recovery...idk how this is possible....tell me how that works!

Popular posts from this blog

Unpacking Android backups

Using app encryption in Jelly Bean