Storing application secrets in Android's credential storage

This article describes how to talk to the system keystore daemon directly and store app-specific secrets in the system credential storage. It will introduce private API's, not available via the Android SDK and some OS services implementation details. Those may change at any time, and are not guaranteed to work. While the techniques described have been tested on a few different devices and OS versions (2.1 to 4.0), there are no guarantees. Use caution if you decide to implement them in a production app.

As described in previous articles, Android has had a system-level credential storage since Donut (1.6). Up until ICS (4.0), it was only used by the VPN and WiFi connection services to store private keys and certificates, and a public API was not available. ICS introduced a public API  and integrated the credential storage with the rest of the OS. However, while the underlying implementation is able to store arbitrary data owned by any app, the ICS API only allows us to store private keys and certificates owned and managed by the OS. While this is could be seen as  a good thing -- it allows for tighter control over who can access what keys, it is also rather limiting. Third party apps often need to store sensitive information, such as passwords, authentication tokens and encryption keys the app uses, but the KeyChain API doesn't allow this. As mentioned in the password-based encryption article, one alternative is to derive a key from a user-supplied password and use it to encrypt sensitive data private to an application. While this works, it requires the user to remember one more password, and increases application complexity -- developers need to implement services, not directly related to app functionality; services that should ideally be provided by the system. The next Android version, reportedly just around the corner, might expose such services via public API's, but you could use them now, if you are willing to take the risk of your app breaking when Jelly Bean comes along.

Android's credential storage is implemented as a native Linux service (daemon), with a few extra layers on top of it that make it available to the framework. Let's quickly review what we know about the keystore daemon (described in more detail here):
  • it's a native daemon, started at boot
  • it provides a local control socket to allow apps and system services to talk to it
  • it encrypts keys using an AES 128 bit master key
  • encrypted keys are stored in /data/misc/keystore, one file per key
  • the master key is derived from the device unlock password or PIN
  • it authorizes administration commands execution and key access based on caller UID
Here's a quick summary of the available commands and who is permitted to execute them:

Keystore daemon commands
Command Description Allowed UIDs Parameters
test Check that the key store is in a usable state anyone but root, vpn and wifi none
get Get unencrypted key anyone (*1) key name
insert Add or overwrite key anyone but root, vpn and wifi key name and value
del Delete a key anyone but root, vpn and wifi (*1) key name
exist Check if a key exists anyone but root, vpn and wifi (*1) key name
saw List keys with the specified prefix anyone but root, vpn and wifi (*1) key prefix
reset Reset the key store system none
password Change the key store password system new password
lock Lock the key store system none
unlock Unlock the key store system none
zero Check if the key store is empty system none
*1 Only keys created with the same UID are visible/accessible

As you can see from the table above, once the credential storage is initialized and unlocked, any app can add, delete, list and get keys. Each key is bound to the UID of the process that created it, so that apps cannot access each other's keys or the system ones. Additionally, even system apps cannot see app keys, and root is explicitly prohibited from creating or listing keys. Thus, if the API were public user apps could use the credential storage to securely store their secrets, as long as it is unlocked. Unlocking, however, requires a system permission. On ICS, the credential storage is unlocked when you enter your device unlock pattern, PIN or password, so in practice the keystore daemon will be already in an unlocked state by the time your app starts. On pre-ICS devices the device unlock password and the credential storage protection password are separate, so unlocking the device has no effect on credential storage state. Fortunately, Android provides a system activity that can unlock the key store. All we have to do is send an intent with the proper action to start the unlock activity. The action is however, slightly different on pre-Honeycomb and Honeycomb/ICS devices, so we need to check the Android version, before sending it:

try {
  if (Build.VERSION.SDK_INT < Build.VERSION_CODES.HONEYCOMB) {
    startActivity(new Intent("android.credentials.UNLOCK"));
  } else {
    startActivity(new Intent("com.android.credentials.UNLOCK"));
  }
} catch (ActivityNotFoundException e) {
    Log.e(TAG, "No UNLOCK activity: " + e.getMessage(), e);
}

Note that the unlock activity is using the transparent theme, so it will look like a dialog originating from your own activity. It is, however, managed by the system, so your app will be paused and resumed only after the unlock activity finishes. You need to handle this in your activity's code (you can't use startActivityForResult() though, since the unlock activity doesn't call setResult()). Additionally, if you don't have a device (or credential storage on pre-ICS devices) password set up, you will be prompted to set one. Control will be returned to your app only after you have set and confirmed an unlock password and initialized the credential storage.

Now that the keystore is unlocked, we can try to actually use it. As briefly mentioned above, it uses a local control socket for IPC, and the protocol is rather simple: a single letter command, followed by the length and value of any parameters (up to two). The protocol is already implemented in the android.security.KeyStore class, which is however hidden from non-system  applications. The reason for not exposing this API given in the JavaDoc comment is that 'it assumes that private and secret key bytes are available and would preclude the use of hardware crypto'. This is a very valid comment: in the current implementation keys are exported and imported as unencrypted blobs. If the keys were protected by a hardware device, the API would have to return some sort of an opaque key handle, since the actual key material would not be available, or would only be exportable if wrapped with another key. If the next Android version introduces hardware cryptography support, the API would have to change dramatically. Having said that, we want to use the keystore now, so we will ignore the warning and go ahead. Since the KeyStore is hidden we cannot import it directly, but we can call it using reflection. This is easy enough to do, but somewhat cumbersome. As the class doesn't really have any dependencies it is easier to copy it in our project, adding a few minor modifications to get it to compile (see sample code). Once this is done, we can list, add and get keys:

KeyStore ks = KeyStore.getInstance();
// get the names of all keys created by our app
String[] keyNames = ks.saw("");

// store a symmetric key in the keystore
SecretKey key = Crypto.generateKey();
boolean success = ks.put("secretKey1", key.getEncoded());
// check if operation succeeded and get error code if not
if (!success) {
   int errorCode = ks.getLastError();
   throw new RuntimeException("Keystore error: " + errorCode); 
}

// get a key from the keystore
byte[] keyBytes = ks.get("secretKey1");
SecretKey key = new SecretKeySpec(keyBytes, "AES");

// delete a key
boolean success = ks.delete("secretKey1");

As you can see from the code above, using the credential storage is pretty straightforward. You save keys by giving them a name (used as part of the file name the encrypted blobs are saved into), and then use that name to retrieve or delete them. The UID of the process that created the key is also a part of the file name, and thus key names only need to be unique within your application. One thing to note is that KeyStore methods that don't return a value (key name(s) or bytes), return a success flag, so you need to make sure you check it. In case of an error a more detailed error code can be obtained by calling getLastError(). All error codes are defined in the KeyStore class, but you are most likely to encounter PERMISSION_DENIED (if you try to call one of the methods reserved for the system user) or KEY_NOT_FOUND (if you try to access a non-existing key).

Check the sample project for a full app that generates an AES key, encrypts some data, then stores the key in the system credential storage and later retrieves it in order to decrypt the data. It generates and saves a new key each time you press 'Encrypt' and you can see the stored keys in the list view. Press the 'Reset' button to delete all keys created by the app. Note that the KeyStore class used is not compatible with the original Donut (Android 1.6) credential storage implementation, but it should work with all (public) subsequent versions. Here's how the app's screen looks like. Full code is, as usual, on github.


Besides keys you can store any sensitive information your app needs such as login passwords or tokens. Since decrypting the files on disk requires a key derived from the unlock password (or a dedicated password on pre-ICS devices), your secrets cannot be extracted even by apps with root access, or someone with physical access to the device (unless they know the password, of course). The master encryption key, however, is not tied to the device (like in iOS), so it is possible to copy the encrypted key files and perform a brute force attack on a different, more powerful machine(s).

You can experiment with other KeyStore API's, but most of those will result in a PERMISSION_DENIED when called from a non-system app. On ICS, there is also a public intent (action: com.android.credentials.RESET) that resets the credential storage, so you could prompt the user to clear it from your app, if necessary. Note that this will delete all stored data (keys, certificates, etc.), not just the ones your app created, so use with caution.

As a final warning, the code presented in this post does rely on private API's and OS implementation details, so it might break with the next Android version, or even not work on all current devices. Keep this in mind if you decide to use it in a production app.

Comments

Geoff Flarity said…
So from ICS forward, unlocking the phone also unlocks the keystore? Including if the user only uses slide to unlock? I really like the design of the keystore,thanks for explaining it.

Unknown said…
Yes and no. Unlocking the phone does unlock the keystore, but you can only use the keystore if have set a PIN or a password. You are required to set one the first time you try to use the keystore, for example when importing a PKCS12 file.
Unknown said…
See comments at Github issue, but looks like a threading problem.
Unknown said…
Have you had a look already on the keystore workings on Jelly Bean 4.2 (tablet version which supports multiple users) ?

In single user scenario's, the pincode of that user unlocks the system keystore. But what happens in multiple user-scenario's ?

Exploring the filesystem on a Nexus 7, I found still only one system keystore, not a-per-user keystore. So that would imply imho that only the first user (0) can unlock the keystore but additional users cannot.

This seems to be confirmed in the settings app, where the master-user has the ability to add certificates to the keystore, but this option is not available to secondary users.

Can you confirm this ?
Unknown said…
I haven't looked in the detail, but it does seem that there is only one keystore, available to the primary user only. This is also the case for secure element (SE) access. It's not a full multi-user system at this point (at probably won't be), only the primary user can do some things ('administrative tasks' if you will).
Unknown said…
It will flat out exit if you call it from another user. Cf. this from the CredentialStorage activity inside Settings:

if (UserHandle.myUserId() != UserHandle.USER_OWNER) {
Log.i(TAG, "Cannot install to CredentialStorage as non-primary user");
finish();
return;
}
Namphibian said…
Hi Nikolay

Thanks for some great articles.

I have an application in which I store really sensitive data. I want to encrypt this data which is the easy part. Since I will be using symmetrical encryption I have the usual headache of key management. Since the data that needs to be encrypted is acquired based on events that has no specific timing i.e. I have no idea when this data is going to arrive it would be extremely clumsy to ask a user for a password when the event occurs. One other limitation is that thus event can occur when there is no data connection so securely getting the key via some ssl connection is not viable. Essentially I have two options in dealing with this problem in my application.

1. Ask the user for the password on every event. This is going to piss the users off and they won't use the application. So not a option.

2. Cache the password somewhere secure. After booting their device I would ask for the password and cache it if it is correct. This seems to be the best way of doing this. However it is tricky. Thus I am trying to implement this.

It is this decision that led me here. I just need to make sure that I understand your blog post above. If I use the technique described above I will be opening a big fat can of compatibility problems? Since this not a official technique my mileage will vary on different devices?

Any other suggestions on how to cache the password? To be honest this is a bit of a problem with android as it makes it very difficult to achieve secure data in my scenario.

Any additional thoughts would be appreciated.

Regards
Neil
Unknown said…
Not sure what the 'password' you mention refers to. The keystore is protected by the device unlock password/PIN and your app will never see it. Keystore encryption keys are derived from this password/PIN and any encryption is automatically done by the system. Once the user unlocks the device the keystore will be automatically unlocked, you don't have to do anything.

As for compatibility, it should work reasonably well on ICS and later. Of course, this is an unofficial API, so it might break in future versions. Additionally, the next Android version is likely to offer an official API to this (there is some relevant code in 4.2, but it is not public).
Namphibian said…
Hi Nikolay

The password in this case would be the piece of secret that the user keeps in their heads. I understand the concept of keystroke unlock as the blog entry describes it very well.

Thanks for the info.
Neil
Unknown said…
So, you are trying/thinking about implementing something similar to the keystore for your app? You can use a similar approach: only ask for the password at startup, then derive any keys you need and store them in memory so you can use them when you need them.
Unknown said…
Nikolay,

I was able to successfully make this work on JB 4.1.1, but I get PROTOCOL_ERROR on 4.2.1. Did you get a chance to look into 4.2? Any pointers on where should I look?

Thanks,
Shah
Unknown said…
The protocol has been extended, but it's backward compatible. Get KeyStore.java from latest AOSP and replace the one in the sample project.
Unknown said…
This comment has been removed by the author.
Unknown said…
Thanks for quick response.

Seems like it's little more complicated than that. They abstracted keystore interactions to IKeyStoreService, and also depends on ServiceManager class to getService("android.security.keystore") ..

Dependencies to non-public APIs are more tighter now..

Any thoughts?
Unknown said…
frameworks/base/keystore/java/android/security/KeyStore.java is what you should be looking at.
Unknown said…
That's the one I'm looking at..
https://github.com/android/platform_frameworks_base/blob/master/keystore/java/android/security/KeyStore.java

Seems like Master version has these changes, I'm going to try with 4.2.1 tag..
Unknown said…
Yes, the Binder stuff is new. 4.2.2 still uses the old local socket version though. As I've said this is not a public API, and the Binder stuff is your warning :) In any case, some of it already has a hidden API in 4.2, maybe there will be an official API in the K version.
Bhavik Patel said…
Really useful . Thank You.
Unknown said…
Great work! I'm looking to use this to store password in my app surespot (https://www.surespot.me). I'm having a problem though, it seems that once "unlock" is executed and the "You need to set a screen lock PIN..." dialog is shown that there is no way to cancel out of the process. Pressing cancel just shows the dialog over and over. Is there anyway to know that the user pressed cancel in this dialog?
Unknown said…
I worked around the issue by using another transparent activity to launch the unlock activity. I store whether or not the unlock activity is going to launch (ie. the keystore needs to be unlocked) in the activity that's launching the new transparent activity. Then when it returns I store the data in the keystore.
androidblogger said…
Hi Nikolay Elenkov,

I want to know if the device is hardware backed. In such scenarios post calling keystore.get() - do we get actual key?
Or is there any way to get the private key from the device if it is hardware backed?
Unknown said…
You can't extract the key on hardware-backed devices. Read the post about Android 4.3 for details.
Unknown said…
Hi Nikolay ,
Great tutorial, I debugged your code on 2.3.3( emulator) am able to see keystore folder but I ran the same code on 4.3(emulator) , I couldn't able to see keystore folder, could please tell me where can I find it ?
Unknown said…
What happens with Keystore encryption keys if the user change the PIN? All Keystore encryption keys are deleted?

Unknown said…
It appears that this is new behaviour, added in 4.4. Prior versions do not clear the keys.
Unknown said…
Looks like it is fixed in the final 5.0 build (andoroid_5.0.0_r1 tag)
Unknown said…
Thanks for informing about Android. We are giving more updates with software free download for windows 64 bit
Unknown said…
Thanks for this wonderful article and your code as well. My question is, is it possible that an app still stores its data in the device's keystore, even when user hasn't protected his/her devices with a pin or password? Many users don't have protected devices. Some use face recognition or pattern to unlock them. Does keystore on Android devices always require a pin or password?
DarkestGirl said…
thank you for the help, i really want to understand that if i am using pbe to derive key from the password and then encrypt the file. how do i share that encrypted file with some other person? do the other person needs to have the same password? please suggest how to share the files with other people securely. what is pkcs-7 and how it can be used to share the files.please help
Unknown said…
Yes, if you want to share an encrypted file, the recipient needs to know the password to decrypt it. You need to send them the password out of band (in person, etc.) for this to be reasonably secure.
DarkestGirl said…
how should i share file then?
Unknown said…
Hello Nikolay,
Any guess to this problem?
http://stackoverflow.com/questions/30234191/android-prevent-app-keystore-deletion-when-clear-credentials
Thanks!
Alex Suzuki said…
Hi there, I have an app that uses your mechanism, and I would like to transition away from the private API. As you mention, Android 4.3+ has the KeyChain API, which allows for securely storing a private/public keypair.
Wouldn't it be best to use the Public Key to encrypt the secret and then store the encrypted version in "less secure" storage such as SharedPreferences, and decrypt it with the Private Key (which would be hardware-protected) when using it to e.g. auto-login a user?
Unknown said…
This comment has been removed by the author.
Unknown said…
Hello Nikolay,
I've stumbled upon this fairly recent article by Godfrey Nolan published on March 31, 2015:
http://www.androidauthority.com/where-is-the-best-place-to-store-a-password-in-your-android-app-597197/

which discusses also Android KeyStore and list's it as a non-secure option of storing sensitive data, because:
"The private key is stored in a file that has [app_id]_USRCERT_[key_alias]. On a rooted phone you can copy the file to another [app_id_malicious]_USRCERT_[key_alias] and then import it from your malicious app, allowing you to recover the password."

But here, in your article, you mention:
"Since decrypting the files on disk requires a key derived from the unlock password (or a dedicated password on pre-ICS devices), your secrets cannot be extracted even by apps with root access, or someone with physical access to the device (unless they know the password, of course)."
so the only way is brute force it.
Could you please elaborate on this one? Is this "extraction" possible?
Vajda Lubos said…
Hello Nikolay,
There are some other functions in IKeystoreService interface which weren't mentioned in Keystore daemon commands table (and also not used in example).
For example "generate","sign", "verify",...

I tried to call function "generate" from my apps to create new asymetric key pair(i hope :-), but return value is 6 (PERMISSION_DENIED).

I would like to know their permissions.

Thank you

Lubos Vajda

Popular posts from this blog

Decrypting Android M adopted storage

Unpacking Android backups

Using app encryption in Jelly Bean