Hi Jeremy,
Many thanks for bringing up this topic and the share your experience to improve TF-M security and increase user awareness of certain limitations. The patches, mentioned by Sherry, intended to fix the issue at least partially. Welcome to
join the project and comment/contribute to the common codebase. That will be very appreciated.
In addition to wear levelling, a device lifecycle can by moved to the decommissioned state when storage integrity problem detected. This needs more thought, and the best solution is use case dependent, probably. In some cases, a flash performance
and footprint might be more preferred than crypto algorithm selection (without compromising security, of cause).
Assume you aware about the
Tech Forum which is a good place to discuss such topics online.
Thanks and best regards,
Anton
From: Sherry Zhang via TF-M <tf-m@lists.trustedfirmware.org>
Sent: Thursday, September 22, 2022 5:51 AM
To: Jeremy Herbert <jeremy.006@gmail.com>
Cc: tf-m@lists.trustedfirmware.org; nd <nd@arm.com>
Subject: [TF-M] Re: Failure mode of protected storage with faulty NV counters?
Hi Jeremy,
a) the flash wearout
After
this fix, the flash wear out should be detected when it happens. Later one possible enhancement we can do is adding support of wear leveling from my rough thought. As for how to select the devices to make the PS service endurance more reasonable, it is
up to users.
b) if you disable rollback protection nonce on the internal flash,…
In the PS integration guide, we can add the description that in case of GCM(the default configuration) rollback protection is essential. And in the code, we can add that runtime
check to ensure that rollback protection is enabled.
- I haven't registered, I will try to do it soon and let you know.
If there is any problem in the process, do not hesitate to contact us.
Regards,
Sherry Zhang
Hi Sherry,
It is good enough in theory, but the problem is a) the flash wearout and b) if you disable rollback protection nonce on the internal flash, then you don't just lose rollback protection but you *also* lose the encryption on the external
device because it is trivially vulnerable to nonce reuse.
Hi Jeremy,
- is it correct that if replay protection is disabled, an attacker can force the nonce used as the IV for encryption to be reused by rolling back the external flash image?
Do you think adding NV counter on a non-volatile memory to prevent rollback is not good enough to solve this problem?
Regards,
Sherry Zhang
Hi Sherry,
I haven't registered, I will try to do it soon and let you know.
I guess my issue is more around the AES-GCM encryption for external flash - is it correct that if replay protection is disabled, an attacker can force the nonce used as the IV for
encryption to be reused by rolling back the external flash image? If so, it seems like the AES-GCM encryption for PS is broken by design as you must not under any circumstances allow nonce reuse with the same key.
Hi Jeremy,
Have you registered access to the Gerrit? I can add you as a reviewer of this patch if possible so that you can comment on this patch directly. Your description is very detailed.
Thanks for that.
For nonce reuse issue, I think what is missed is that the template implementation of nv counter layer should check whether the NV counter is programmed successfully. There is a
patch for fixing that. Thanks for pointing this out. By the way, this is a template implementation. Users can implement their own NV counter APIs defined in tfm_plat_nv_counters.h setting
PLATFORM_DEFAULT_NV_COUNTERS OFF.
(Removed part of the initial email thread as it reached the 80 KB maximum size.)
Thanks!
Regards,
Sherry Zhang
Can I suggest that the following be added instead? I'm not sure that is clear enough to explain the pitfalls to non-experts like myself.
"If this flag is enabled, the lifecycle of the PS service depends on the minimum write endurance of both the device that stores the assets and the device that stores the NV counters
- typically the NV counters can only be stored to on-die storage (such as internal flash) for security reasons. As an example, if the NV counter is stored inside the internal flash of a microcontroller with a write endurance of 10k writes, and the PS assets
are stored in an external flash with a write endurance of 100k writes, the useful number of writes to the external flash is constrained by the endurance of the internal flash, after which point the rollback protection will fail as the internal flash can no
longer correctly store the NV counter."
Also, this doesn't mention that the AES-GCM encryption can be trivially broken if the rollback protection is not enabled. To quote SP 800-38D section 8:
The probability that the authenticated encryption function ever will be invoked with the same IV and the same key on two (or more) distinct sets of input data shall be no greater
than 2-32. Compliance with this requirement is crucial to the security of GCM. Across all instances of the authenticated encryption function with a given key, if even one IV is ever repeated, then the implementation may be vulnerable to the forgery attacks
that are described in Ref [5] and summarized in Appendix A. In practice, this requirement is almost as important as the secrecy of the key.
I understand it may not be up to you and that there are already silicon vendors with shipped products, but if the scope of TF-M is advertised to end developers as taking care of
security from the beginning, it does seem like it should probably be considered in-scope that the PS encryption is not trivially defeated using the default implementation.
Hi Jeremy,
I totally agree with your analysis on the necessity of NV counter in ps service. I think another reason why rollback protection is necessary is to prevent the ps data itself rollback.
For example, if the ps asset with a specific pair of (uid, client_id) has been once created, then later the asset is updated to another value. Without rollback protection, the old data may be recovered and used by an attacker.
So, we need a trusted non-volatile memory to store the NV counter. As for which memory device should be used(like FRAM or other flash devices), it is beyond our scope. Users can
make the decision based on their own specific requirements.
For your option 3, if the algorithm is compatible with psa crypto interface, users can configure the ps algorithm via
PS_CRYPTO_AEAD_ALG build flag with that algorithm. Even nonce reused protection is not needed, I think rollback protection is still necessary.
I created
this patch to remind the users that the lifecycle of PS service may also depends on the device that stores NV counter.
Regards,
Sherry Zhang
Hi Sherry,
I have actually come up against this problem a bunch over the past few years. Here are my thoughts.
It seems like there are two major problems:
1. Nonce reuse in AES-GCM breaks the encryption to the point of it being near useless
2. A random nonce is not recommended for AES-GCM as the nonce size is a bit small, but having a secure non-volatile counter on an MCU requires specific hardware support as far as
having high endurance non-volatile memory (see ATECC608 for example) - and basically nobody has this in their MCUs (though I know STM32s sometimes come with internal EEPROM which can take ~100k writes per word). Secure NFC tags like the DESFire EV3 are often
rated for 1million+ writes per word, and with wear leveling you can basically make this number beyond the useful life of the device.
This results in it being impossible to implement external encrypted flash with rollback protection if the device ever loses power or is reset (or at least as far as I can see):
1. If you don't use a non-volatile counter, resetting the device will cause nonce reuse => AES-GCM broken.
2. If you do use a non-volatile counter stored on internal flash, it seems that eventually the area of flash will fail to all bits cleared (ie 0) - given typical numbers, you are
looking at a maximum of 50k erases *per page*, not per byte (though most vendors guarantee way less than this ie nordic is 10k). And given in TF-M all of the different counters are probably stored in the same page, that means that writing to any counter uses
up one of your cycles. You could potentially wear level across pages, but with 1K/4K page sizes that is a lot of flash to consume just for a counter.
3. If you store the nonce for AES-GCM in external flash, and then read it back and increment for the next operation, the attacker can just roll back the flash => nonce reuse =>
AES-GCM broken.
Also it appears that you *must* always enable rollback protection in TF-M, otherwise you just need to dump the external flash, reset the device so it does another write, dump the
flash again and you have caused nonce reuse => encryption is broken.
The solutions I can propose:
1. I realise this is probably unrealistic, but it is the best solution nonetheless: require silicon vendors to add a small amount of some special non-volatile memory on their chip
like FRAM (maybe 64 bits?) to get a "Security Lvl. 99 (TM): Extreme" certification that they can put on the marketing pages for their device
2. Use an external device like the ATECC608 which has an encrypted, authenticated link and high endurance non-volatile storage for counters (except normal people like me have a
2+ year lead time on this and similar parts)
3. Don't use AES-GCM. There is a much newer, nonce-reuse resistant variant of AES-GCM called AES-GCM-SIV which has only a small performance penalty (I have seen ~30% quoted). I
had planned to do it a while ago, but I have just spent the last few days or so hacking this cipher into mbedtls, and it isn't too far from AES-GCM in terms of functionality: https://github.com/Mbed-TLS/mbedtls/pull/6294
- I hope it can eventually be merged once I tidy it up. While this doesn't solve the problem of the rollback counter failing due to being stored in internal flash (you need option 1 or 2 for that), if the counter fails to a constant, at least you just lose
the rollback protection rather than losing all security.
If you have any other suggestions as to how I can solve this problem or something that I am missing, I am all ears!
Hi Jeremy,
About which flash should be the limitation of the PS service usage, this is really a good perspective to assess the storage service. I thought over it again and I think it is hard
to completely separate the PS away from the internal flash(as we need a trusted area for implementing the rollback protection). Also, the NV counter area is not only used by PS but also by other components like BL2. MCUboot writes the BL2 NV counter once each
time booting up. That can also make the NV area flash wear out. Do you have any suggestions/proposal on making PS storage only lives on the external flash?
I checked with someone who is working on the CMSIS drivers and got confirmation that the CMSIS Flash driver API is intended to only check the status of the operation without verification
of the written data. But I wonder if the write error can be detected by the flash device itself with ECC check internally when flash wear out happen or flash data corrupt happen. It maybe depend on the specific flash device. So, I propose to add the check
in the NV counter layer and leave that check as an optional choice. User can decide whether enable that check or not based on their specific flash device. How do you think about?
About your proposal of adding the warnings about endurance, this is a good suggestion. We will add that later. Thanks for the feedback!
The nonce value is stored into flash together with the ps object table. After reboot, it is read out from flash and the read out value is used as the start value after the reboot(not
starts from 0 each time reboot). See code at
here.
Regards,
Sherry Zhang
Hi Sherry,
Thanks for your reply. (I also received an off-list reply from Nordic about this - thank you Sebastian). I have not yet encountered a write endurance failure, but I am developing a device which will be deployed for a long time. Using the nrf53 example, writing
encrypted sensor data to external EEPROM/FRAM/MRAM once per hour would cause the device to fail in