Hi,

Otherwise we could update to 2.28.1 (and spend time doing that, plus back-editing various patches) and find that it doesn't fix it.

Absolutely, but that wasn't the reason why I was recommending you use a maintained version. There are mainly two reasons.

Firstly, there have been 43 entries in the Security section of ChangeLog between the release you're using and 2.28.1. Are you confident none of those vulnerabilities affect you? Have you reviewed those entries against your use case & threat model ? If you haven't, perhaps an attacker has...

Secondly, what about the next vulnerabilities that are going to be disclosed as we fix them in future 2.28.x? If one of them impacts your code in a severe way, you'll have to do an emergency upgrade from 2.16.2 to 2.28.x, which as you point out can be non-trivial work, while upgrading from 2.28.x to 2.28.x+1 is usually much less work.

Updating to v3 is reportedly a LOT of work.

________________________________ From: Peter peter@peter2000.co.uk Sent: 07 December 2022 12:57 To: Manuel Pegourie-Gonnard Manuel.Pegourie-Gonnard@arm.com Cc: mbed-tls@lists.trustedfirmware.org mbed-tls@lists.trustedfirmware.org Subject: Re: [mbed-tls] MbedTLS crashes with some websites in mbedtls_mpi_free()

Hello Manuel,

I am hoping somebody may recognise the issue.

Otherwise we could update to 2.28.1 (and spend time doing that, plus back-editing various patches) and find that it doesn't fix it.

Updating to v3 is reportedly a LOT of work.

This is almost certainly not a security vulnerability.

Apart from this issue, the MbedTLS code has been working solidly for a couple of years.

Interestingly it happens only on the ST port of MbedTLS, not on a Windows compilation of MbedTLS.

Regards,

Peter

Hi Peter,

Mbed TLS v2.16.2 is no longer maintained. I would strongly recommend you, and in fact everyone, to use a maintained version of Mbed TLS. Currently the two maintained versions are 2.28.1 (2.28.x is the current LTS branch) and 3.2.1 (3.x is where new features arrive). Versions that are no longer maintained have publicly known security vulnerabilities.

(I didn't analyze the specific problem you're facing so I can't say if this is something that has been fixed in the meantime.)

Best regards, Manuel.

---

From: Peter via mbed-tls mbed-tls@lists.trustedfirmware.org Sent: 06 December 2022 17:25 To: mbed-tls@lists.trustedfirmware.org mbed-tls@lists.trustedfirmware.org Subject: [mbed-tls] MbedTLS crashes with some websites in mbedtls_mpi_free()

Hi All,

Some websites crash MbedTLS, in

void mbedtls_ecp_point_free( mbedtls_ecp_point *pt ) { if( pt == NULL ) return;

mbedtls_mpi_free( &( pt->X ) ); mbedtls_mpi_free( &( pt->Y ) ); mbedtls_mpi_free( &( pt->Z ) ); }

It crashes in the last function call of the three above.

   Thread #1 [main] 1 [core: 0] (Suspended : Signal :

SIGTRAP:Trace/breakpoint trap) HardFault_Handler() at stm32f4xx_it.c:103 0x8053c22 <signal handler called>() at 0xffffffed 0x3810180 mbedtls_ecp_point_free() at ecp.c:594 0x8029c7a 0x81030100

It crashes properly i.e. invalid opcodes etc.

There is a fair bit on google with others having gone down the same road, never resolved.

One site which does it is socata.org but quite a few others do it. Probably 10% of "major name" websites cause it to crash.

It may have been undiscovered for a long time, or ever, because most MbedTLS clients are connecting to specific private servers only. People aren't using it to connect to microsoft.com (which doesn't actually crash but the handshake returns 0x2700 which can be investigated).

There is a Windows build of MbedTLS (v2.16.2) which seems to work in the crashing cases. Stepping through this in Cube IDE (32F417) it looks like the crash is due to a buffer being filled with random numbers. The guy working on this is contactable only on Monday afternoons so I am a bit screwed :) He said he found something about that buffer, or maybe a corrupted function pointer.

But as I said this issue has come up before according to numerous online searches and maybe someone can recognise it.

Hardware AES is enabled but this occurs with software AES also. No other 32F417 hardware crypto features are used, other than its random number generator.

TLS is v2.16.2.

Thank you in advance for any pointers.

Peter

Hello,

I didn't want to divert this topic onto what must be a well worn area :) but all IOT / embedded stuff will have vulnerabilities, for ever, which is why no IOT product should ever be on an open port. The industry is gradually realising this, which is why they are putting IOT behind private servers, which the IOT device accesses as a client. That is also a revenue opportunity because you can charge say $2/month for that server :) That, in turn, has the potential to give IOT a really bad name, as these servers cease to be maintained :)

I "run" several servers and they are hit (from China and Russia mostly) at 1Hz to 10Hz :) This is OK with Centos/Linux and Apache/Nginx, etc, but these have had a thousand man-years put into them. Microsoft is a 100BN $ company and they are still patching Windows after 30 years, so the chances of MbedTLS etc not having back doors is absolutely zero. It will have back doors now and will have back doors for ever. And probably quite obvious ones.

And even if you know of them, how does one patch them, in the field? Doing that is a minefield. You can't risk doing it automatically; that could destroy your company if it goes wrong (products getting bricked). If the product has no UI (LCD, keypad, etc) then how does one control it? There needs to be security around any form of OTA which needs certificate admin, preferably secure key storage in the target too, and this needs to be maintained for ever.

So the box I am working on has

- http server (for local config only) which I wrote myself - http client (for stuff like "I am alive" pinging) - https client (for accessing specific servers only)

The last one uses MbedTLS. I am working on the assumption that it WILL have back doors :) But the user manual will tell customers to never put it on an open port, and the http server (for config etc) is to be accessed via a VPN only.

Our MbedTLS has been patched with this https://github.com/mongoose-os/mbedtls to enable it to process a ~200k root certificate chain without needing 200k of RAM, which very few "embedded" systems will have spare. MbedTLS as it comes is really capable of processing just one server certificate. This patch works but took a lot of work to put in, would need to be re-applied to any MbedTLS update, and it may even be the cause of the crashing problem.

I will get 2.28 looked at (not doing this side of it myself) but I doubt this issue is going to go away because the Windows port of our existing one works ok.

Regards,

Peter

Hi,

...

Otherwise we could update to 2.28.1 (and spend time doing that, plus back-editing various patches) and find that it doesn't fix it.

Absolutely, but that wasn't the reason why I was recommending you use a maintained version. There are mainly two reasons.

Firstly, there have been 43 entries in the Security section of ChangeLog between the release you're using and 2.28.1. Are you confident none of those vulnerabilities affect you? Have you reviewed those entries against your use case & threat model ? If you haven't, perhaps an attacker has...

Secondly, what about the next vulnerabilities that are going to be disclosed as we fix them in future 2.28.x? If one of them impacts your code in a severe way, you'll have to do an emergency upgrade from 2.16.2 to 2.28.x, which as you point out can be non-trivial work, while upgrading from 2.28.x to 2.28.x+1 is usually much less work.

...

Updating to v3 is reportedly a LOT of work.

---

From: Peter peter@peter2000.co.uk Sent: 07 December 2022 12:57 To: Manuel Pegourie-Gonnard Manuel.Pegourie-Gonnard@arm.com Cc: mbed-tls@lists.trustedfirmware.org mbed-tls@lists.trustedfirmware.org Subject: Re: [mbed-tls] MbedTLS crashes with some websites in mbedtls_mpi_free()

Hello Manuel,

I am hoping somebody may recognise the issue.

Otherwise we could update to 2.28.1 (and spend time doing that, plus back-editing various patches) and find that it doesn't fix it.

Updating to v3 is reportedly a LOT of work.

This is almost certainly not a security vulnerability.

Apart from this issue, the MbedTLS code has been working solidly for a couple of years.

Interestingly it happens only on the ST port of MbedTLS, not on a Windows compilation of MbedTLS.

Regards,

Peter

...

Hi Peter,

Mbed TLS v2.16.2 is no longer maintained. I would strongly recommend you, and in fact everyone, to use a maintained version of Mbed TLS. Currently the two maintained versions are 2.28.1 (2.28.x is the current LTS branch) and 3.2.1 (3.x is where new features arrive). Versions that are no longer maintained have publicly known security vulnerabilities.

(I didn't analyze the specific problem you're facing so I can't say if this is something that has been fixed in the meantime.)

Best regards, Manuel.

---

From: Peter via mbed-tls mbed-tls@lists.trustedfirmware.org Sent: 06 December 2022 17:25 To: mbed-tls@lists.trustedfirmware.org mbed-tls@lists.trustedfirmware.org Subject: [mbed-tls] MbedTLS crashes with some websites in mbedtls_mpi_free()

Hi All,

Some websites crash MbedTLS, in

void mbedtls_ecp_point_free( mbedtls_ecp_point *pt ) { if( pt == NULL ) return;

mbedtls_mpi_free( &( pt->X ) ); mbedtls_mpi_free( &( pt->Y ) ); mbedtls_mpi_free( &( pt->Z ) ); }

It crashes in the last function call of the three above.

   Thread #1 [main] 1 [core: 0] (Suspended : Signal :

SIGTRAP:Trace/breakpoint trap) HardFault_Handler() at stm32f4xx_it.c:103 0x8053c22 <signal handler called>() at 0xffffffed 0x3810180 mbedtls_ecp_point_free() at ecp.c:594 0x8029c7a 0x81030100

It crashes properly i.e. invalid opcodes etc.

There is a fair bit on google with others having gone down the same road, never resolved.

One site which does it is socata.org but quite a few others do it. Probably 10% of "major name" websites cause it to crash.

It may have been undiscovered for a long time, or ever, because most MbedTLS clients are connecting to specific private servers only. People aren't using it to connect to microsoft.com (which doesn't actually crash but the handshake returns 0x2700 which can be investigated).

There is a Windows build of MbedTLS (v2.16.2) which seems to work in the crashing cases. Stepping through this in Cube IDE (32F417) it looks like the crash is due to a buffer being filled with random numbers. The guy working on this is contactable only on Monday afternoons so I am a bit screwed :) He said he found something about that buffer, or maybe a corrupted function pointer.

But as I said this issue has come up before according to numerous online searches and maybe someone can recognise it.

Hardware AES is enabled but this occurs with software AES also. No other 32F417 hardware crypto features are used, other than its random number generator.

TLS is v2.16.2.

Thank you in advance for any pointers.

Peter

Hello Gilles,

Sure, this could be. I was hoping somebody might recognise this specific issue.

We did try to give TLS another 10k for its private heap block; it made no difference. But also there were no errors.

Kind regards,

Peter

Hi Peter,

Crashes in free() are a very common symptom of heap corruption, due to a previous buffer overflow or use-after-free or double free(). Often the bug corrupting the heap happened earlier, and the crash is just where the corruption starts to become noticeable. This makes debugging difficult.

If you can, compile your code with instrumentation such as ASan. This makes it far more likely that the corruption will be detected as soon as it happens, so debugging is easier.

You need to look at the code of your whole application. It's quite possible that the problem is not due to Mbed TLS itself, but to some other part that's corrupting the heap.

Best regards,

-- Gilles Peskine Mbed TLS developer

On 06/12/2022 17:25, Peter via mbed-tls wrote:

...

Hi All,

Some websites crash MbedTLS, in

void mbedtls_ecp_point_free( mbedtls_ecp_point *pt ) { if( pt == NULL ) return;

 mbedtls_mpi_free( &( pt->X ) );
 mbedtls_mpi_free( &( pt->Y ) );
 mbedtls_mpi_free( &( pt->Z ) );

}

It crashes in the last function call of the three above.

Thread #1 [main] 1 [core: 0] (Suspended : Signal : SIGTRAP:Trace/breakpoint trap) HardFault_Handler() at stm32f4xx_it.c:103 0x8053c22 <signal handler called>() at 0xffffffed 0x3810180 mbedtls_ecp_point_free() at ecp.c:594 0x8029c7a 0x81030100

It crashes properly i.e. invalid opcodes etc.

There is a fair bit on google with others having gone down the same road, never resolved.

One site which does it is socata.org but quite a few others do it. Probably 10% of "major name" websites cause it to crash.

It may have been undiscovered for a long time, or ever, because most MbedTLS clients are connecting to specific private servers only. People aren't using it to connect to microsoft.com (which doesn't actually crash but the handshake returns 0x2700 which can be investigated).

There is a Windows build of MbedTLS (v2.16.2) which seems to work in the crashing cases. Stepping through this in Cube IDE (32F417) it looks like the crash is due to a buffer being filled with random numbers. The guy working on this is contactable only on Monday afternoons so I am a bit screwed :) He said he found something about that buffer, or maybe a corrupted function pointer.

But as I said this issue has come up before according to numerous online searches and maybe someone can recognise it.

Hardware AES is enabled but this occurs with software AES also. No other 32F417 hardware crypto features are used, other than its random number generator.

TLS is v2.16.2.

Thank you in advance for any pointers.

Peter

Hello Gilles,

Fixed.

It was the setting

MBEDTLS_ECP_MAX_BITS set to 521.

Was 384, for some reason.

Kind regards,

Peter

Hi Peter,

Crashes in free() are a very common symptom of heap corruption, due to a previous buffer overflow or use-after-free or double free(). Often the bug corrupting the heap happened earlier, and the crash is just where the corruption starts to become noticeable. This makes debugging difficult.

If you can, compile your code with instrumentation such as ASan. This makes it far more likely that the corruption will be detected as soon as it happens, so debugging is easier.

You need to look at the code of your whole application. It's quite possible that the problem is not due to Mbed TLS itself, but to some other part that's corrupting the heap.

Best regards,

-- Gilles Peskine Mbed TLS developer

On 06/12/2022 17:25, Peter via mbed-tls wrote:

...

Hi All,

Some websites crash MbedTLS, in

void mbedtls_ecp_point_free( mbedtls_ecp_point *pt ) { if( pt == NULL ) return;

 mbedtls_mpi_free( &( pt->X ) );
 mbedtls_mpi_free( &( pt->Y ) );
 mbedtls_mpi_free( &( pt->Z ) );

}

It crashes in the last function call of the three above.

Thread #1 [main] 1 [core: 0] (Suspended : Signal : SIGTRAP:Trace/breakpoint trap) HardFault_Handler() at stm32f4xx_it.c:103 0x8053c22 <signal handler called>() at 0xffffffed 0x3810180 mbedtls_ecp_point_free() at ecp.c:594 0x8029c7a 0x81030100

It crashes properly i.e. invalid opcodes etc.

There is a fair bit on google with others having gone down the same road, never resolved.

One site which does it is socata.org but quite a few others do it. Probably 10% of "major name" websites cause it to crash.

It may have been undiscovered for a long time, or ever, because most MbedTLS clients are connecting to specific private servers only. People aren't using it to connect to microsoft.com (which doesn't actually crash but the handshake returns 0x2700 which can be investigated).

There is a Windows build of MbedTLS (v2.16.2) which seems to work in the crashing cases. Stepping through this in Cube IDE (32F417) it looks like the crash is due to a buffer being filled with random numbers. The guy working on this is contactable only on Monday afternoons so I am a bit screwed :) He said he found something about that buffer, or maybe a corrupted function pointer.

But as I said this issue has come up before according to numerous online searches and maybe someone can recognise it.

Hardware AES is enabled but this occurs with software AES also. No other 32F417 hardware crypto features are used, other than its random number generator.

TLS is v2.16.2.

Thank you in advance for any pointers.

Peter