On 14/09/2024 04:03, Mbed TLS via mbed-tls wrote:
My mbedtls client has been working for 2 years. It did what I required and has been stable.

However, I now need to force a new server to use my preferred cipher suite.

I found the helper function to force the cipher suite here:

https://github.com/Mbed-TLS/mbedtls/blob/de4d5b78558666d2e258d95e6c5875f9c72687ed/tests/src/test_helpers/ssl_helpers.c#L1039

I added mbedtls_ssl_conf_preference_order(conf, MBEDTLS_SSL_SRV_CIPHERSUITE_ORDER_CLIENT) to the end of the function to force the server to choose my ciphersuite.

Note that this is a server-side function, it has no effect on clients. The way the TLS protocol works is, the client sends a list of cipher suites that it supports, then the server picks one of them. With Mbed TLS (and I think that's typical of TLS server implementations), there are two ways this can go:
  1. The server goes through its list of cipher suites in order of preference, and picks the first one that the client supports. (ORDER_SERVER)
  2. The server goes through the list of cipher suites sent by the client, and picks the first one that the server supports. (ORDER_CLIENT)

If you want to force a server to use your preferred cipher suite, don't send other cipher suites. Call mbedtls_ssl_conf_ciphersuites with a list that only includes cipher suites that you're happy with.

(…)

However, mbedtls_ssl_handshake returns with value -26112, which I have looked up to be MBEDTLS_ERR_SSL_ILLEGAL_PARAMETER.

This error indicates that your client received something that it thinks is not valid according to the TLS protocol. So either the server is sending something wrong, or it's hitting some limitation in the client.

To find out more, make sure you compile Mbed TLS with MBEDTLS_DEBUG_C enabled, call mbedtls_ssl_conf_dbg() to set up a debug callback (typically printing its arguments, see examples in e.g. ssl_client1.c), and call mbedtls_debug_set_threshold(4). You'll get a detailed trace that tells you exactly where the error was detected and what happened before.

Best regards,

--
Gilles Peskine
Mbed TLS developer