Hi guys,

I have a mbetls based server client application (DTLS) written in Apple Swift language, wrapping the mbedtls C api, running just fine for many years now on iOS, Android, and Linux. We have been compiling the code with Swift 5.0, 5.1, 5.2, 5.3, 5.4 in both release and debug mode.

Recently I just recompiled my code with Swift 5.5 in release mode, and my handshakes started failing on the Linux server with cryptic mbedtls_pk_sign() returned -17280 (-0x4380). When I take the same code and recompile in debug mode (-Onone) it works as expected. The same code compiled with Swift+C for iOS and macOS still works in both debug and release mode on both client/server.

If I understand the compilation process correctly, the Swift Package Manager that we use to wrap mbedtls simply uses CMakeLists.txt to compile mbedtls and pick up the right flags.

I am attaching excerpt from the log where the server part of the handshake fails (in release mode) and succeeds (in debug mode).

I am certain this is probably caused by my code and how I wrap C mbedtls sources in Swift, but I can not yet rule out a bug in Swift/C compiler part for Linux which from time to time does have bugs related to code optimization.

Could any good soul who understands the parts of the code below point me in the direction where to look and why would the mbedtls_pk_sign fail this way? What parts of the SSL context does it use? I will deep dive into the code to see whether this can be caused by memory corruption or anything else.

It’s on LTS mbedtls-2.16.11.

Could I run some tests to perhaps verify that mbedtls is compiled and working correctly?

thanks for any help, Martin

— failing handshake — Dec 7 15:54:53 ssl_srv.c:3296 => write server key exchange Dec 7 15:54:53 ssl_srv.c:3074 ECDHE curve: secp521r1 Dec 7 15:54:53 ssl_srv.c:3100 value of 'ECDH: Q(X)' (521 bits) is: Dec 7 15:54:53 ssl_srv.c:3100 01 d0 5f 4a fa 12 f8 46 75 fb 3e 38 db ba 88 ff Dec 7 15:54:53 ssl_srv.c:3100 73 72 79 78 21 dd 9f ac ac 02 db e7 7e 91 6e 5c Dec 7 15:54:53 ssl_srv.c:3100 06 ed 77 19 42 f1 11 6a f5 75 04 20 d8 51 50 a5 Dec 7 15:54:53 ssl_srv.c:3100 8f 5a 3d 8c b3 c0 15 df cc d7 5b e5 eb 04 2e b6 Dec 7 15:54:53 ssl_srv.c:3100 2f e7 Dec 7 15:54:53 ssl_srv.c:3100 value of 'ECDH: Q(Y)' (518 bits) is: Dec 7 15:54:53 ssl_srv.c:3100 26 c1 52 f0 b6 20 4c aa 53 7c aa 27 12 5f 72 7f Dec 7 15:54:53 ssl_srv.c:3100 11 ea a9 34 9a f5 ef 63 28 76 d8 52 4e 31 a3 13 Dec 7 15:54:53 ssl_srv.c:3100 52 a5 c9 52 5f 6b 41 b6 7e 47 0e 2a 68 c5 e5 80 Dec 7 15:54:53 ssl_srv.c:3100 14 27 ad aa 96 5b 89 41 55 5b d5 52 61 66 8c 06 Dec 7 15:54:53 ssl_srv.c:3100 4b Dec 7 15:54:53 ssl_srv.c:3163 pick hash algorithm 8 for signing Dec 7 15:54:53 ssl_srv.c:3201 dumping 'parameters hash' (64 bytes) Dec 7 15:54:53 ssl_srv.c:3201 0000: 55 c8 d2 5c 83 d8 6b e3 9f 9c b2 f4 e7 f7 4d 26 U....k.......M& Dec 7 15:54:53 ssl_srv.c:3201 0010: b4 ce c9 df c2 ae 7a 60 cb e8 02 13 d5 6d d8 ec ......z`.....m.. Dec 7 15:54:53 ssl_srv.c:3201 0020: 48 72 77 9f e2 69 4e f5 b4 1d db 9a 75 2a 5e c7 Hrw..iN.....u*^. Dec 7 15:54:53 ssl_srv.c:3201 0030: e1 1b 66 fc a6 ff 37 c6 6d 42 9b b3 cf 63 bb 0e ..f...7.mB...c.. Dec 7 15:54:53 ssl_srv.c:3274 mbedtls_pk_sign() returned -17280 (-0x4380) Dec 7 15:54:53 ssl_tls.c:8219 <= handshake

— successful handshake — Dec 7 16:13:52 ssl_srv.c:3296 => write server key exchange Dec 7 16:13:52 ssl_srv.c:3074 ECDHE curve: secp521r1 Dec 7 16:13:52 ssl_srv.c:3100 value of 'ECDH: Q(X)' (519 bits) is: Dec 7 16:13:52 ssl_srv.c:3100 52 5f 4e 78 48 bd be 6a e4 98 21 30 ce 1c ba 08 Dec 7 16:13:52 ssl_srv.c:3100 b7 ae 88 d3 ff c9 c4 a5 8c 13 1e 73 5b 1f 08 60 Dec 7 16:13:52 ssl_srv.c:3100 7c 51 a7 ec 91 54 d7 52 f3 55 6a 34 92 cf 92 e8 Dec 7 16:13:52 ssl_srv.c:3100 c8 39 33 5e d3 46 3d 89 e7 6d 8f 41 d9 e0 67 e6 Dec 7 16:13:52 ssl_srv.c:3100 7c Dec 7 16:13:52 ssl_srv.c:3100 value of 'ECDH: Q(Y)' (520 bits) is: Dec 7 16:13:52 ssl_srv.c:3100 87 48 29 20 93 8e f8 9a a9 54 70 26 58 79 9d 67 Dec 7 16:13:52 ssl_srv.c:3100 8c ec 0f 06 ae 73 13 40 72 58 fe c1 6e 14 b4 48 Dec 7 16:13:52 ssl_srv.c:3100 55 a1 27 74 9d 63 36 aa db 80 29 f8 ab 9c 64 ba Dec 7 16:13:52 ssl_srv.c:3100 f9 0a 19 af 9a 65 6f 02 96 c6 53 4d 49 4f b4 f5 Dec 7 16:13:52 ssl_srv.c:3100 46 Dec 7 16:13:52 ssl_srv.c:3163 pick hash algorithm 8 for signing Dec 7 16:13:52 ssl_srv.c:3201 dumping 'parameters hash' (64 bytes) Dec 7 16:13:52 ssl_srv.c:3201 0000: e1 65 f4 f0 a6 a3 67 7d 32 55 ed 2a 13 08 71 8a .e....g}2U.*..q. Dec 7 16:13:52 ssl_srv.c:3201 0010: 7e 13 f6 30 85 ed c3 c9 10 e8 b2 43 75 bc 6c 24 ~..0.......Cu.l$ Dec 7 16:13:52 ssl_srv.c:3201 0020: 8f fb d8 90 cb 53 7f 13 53 13 1b f4 eb bd 38 5c .....S..S.....8\ Dec 7 16:13:52 ssl_srv.c:3201 0030: ec 46 c2 0c a9 93 5b 60 5e 5d 81 f2 b3 b0 b5 a8 .F....[`^]...... Dec 7 16:13:52 ssl_srv.c:3361 dumping 'my signature' (256 bytes) Dec 7 16:13:52 ssl_srv.c:3361 0000: 7a a7 0f 17 79 33 36 b7 0c 17 84 5b cc ca 1c 0c z...y36....[.... Dec 7 16:13:52 ssl_srv.c:3361 0010: cb f5 56 80 ea 5d 7f cf ab 86 d5 f9 91 36 0e 68 ..V..].......6.h Dec 7 16:13:52 ssl_srv.c:3361 0020: 76 c6 c0 4b 77 bf 28 e7 b8 c0 a7 de ea d8 a0 30 v..Kw.(........0 Dec 7 16:13:52 ssl_srv.c:3361 0030: 3a b3 4e 24 ea e0 c8 0b e7 ad ba 4d 35 30 c0 39 :.N$.......M50.9 Dec 7 16:13:52 ssl_srv.c:3361 0040: a8 ff f0 6d 8f 28 db fc 5e c3 fb 21 fe 76 dd 07 ...m.(..^..!.v.. Dec 7 16:13:52 ssl_srv.c:3361 0050: 1b 2c 58 fe 7b 0a f5 72 49 d9 d1 9a 10 af 29 21 .,X.{..rI.....)! Dec 7 16:13:52 ssl_srv.c:3361 0060: 7a d9 01 a4 99 01 de 6e ae fd 76 ce b9 64 84 49 z......n..v..d.I Dec 7 16:13:52 ssl_srv.c:3361 0070: 96 97 2d 3a 15 ed 5b de d5 99 9d 1b eb 4c 1a aa ..-:..[......L.. Dec 7 16:13:52 ssl_srv.c:3361 0080: a8 e6 36 29 bd 67 b0 50 91 6e b8 cb 0a 42 4a 79 ..6).g.P.n...BJy Dec 7 16:13:52 ssl_srv.c:3361 0090: 4a 65 ae 95 d0 bc 5a e1 e7 75 2a f9 ba 51 1c 2c Je....Z..u*..Q., Dec 7 16:13:52 ssl_srv.c:3361 00a0: ff 67 e3 df eb a0 d7 6c d2 cf 71 d2 ea 24 7c 16 .g.....l..q..$|. Dec 7 16:13:52 ssl_srv.c:3361 00b0: f8 5b 72 83 56 a9 48 e1 b6 cd d0 bd dd 20 23 d5 .[r.V.H...... #. Dec 7 16:13:52 ssl_srv.c:3361 00c0: 80 c4 bd 6e 42 1c 61 b2 e4 41 01 ce c8 85 26 21 ...nB.a..A....&! Dec 7 16:13:52 ssl_srv.c:3361 00d0: b4 0a 5e fe 50 0d 6f 45 a8 fe 15 18 87 e3 ae b3 ..^.P.oE........ Dec 7 16:13:52 ssl_srv.c:3361 00e0: ac 46 c5 c6 bb bc 70 1b 32 b7 15 c5 3d fb 09 1e .F....p.2...=... Dec 7 16:13:52 ssl_srv.c:3361 00f0: 8f 69 f7 e7 a8 86 4d 0b 87 ea 4b fe 08 7e 00 0b .i....M...K..~.. Dec 7 16:13:52 ssl_tls.c:3289 => write handshake message Dec 7 16:13:52 ssl_tls.c:2917 => ssl_flight_append Dec 7 16:13:52 ssl_tls.c:2953 <= ssl_flight_append Dec 7 16:13:52 ssl_tls.c:3425 <= write handshake message Dec 7 16:13:52 ssl_srv.c:3380 <= write server key exchange