Greetings!
I'm having an issue while veryfing signature with imported RSA2048 public key, generated with Win7 CryptoAPI (PUBLICKEYBLOB) into latest mbedtls 2.28.x. The blob contains RSA modulus N (256 bytes) and public exponent E (4 bytes) - I do extract them succesfully, then provide into mbedtls_rsa_import_raw. It all goes smth. like this:
u32 errval = mbedtls_rsa_init(ctx, MBEDTLS_RSA_PKCS_V15, 0); // errval == 0 here errval = mbedtls_rsa_import_raw(ctx, n, nlen, 0l, 0, 0l, 0, 0l, 0, e, elen); // errval == 0 here errval = mbedtls_rsa_complete(ctx); // errval == 0 here errval = mbedtls_rsa_check_pubkey(ctx); // errval == 0 here
Then i ran: errval = mbedtls_rsa_pkcs1_verify(ctx, 0l, 0l, MBEDTLS_RSA_PUBLIC, MBEDTLS_MD_SHA512, 0, _src, _sign); and get -0x4380 (verify failed) _src - is sha512 hash of data to be verified (64 bytes) _sign - is 256 bytes of signature, provided by win7 cryptoapi P.S. just in case, i did tried messing with endianess in every way for e AND n, it didn't help.
I added a little debugging inside library/rsa.cpp, turned out we do call mbedtls_rsa_rsassa_pkcs1_v15_verify, and there is a memcmp between 'encoded' and 'encoded_expected' bufs. 'encoded' is derived from signature (_sign), and 'encoded_expected' is derived from hash (_src)
printhex for 'encoded' looks like this: 1a1da83b 14be17a2 c8401d41 1d453909 ... total 16 lines (256 bytes) ... 7fb37ea2 719a5562 aebdb3ed 296e0ed1
but printhex for 'encoded_expected' looks like this: ffff0100 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ... wtf??? padding ??? ... ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 30513000 6009060d 65014886 03020403 40040005 b190b45a a40b354f 32271b34 f022abd3 ... sha512-derived data here, 64 bytes 557abf2b e2cc4e0f 0b77bdfc b45688b0
So, there is no way these two bufs match. I wonded if there is some issue in parsing _sign, or I didn't prepared input data good enough. Any ideas?
B.R., m4D.