Hi Raoul,
Do I understand correctly that the problem with Mbed TLS as it is now is that you need MBEDTLS_HAVE_ASM enabled for AESNI and disabled for bignum? Would you want even more granularity?
In principle, we could create a new compile-time option MBEDTLS_MPI_ASM, defaulting to what it does now (enabled if MBEDTLS_HAVE_ASM if the platform is one of the known ones) but overridable in either direction. However I don't like it much because we're trying to cut down on compilation options to reduce the maintenance burden: each compilation option adds complexity which makes it harder to understand all interactions between different parts of the library, and adds some testing burden. Symmetric and asymmetric crypto are mostly decoupled, so this one wouldn't be too bad, but it all adds up.
In the long term, I expect that the different cryptographic algorithms will become more decoupled, with a possibility to treat each one as a black box that could be implemented by a hardware driver, and with a uniform mechanism to select drivers (or maybe they should be called engines). The evolution of the crypto part of the library to PSA is going in this direction. But we aren't there yet.
Sorry that I don't have a more satisfying answer here.