[mbed-tls] Re: Performance tuning of SHA256 on big files

-- Gilles Peskine Mbed TLS developer On 24/10/2022 13:35, James Liu via mbed-tls wrote: > Hi, > > Thanks for the tip. I test mbedtls-3.2.1 in M1 by adding two options > in mbedtls_config.h: > > MBEDTLS_SHA256_USE_A64_CRYPTO_IF_PRESENT, > MBEDTLS_SHA512_USE_A64_CRYPTO_IF_PRESENT. > > There are substantial improvements on two big files using sha256: > CentOS-8.5.2111-x86_64-boot.iso (827.3 MB): (before)*5.9 > sec,*(after)*32 sec* > CentOS-8.5.2111-x86_64-boot.iso (10.79 GB):   (before) *78 sec, > *(after)*41 sec* > * > * > But the problem I'm trying to solve is still there: > 1) sha256 incurs high overhead on big files (less than a few seconds > are desired), considering there are many big files to process in real > time; > 2) not sure if tuning could work in x86. > > Is it possible to slice a big file into chunks and compute hash > separately and merge?  I guess other crypto libraries or utilities > have same overhead on big files. > > Regards > > Tom Cosgrove Tom.Cosgrove@arm.com 于2022年10月24日周一 16:24写道： > > Hi > > > I use same code with mbedtls-3.1.0 to run tests in x86, and > performance is still downgraded > > Mbed TLS has no acceleration for SHA-256 on x86 or x86_64 - > optional or otherwise - it just uses C code. So this is as expected. > > Thanks > > Tom > > ------------------------------------------------------------------------ > *From:* Liu James via mbed-tls mbed-tls@lists.trustedfirmware.org > *Sent:* 22 October 2022 10:28 > *To:* mbed-tls@lists.trustedfirmware.org > mbed-tls@lists.trustedfirmware.org > *Subject:* [mbed-tls] Performance tuning of SHA256 on big files > Hi, > > This is an updated post from > https://github.com/Mbed-TLS/mbedtls/issues/6464, which should be > posted in mbedtls mail list. > > My question is how to significantly improve SHA256 performance on > big files (regardless of architectures). > > *=== Updates* > I use same code with mbedtls-3.1.0 to run tests in x86, and > performance is still downgraded. > > Mbed TLS version (number or commit id): *3.1.0* > Operating system and version: * Centos-8.5, CPU 11900K* > Configuration (if not default, please attach|mbedtls_config.h|): > Compiler and options (if you used a pre-built binary, please > indicate how you obtained it): *gcc/g++ 8.5* > Additional environment information: > > *Test files and performance* > CentOS-8.5.2111-x86_64-boot.iso (827.3 MB):|sha256|*5 sec* > CentOS-8.5.2111-x86_64-boot.iso (10.79 GB):|sha256|*66 sec* > > > Also, as advised I try to turn on > "MBEDTLS_SHA256_USE_A64_CRYPTO_IF_PRESENT " and > "MBEDTLS_SHA512_USE_A64_CRYPTO_IF_PRESENT" using mbedtls-3.2.0 in > M1, but compiler reported the following error: > > CMake Error at library/CMakeLists.txt:257 (add_library): >   Cannot find source file: > >     psa_crypto_driver_wrappers.c > >   Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm > .ixx .cppm .h >   .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 > .hip .ispc > > > CMake Error at library/CMakeLists.txt:257 (add_library): >   No SOURCES given to target: mbedcrypto > > > Thanks for your help. > > *=== Original message at github* > > > Summary > > |sha256()|and|sha1()|incurs significant overhead on big files(~1G > above).*This might not be an issue*, and I'm looking for an > efficient way to calculate hash on big files. > > > System information > > Mbed TLS version (number or commit id): 3.1.0 > Operating system and version: M1 OSX > Configuration (if not default, please attach|mbedtls_config.h|): > Compiler and options (if you used a pre-built binary, please > indicate how you obtained it): Clang++ > Additional environment information: > > > Expected behavior > > Fast calculation of big files in less than 1 second > > > Actual behavior > > Test files: > CentOS-8.5.2111-x86_64-boot.iso (827.3 MB):|sha1|*3.3 > sec*,|sha256|*5.9 sec* > CentOS-8.5.2111-x86_64-boot.iso (10.79 GB):|sha1|*40 > sec*,|sha256|*78 sec* > > > Steps to reproduce > > ISO files can be downloaded > at:http://ftp.iij.ad.jp/pub/linux/centos-vault/8.5.2111/isos/x86_64/ > > Make sure use fast disk, say nvme, to store ISO files, or else > loading big files could take lots of time. Also > use|user|from|time|command to measure performance. > > Workable code of sha256: > > |string test_sha256(string file_path) { mbedtls_sha256_context > ctx; FILE *fp; string output; int BUFFER_SIZE = 4096; uint8_t > buffer[BUFFER_SIZE]; size_t read, k_bytes; uint8_t hash[32]; > mbedtls_sha256_init(&ctx); mbedtls_sha256_starts(&ctx, 0); fp = > fopen(file_path.c_str(), "r"); if (fp == NULL) { > mbedtls_sha256_free(&ctx); return output; } while ((read = > fread(buffer, 1, BUFFER_SIZE, fp))) { mbedtls_sha256_update(&ctx, > buffer, read); } mbedtls_sha256_finish(&ctx, hash); > mbedtls_sha256_free(&ctx); fclose(fp); // update hash string, omit > here return output; }| > > >