I have submitted PR#3245 in order to add Edwards curves support to Mbed TLS, with the eventual goal to add support for the EdDSA algorithm. There are still a few things to fix that require discussion.
Let's start the discussion with the first one. Adding a new curve type requires to add a new entry to the mbedtls_ecp_curve_type enum. The curve type used by a group is returned by the mbedtls_ecp_get_type function. It currently uses the coordinates type of the base point to determine the curve type. Montgomery curves are lacking the y coordinate, and the Short Weierstrass curves use the three x, y and z coordinates.
The Edwards arithmetic implementation in this PR uses the projective coordinates. As such it also uses the x, y and z coordinates and this gives no way to differentiate a Short Weierstrass from an Edwards curve.
I have currently implemented that by checking if the curves are the Ed25519 or Ed448 ones using the group id . I am not sure it's very clean and it won't scale if more curves are added later. Another alternative would be to add another entry to mbedtls_ecp_group to hold the curve type.
What do you think is the best option? Any other idea?