TLS/HTTPS has to validate that the certificate has the name which you told it to connect to. For DNS names, that works. For IP addresses, that often does not work, because the certificate does not have the IP address in the subjectAltname. (It can be done).
THe question is really, why do you want to connect by IP? That just makes your life significantly harder.