I’ve spent the better part of two decades running servers for web hosting companies. I think I can safely say that in that time, I’ve come to understand the basics of the various protocols the underpin the web. There are problems with all of them, to one extent or another, but most are reasonable solutions to the problem they claim to solve. There is one protocol, however, that I can say with certainty is the wrong solution.
You have, no doubt, encountered an SSL secured web site along the way. Any ecommerce site, for instance, likely has an https:// url. Underlying the https:// scheme is a technology commonly called SSL. Now, before I go any further with this, I should point out that SSL itself is not necessarily a bad protocol if used the way it is designed in a situation where that design makes sense. What SSL does is provide a means for encrypting communication between two endpoints. It also provides a means for endpoints to authenticate each other. Both goals are laudable.
The problem with using SSL for https is that SSL does not separate the encryption and authentication features. Furthermore, for authentication, it relies on one of two things. Either a certificate provided by one end must be explicitly accepted by the other or it must be signed by a trusted authority. This sounds reasonable on the surface and for things like ecommerce, it makes perfect sense.
There are, however, many cases encrypting the communication so that a casual evesdropper cannot listen in is important but the specific identities of the participants in the communication makes no difference. Perhaps authentication is being handled out of band by some other mechanism or what have you. Futhermore, the SSL authentication phase occurs before the HTTP protocol phase starts up so it becomes impossible for HTTP to determine which endpoint it should behave as because the authentication stage has already failed.
The proper solution would be to have two different protocols in play. The first would establish a private communication channel using some sort of encryption, possibly using something like diffie-hellman key exchange. This stage would require no knowledge of who the endpoints are. Then, if the endpoints don’t care about authentication but instead just about keeping their chat private, everything can happily move along without any further confusion.
On the other hand, if some sort of authentication is expected, then whatever mechanisms are available for mutual authentication can be used, ranging from plain old http authentication to some form of certificate with a chain of trust.
I should mention here that many cases where https is used for web pages, both mechanisms (privacy and authentication) are desired. However, http is used for many things that are not the web and there are even web sites where some level of privacy matters but authentication is not critical. Of course, one would probably need two different url schemes to support two types of connection. The https one could continue as it already does while another scheme could specify an unauthenticated secure layer, one which does not rely on centralized certificate authorities and other such things.
All of this leaves aside the fact that the vast majority of people do not understand certificates, certificate authorities, and their trappings. Thus, when faced with a security warning from their web browser, they simply click “yes, fine, go ahead” (not good for things where authentication matters) or panic and do nothing (a better but rarer outcome). Security is hard and requires active participation from all parties. No amount of engineering can erase that problem, so why do we persist in making SSL more and more complex with more and more warnings for people to misunderstand and ignore?
However you slice it, https as it currently exists is the wrong solution to the problem. We need a simple solution that provides encrypted communication for the base level communication channel and does nothing else. No authentication or other validation of endpoints. It simply creates a private communication channel. This protocol needs to be able to do its job without any external inputs from anywhere. Such a protocol could be used in many places, not just http.
Then, for those things that need authentication, we layer an authentication scheme on top of the encrypted communication channel. Such might be a certificate exchange which relies on cryptographically signed certificates and certification authorities if appropriate, or it could rely on simple shared secrets or other authentication mechanisms. Then, the level of security (both privacy and authentication) can be chosen by an application based on its actual needs rather than the current largely binary choice.