I don't even know why anyone wouldn't use the DNS challenge unless they had no other option. I've found it to be annoying and brittle, maybe less so now with native web server support. And you can't get wildcards.
Spivak is saying that the DNS method is superior (i.e you are agreeing - and I do too).
One reason I can think of for HTTP-01 / TLS-ALPN-01 is on-demand issuance, issuing the certificate when you get the request. Which might seem insane (and kinda is), but can be useful for e.g crazy web-migration projects. If you have an enormous, deeply levelled, domain sprawl that are almost never used but you need it up for some reason it can be quite handy.
One problem with wildcards is that any service with *.foo.com can pretend to be any other service. This is an issue if you're using mutual TLS authentication and want to trust the server's certificate.
The advantage to HTTP validation is that it's simple. No messing with DNS or API keys. Just fire up your server software and tell it what your hostname is and everything else happens in the background automagically.
And this is different from DNS how exactly? The key and resulting cert still needs to be distributed among your servers no matter which method is used.
With dns-01, multiple servers could, independently of each other, fetch a certificate for the same set of hostnames. Not sure if it’s a good idea though.
I guess it depends on the CA, but some do. Let’s Encrypt does, for example. I guess it’s useful for HA deployments, where load balancers might be spread out across multiple datacenters and stuff like that.
Not really, just forward .well-known/acme-challenge/* requests to a single server or otherwise make sure that the challenge responses are served from all instances.
Just like you can point .well-known/acme-challenge/ to a writable directory you can also delegate the relevant DNS keys to a name server that you can more easily update.