I cant imagine where we would be if people were having this mentality back in the early 1900s when rich people were trying to fly using loud, terrible airplanes. Fwiw I don't think super sonic travel is going to succeed any time soon, but it's crazy how prevalent this type of NIMBY mentality has become.
There's nothing wrong with people finding sonic booms annoying. Even Boom conceded this by naming itself after its worst externality, and pretending (in its early startup phase) that it was pursuing technology to make supersonic flight less loud.