After watching pretty much every swimming event at the recent Olympics, there was definitely a lot of those turns spent underwater and not one of the swimmers seemed negatively impacted by it.
It's actually possible to swim faster underwater than at the surface due to differences in drag. In sanctioned swim meets such as the Olympics, competitors aren't allowed to go more than 15m underwater.
I guess the fastest stroke is the Fish Kick. Since you are kicking sideways and underwater, the pressure wave from your kick isn't being reflected from the surface or the bottom of the pool, allowing you to go way faster. It's bonkers! But I guess it's extremely difficult to pull off because of how hard it is to balance on your side while kicking underwater.