As a developer, I find retros most useful when EMs (and probably PMs, depends on how things are configured organizationally) aren't present, as it should give a chance to talk freely about challenges you or the team face, which often are organizational in nature and may involve one's management chain. My current role doesn't adhere to this, and it makes retro often really painful.
I would agree with you. I never want my manager there. As an EM, I never want to be there. And yet, as the years go by, when I have an EM role I find my presence is requested. Again, who knew?
I don't know, at least for my current team a lot of the stuff we complain about in the retro is beyond the team level: coordination between teams, infrastructure issues, ... The intra team issues are usually talked about before (e.g. during standup, or just asynchronously).
If the EM is not there, those are not actionable. What we try to do is: do the retro by ourselves, and invite the EM at the end of there is stuff for them. Usually they are not available though...