It sounds like you're looking at it from the perspective of someone who's wealthy enough to take on a few rental properties of their own. The economics start looking rather different from the perspective of a real estate speculation hedge fund. The same forces that make it such easy money for them are also the ones that make it not such easy money for you. When they drive up prices it curtails any more liquid form of asset growth you might have by pulling all the money over to the on-paper value of the property. That's fine for them because real estate is still pretty darn liquid from the perspective of hedge funds and REITs. But it's not very liquid at all from the perspective of a small-time landlord who needs to actually look their tenant in the eye and tell them they're facing eviction because the owner of their home needs to free up some spending money. This, in turn, helps them by creating barriers to entry that push smaller competitors out of the business and securing their place in the oligopoly they're trying to engender.
The actual number of residential properties owned by hedge funds is a pittance compared to the number owned by individuals and small time land lords. So I have a hard time seeing how they're pushing smaller competitors out of the business when the smallest competitors aren't even doing it as a business.