Completely agree, speaking as a mathematician who put in the time to understand the details of sigma-algebras. It's an implementation detail. We don't need to talk about inductors and MOSFETs to describe how bubble-sort works, and we don't need to talk about sigma-algebras to talk about conditional probabilities or expectations.

A wacky alternative implementation of probability theory can be found in Nelson's book "Radically Elementary Probability Theory" - I highly recommend it as a way to see how little the underlying framework matters.