He was brilliant in physics, not physical relationships. He married his fourth wife because he had a dream with a pool table where he couldn't score a ball for goodness' sake.
Important context is that this was his first wife, his childhood sweetheart who was already terminally ill with tuberculosis when they married. He was also working on the Manhattan Project at the time; while there were, no doubt, warm bodies around, some of them radioactive, this wasn’t either of his later two marriages.