One cool thing about math though is that continuous/analysis concepts and discrete concepts interact with each other. There's a fair bit of calculus in e.g. AoCP. At some point number theory as well. Kind of depends on how deep you want to go. You can learn a lot about cryptography without really needing a very strong mathematical base.