Pursue a program for physics in University even if i didn't take Math AA HL?

[RahilJoshi]

Right now i've taken Math AA HL, Physics HL, Economics HL and English A SL, Chinese AB-initio, Chemistry SL

I've tried Math HL for a bit and its incredibly hectic, coming from International maths from IGCSE even the basic concepts are hard for me, so i'm considering stepping down to AA SL because i could get a 6,7 for SL rather than struggle and take down all my HL's, as physics HL also a very intensive course (also math based) is a huge course too, i've decided to take SL math in order to invest time and effort in Physics and Sl math in order to get a good IB result, But what i worry about is that if i take this step down, will universities in the US or Canada (where i am looking forward too right now) underlook me? or simply reject me if i don't take AAHL? i've looked and seen many websites but can't manage to find the "IB REQUIREMENTS" page, if any of you around have experience or advice i would highly appreciate it, I love physics and want to pursue a career, but if math is a problem, physics will obviously be ruled out, i just want to see what i can do, thanks to anyone who helps!