Ya I got a feeling you forgot to or didnt see the clip on there, once that clip is removed theres nothing stopping it from sliding off.

Also you dont need to remove the rotor like said above, you should be able to just take out the axle itself and replace the bearings then reinstall the axle.

Unless you bought a whole new carrier with bearings, then it would have to come out I guess