Okay, so the problem is, whenever the player tries to move, he does not always wander in the direction in front of him but instead may go diagonal to it. The only way to fix this is to his collider's friction to 0 which I really don't want to do.
I use applyForceLocal to move the player. Try to get the player to head straight for the red cube. You don't always go straight to it. Mouse is for moving camera, and W key is for forward movement:
Thanks for any help.