This video shows the problem. It’s consistent like this. I run the game in the editor, it either does perfectly as expected or bounces around like crazy. If it works fine, then next time I run the game it will bounce around like crazy. If it bounces around like crazy, then next time it will work as expected.
HOWEVER, if I run it outside of the editor it will bounce around like crazy every time.
So instead of using the primitive box with the rigid body I used a custom mesh for the collider and that seems to have fixed it. I dunno if this is due to something wonky with the primitive box shape and collision or what.