I did happen to have a basic dell keyboard and basic logitech mouse stashed away. I tried changing out my basic dell mouse and logitech gaming keyboard with them, but it didn't fix it. I don't have any other devices plugged in either like a trackball or any other USB devices.
I'm going to keep tinkering but open to other ideas. The mouse was drifting wildly when I first got it, but then when I relaunched the game later that night, it was fine. Unfortunately it went back to being all wonky the next day. So just need to figure out the magic combination of whatever I did again...

*UPDATE*
Hmm so installed some windows updates and it is working again. Will be back if the problem comes back up.