My Xbox one controller is connected, but I can't do anything

Whenever the controller is plugged directly into the front of the console, via a micro USB to USB plug, I can move around and select everything just fine, but when I move to battery or battery pack power I cant select anything or move around at all.

It still shows that the controller is connected, and I am able to hit the xbox button to bring up power off console, controller, and restart, but I am unable to select anything then either. So it reads that its connected but I cant do anything. 2 different controllers same problem, I have tried regular energizer batteries, and a battery pack with the same results.

Ive thought about buying a new controller but I dont want to spend that money if the controller isn't necessarily the problem.

Have done a hard reset and have reset the connection between the xbox and the controller itself with the same results.

Any help or advice would be Great!!!