The joysticks are screwed onto the potentiometers and are very tight, probably more secure than your grip after a long gaming session.

Don’t get tricked by that tempting screw in plain sight—that’s a tensioning screw. You will need to turn the joystick counter clockwise to remove using either the thumbstick or a pair of pliers.

Note: If you are reusing the joysticks be careful not to let the pliers slip and damage the thumbstick mounting grooves.