Why iMac 2011 mid 27' only run smoothly with an external monitor?

I would reconnect the internal display and then run the onboard diagnostics to see what it tells you.

Restart your system and press the D key to enter

Reference: Mac startup key combinations

I'm suspecting your discreet GPU has failed.

Diagnostics runs on the firmware driver for the Intel Intergraded graphics engine it has.

Your external display is able to run using the same Integrated graphics engine, but this time using the OS driver, as it we default to that.

You can also prove this using this great app gfxCardStatus

Remove the unit power supply and check for faulty capacitors. It would not be surprising that a 11 years old PS needs a recapping. I just worked on this iMac model and even if the power supply was working there was 3 damaged capacitors. Top of the cap is doomed, its a sign the cap is failing soon or later.