Doesn't recognize sim card

Samsung Galaxy Note II T-889 (T-Mobile) doesn't recognize the sim card after I dropped it, it shows circle with cross line instead of antenna bars on the top bar.

If I restart the phone few times, it picks up the sim card (One out of twenty times). However just after few minutes, the phone automatically restarts and again it doesn't recognize the sim card (probably the sim holder is still working). I tried to open the phone, once I pop out display assembly from the mid frame, it picked up the sim card (for few minutes again).

I am thinking to replace the sim holder if it doesn't solve the issue, what else issue it could be? Is it something with the software, or some other component I need to change?