@ajkr it might be time to replace the speaker. Follow something like [https://www.youtube.com/watch?v=iM-8MeUsLCc|this video] to do so. At around 2:10 it shows the speaker and how to remove it.
Replacement speakers are readily available at many online stores. Purchase from a vendor that meets your standards.
While you work on your phone, take lots of pictures. Once you are done with it, take those pictures and create a guide for iFixit. That will help the next person that has the same problem as you do. It's easy and fun to create these guides. https://ifixit.com/Guide/new The iFixit community will appreciate this.