You need to up your concentration levels.
The ideal to reach is that ytou don;t hear conversation, just background noise.
Start by deliberately tuning out any specific conversation which your ears pick up. This can be done by leaning a little closer over your desk and work and really focusing on what you are doing.
With practice, this will become second nature.
Our building has had a major flood in its east wing, so a lod of people have been relocated temporarily with us for a few months while their accomodation is sorted out. In the space of one day, i have gained an immediate neighbour, someone opposite, four people being him, and four more behind me, in what were empty seats.
It bothers me not at all because i simply tune out their (considerable) noise and chatter.
It takes patience and practice but it can be done.
Good luck.