Needing prismatic correction in spectacles is a common issue. It seems more of a problem because obviously you have good vision and it seems strange to need glasses to correct this muscle balance issue.
The muscles that control your eyes can become weaker with age and this can couse these problems. Whenever you look at a target your muscles which control your eye movement have to work to allow your eyes to fixate on the target together. Sometimes your muscles can be a little bit weak and find it had to maintain this work.
If these problems happen more when you do close work this could be due to a 'convergence insufficiency' and sometimes people are given exercises such as 'pen to nose' to try to help these muscles work harder. The problem with doing these exercises is that they can cause the symptoms you have as they are doing the work that you feel is a problem. These are best carried out before going to bed I find.
If these are difficult to do or you still have problems then the prism in the glasses helps you control the muscles but personally I would recommend you use them only when doing the tasks you have found to aggrevate the problem. Some people may say they help retrain your eyes to do the work others may say your eyes become dependant on the help.
Surgery would not be a good option if there is no big problem to solve.
I hope this helps.