It sounds like he has simply thrown it up there, out of sight/out of mind. He shouldve come to you and said it upsets him and would you mind putting it away.
However, I dont think this is something you should be worried about. In fact, it shows how much he cares for you and doesnt wanted to be reminded of your past with another man.
I have to admit, I am the jealous type when it comes to silly things like that. As far as I am concerned with past reltionships, the past is the past and thats where it should stay. No matter what, you'll always have your special memories in your head.
If my fella did this to me, I think I would actually be understanding, although I may approach as to why he did it that way. He may have just acted on impulse, and we've all done that!!