i think theres one rather large factor you have skipped over here...why haven't you seen him for 21 years?
perhaps this is the reason he is a bit laid back.
and i have to say IAP has a good point, whether you wanted to hear it or not - just because someone gave birth to you does not give them the right to come into your life 20-odd years later and try to take over, and also play mind games to try to gain control as IAP mum did - this is unacceptable behaviour, and she did the right thing. quite simply you forfeit your rights when you give them up - and as adults they owe you nothing.
it sounds like you haven't fallen out with him though, so just give him time.
he has had a life for 21 years without you and perhaps he feels he is betraying his mum, by seeing you too often?
give him time, but why wait for him to contact you?
contact him.
he may well be sitting at home feeling upset that you haven't been in touch with him!