Part of the responsibility we undertake when we start life with a pet is to always do the best for them - and sometimes that means unbearable pain for us.
As the other posters have said, you did what was right for him. His life was effectively over, he was simply existing, and it would be cruel to keep him suffering because you could not bear to part with him. That means a difficult decision to be made, carried through, and lived with, but it is still the right thing to do.
As your grief eases, you will remember the good times with your dog, trhe pleasure you had being together, and that is what will remain when the pain receedes and eventually becomes manageable.
Hang on to that thought, and know that although losing him was so very hard, it was still the way to love him best.