I agree with redcrx- we have always travelled with our boys - they are now 8 and 7, but tbh it was a lot easier when they were that age. We left probably about an hour before they were due theie afternoon nap and the generally dropped of before they got bored and slept for a good few hours, waking up in time for a stop and snack and then you are usually nearly there. We would let them get out to stretch their legs, or if it was raining, we let them have a roam around in the boot (we had a scenic, so plenty of room for them to stand up in the back and not when the car was moving!) - the boot also becomes really useful as a changing station when required! Just keep your usual changing things available for the journey, plenty of snacks, a couple of favourite toys and maybe a couple of new things to keep him occupied and you will be fine. Babies are enormously adaptable and he will be fine, they seem to know and accept that rules for places are different and they will behave in different ways in different environments - for instance, they will accept from an early age that although a granparent lets them jump on the sofa, they know that they are not allowed to do it at home. Clever wee things really. I am sure he will know when you are back at home and holiday is over that he goes back to his old routine.
I guess it is up to you, it sounds to me that you would enjoy taking him with you and doing the family trip rather than going off on your own so go for that. On the other hand, if you would enjoy the freedom of being away by yourself, knowing that a bit of father/son bonding was going on, then do that.
Enjoy!