Beautiful Mind I can understand the hesitation if he doesn't make you laugh and the difference in lifestyle habits, but we are all different and we need someone who will complement us as individuals (not necessarily the same). Maybe you just need to put your doubts aside and give it a go, might as well try the fish before you throw it back in the sea.
Have you asked him how he feels about you? If you knew that he fancied you, would you fancy him? If the answer is no, put him in the friend zone, nicely.