Family give a lot more leeway than strangers, because they are family.
Just be straight with him, he knows you love him, and you are not being awkward - just explain that you need to make plans, I am sure he will understand better than you imagine.
You can't live your life worrying about not upsetting others at the expense of your own happiness, and I am sure your brother would not want you to, and will be happy to give you a clear plan so you can organise everything.