I'd agree with the previous answer.
Just because your nephew's parents don't seem to teach him appropriate behaviour is no reason why you shouldn;t do so in your own situation.
At eight, he is of an age where he knows what 'no' means, in what ever area you chose to use it, and you certainly shouldn;t feel bad for laying down a few basic rules about areas of access in your own home.
I would be equally firm with your brother in a similar way.
Just because they are your relatives, does not give anyone the right to behave in your house as they do in their own, and it is for you to enforce your rules to have things your way.
If it makes things uncomfortable, ride it out, but the way children operate, your nephew will actually appreciate some guidlnes and knowing where he is at your home, and it may even increase his desire to come and visit!
Your brother may be more difficult, but bite the bullet, and talk it over with him.
Just be firm, and calm, and he will soon learn that respect is something you offer, and expect, and the rules are there at your house.