Not necessarilly.
There is nothing that fundamentally bars travelling backwards in time although there are a number of paradoxes like the Grandfather one.
Killing your own Grandfather is counter intuitive but there are a lot of things that are counter intuitive. It kind of breaks off an island of causality - that may be possible - just hard to imagine in the same way that many Quantum effects are
You also have to understand the real nature of the so called "Butterfly effect" this comes from what has been known as Chaos theory. The name comes from the idea that the flap of a butterfly's wings in New York can trigger a tornado in London.
Many systems are very sensitive to initial conditions, your presence in a past time would trigger so many subtle changes that history would change rapidly. Basically things don't come out the same way twice - go back with the lottery results you're likely to be disappointed.
However the biggest paradox is the possibility that I might travel forwards in time find a new discoverand travel back to explain it to it's inventor. ( I think this was hinted at in one of the Terminator films ) This is effectively new information coming from nowhere - much harder to live with.
If anybody is interested in the details of time travel I can recommend this video lecture by Paul Davies of Imperial college London. The Vega Trust hasit on-line here:
http://vega.org.uk/video/programme/61
Similarly