We like to think of time like "Back to the Future" in terms of a timeline that could possibly be traversed forward and backward, like time itself is this real thing we can go to and come back to.
According to physicists, this isn't the case, rather it is just entropy alone.
Just for a second, throw the concept completely away of past and future so there is only the present. Now imagine, things are always changing, including your brain's wiring (memory) where it retains a pattern that continually stores patterns of previous states.
Time itself exists only as a concept in that space. Time is just the memory of previous states of existence.
The only way to go back in time then is to have a huge machine that actually arranges all the atoms into the previous state. There is no where to go into the past.
Any hoo. I thought that was interesting.
Here's a really great TED video on this subject:
Here's an awesome video on the arrow of time by Sean Carrol who is a genius physicist.