Here's a thought.
What if the point in time where the man in black fled across the desert, and the gunslinger followed, i.e. when Roland pops through the door at the top of the Tower (this time), is...