Part of it is that minor changes can be incredibly disruptive to our personal lives, while having no meaningful effect on the bigger picture.

There was a movie (About Time I think) where this lineage of guys can time travel as some inhereted ability. They have a strong rule within the family though to never time travel to before one of their children is born, because the smallest change can result in a different sperm/etc and finding they have completely different child when they return to the present.