You have a good point, but I think you can also read it in a different way. You can understand it as an encouragement. It is an answer to the question “what can I do? I am just a small nobody”. If you understand that time travelers are scared to change even the smallest things because they will alter the future, it will give you a perspective to see that everything anyone does can make a difference.