I think you can say that someone was “left alone” without any implication that it was done to them, just that it merely happened.