• tooclose104@lemmy.ca
    link
    fedilink
    arrow-up
    51
    ·
    2 days ago

    A typo in software development or other shell based work could completely ass womp a system in ways that could lose a company lots of money.

    Oopsies on prod systems, even with an outage window, can really fuck shit up. Seemingly small mistakes can quickly snowball into systemwide outages.

    • jjjalljs@ttrpg.network
      link
      fedilink
      arrow-up
      52
      ·
      2 days ago

      It’s wild to me how some places I’ve worked are like locked down, all the infrastructure is in terraform or whatever and can be deployed immediately… and other places are like “ssh into prod with the credentials from confluence, edit the config in vim, and paste the new code into a new file”

      • fibojoly@sh.itjust.works
        link
        fedilink
        arrow-up
        1
        ·
        9 hours ago

        We literally have both and our new Tech Lead is absolutely terrified that I can deploy shit into prod right from my command line. I think it’s hilarious (because yeah, it’s total shit, but our Dear Leader was a complete muppet who refused to listen and prioritise doing things right, so here we still are). Also our GitLab is the free licence because why pay for the thing that takes care of our entire codebase, eh ? 👌

      • tooclose104@lemmy.ca
        link
        fedilink
        arrow-up
        4
        ·
        1 day ago

        I’m at one of the latter, so I feel this in my bones. I’ve watched what should have been an innocent config change snowball into a pair of VM clusters shitting back and forth for 2 hours. Implemented strict change control that day. Kind of a pain, but the team learned a lot that day!