• cobysev@lemmy.world
    link
    fedilink
    English
    arrow-up
    46
    arrow-down
    4
    ·
    1 day ago

    I guarantee this update didn’t drop on Thanksgiving. Photo OP probably hasn’t turned it on since their last BBQ months ago and is just noticing - on Thanksgiving - that an update pushed a while ago that they now need to install to get started.

    Pro tip: Start up your electronics a day or two in advance of events, so you can pre-patch anything that needs it.

    Source: Former IT guy here, who had to ensure that updates ran at the most convenient times possible for thousands of users. “Patching Tuesday” is an unofficial but well recognized “holiday” for IT folks. It’s not first thing Monday morning, which could throw off the workflow for the week, but it also gives the max amount of time to resolve any issues that patching might cause, so we (hopefully) don’t have to work through the weekend.

    Pay attention to when your stuff requires patches. A lot of the time, it’ll pop up on Tuesdays.

    • mic_check_one_two@lemmy.dbzer0.com
      link
      fedilink
      English
      arrow-up
      31
      ·
      1 day ago

      Source: Former IT guy here, who had to ensure that updates ran at the most convenient times possible for thousands of users.

      I used to work at a theater owned by a city. So we used the city’s IT department, and their network. During COVID, live-streaming took off. The city wanted us to install a streaming video package. After a month or two of installing a full video system, we finally get around to testing the stream. Boot up AWS, and it runs fine. We’re streaming in full 4K. Great!

      So the show rolls around. It’s Saturday, 7:30pm start time. We start the show… And the stream instantly shits the bed. Like we go from full gigabit upload speed, to less than a single megabit. We’re lucky to get 56kbps speeds. We’re getting one or two frames per second if we’re lucky.

      Sunday, we test the stream ahead of time, and it works flawlessly. Show starts, and the upload speed drops to fucking dial up.

      Monday morning rolls around, and IT strolls in to check their tickets. Sees a hundred from us, and gives us a call. They run a test on their end. No issues. They run a test on AWS. No issues. They run a test on the fiber backbone between the theater and city hall. No issues. They call the ISP. ISP said they didn’t have any issues over the weekend. IT shrugs, and marks the tickets as solved.

      Next weekend, same thing. We’re wondering if IT is automatically throttling us, or if we have a malicious user on the network. We’re asking about QoS, or maybe automatic port control kicking in when the stream starts. Monday rolls around, and IT marks it as solved again.

      Third weekend, same thing. This time, the city manager’s office is getting calls from angry patrons who paid for streaming and can’t watch their streams. Monday morning, IT rolls up. They run some more tests, and still can’t find anything wrong. They swear up and down that it’s nothing on their end, and it must be something on ours.

      After four months of this back and forth, IT finally admits that they have all of their maintenance tasks to run at 7:30 over the weekend. Every single computer, server, and fucking toaster connected to the city network begins their updates at exactly 7:30. Thousands of city devices, all singularly focused on devouring our upload speeds. Servers run off-site backups. Those backups consume all of the upload speeds for the entire city network. IT refuses to change the time, because “this is what works for us. It’s after city hall closes, so we don’t have any users who are affected. It hasn’t been a problem in the past.”

      • rekabis@lemmy.ca
        link
        fedilink
        arrow-up
        8
        ·
        1 day ago

        And in those four months, did no-one think of firing up WireShark to see what was floating across that network during that time period?

        Seems like someone dropped the debug/analysis ball…

        • Lv_InSaNe_vL@lemmy.world
          link
          fedilink
          arrow-up
          2
          ·
          edit-2
          19 hours ago

          As someone in IT, the answer is in the comment.

          City Hall is closed by 7:30

          So nobody was ever in the office and nobody on the team wanted to stay (I’m guessing here) 2.5 hrs after work to actually do any troubleshooting and because it was never a problem during office hours then who cares

          • rekabis@lemmy.ca
            link
            fedilink
            arrow-up
            1
            arrow-down
            1
            ·
            12 hours ago

            Read the comment more carefully… while IT was most certainly not at their posts, this implementation team was actively monitoring the rollout and witnessing the carnage.

            • Lv_InSaNe_vL@lemmy.world
              link
              fedilink
              arrow-up
              1
              ·
              edit-2
              11 hours ago

              Yes, I you agree with me. They absolutely were not at their posts 2.5 hrs after end of day on a Friday, which is likely why they responded to the tickets Monday morning.

              And I am also agreeing with you that there should have been someone(s) who stayed late to actually watch this. But it’s not gonna be me haha

              And of course, this is all speculation from a random unsourced comment on social media. It’s not that deep ❤️

        • mic_check_one_two@lemmy.dbzer0.com
          link
          fedilink
          English
          arrow-up
          6
          ·
          1 day ago

          I wasn’t in IT, so my hands were tied. If I tried running a network scan, I’d have been able to hear the screeching all the way from city hall.

          • rekabis@lemmy.ca
            link
            fedilink
            arrow-up
            1
            ·
            12 hours ago

            Never said it had to be you.

            But a threat to do exactly that would have likely called IT’s bluff long before the four-month mark.

        • GreenKnight23@lemmy.world
          link
          fedilink
          arrow-up
          5
          ·
          1 day ago

          what can you expect, they’re probably getting paid 40-50% of what they should be getting paid.

          pay less get less.

          my pride as an IT worker wouldn’t have allowed me to let it fester for 18 weeks though.

    • zipzoopaboop@lemmynsfw.com
      link
      fedilink
      English
      arrow-up
      29
      arrow-down
      2
      ·
      1 day ago

      Pro tip: don’t buy a fucking BBQ that connects to the Internet.

      No appliances in general while we’re at it

      • ArcaneSlime@lemmy.dbzer0.com
        link
        fedilink
        arrow-up
        6
        ·
        edit-2
        1 day ago

        Seriously. “Start it a day early” My brother in Christ why does your grill need wifi? Do you get updates when the steak is ready? Can it flip your burger?!

      • ZMonster@lemmy.world
        link
        fedilink
        arrow-up
        4
        arrow-down
        1
        ·
        1 day ago

        Have tons of devices that can connect to the Internet. Apparently I’m the only one here resourceful enough to not connect them

        • JcbAzPx@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          ·
          15 hours ago

          I’d rather not reward manufacturers that include that just to raise the price and track and sell your data with my money.

          • ZMonster@lemmy.world
            link
            fedilink
            arrow-up
            1
            ·
            15 hours ago

            I think you missed the bus.

            I’d rather not reward manufacturers that include that just to raise the price

            Cool cool, you do you homie

            and track and sell your data with my money.

            If you didn’t connect the device to the internet… Then… There is no “data”

            This isn’t a difficult concept. The device didn’t ship with your WiFi broadcast ID and password preset in the firmware. Are you saying that the product has a sim card that will just relay info back regardless of whether or not you connect it? And the manufacturer is cool just paying for this connection? What “data”???

            Yet again, don’t willingly and intentionally connect it to anything that has access to the internet. JFC, is it really that abstract of a concept that confounds all others? I don’t think I needed to be told this. It’s the very definition of self-evident.

            What the absolute fuck is happening? Did everyone eat brain tumors for breakfast?

            • JcbAzPx@lemmy.world
              link
              fedilink
              English
              arrow-up
              1
              ·
              14 hours ago

              Whether it actually happened or not, that was their intent. If you’re happy to reward that behavior, well, “you do you.”

        • vaultdweller013@sh.itjust.works
          link
          fedilink
          arrow-up
          6
          arrow-down
          2
          ·
          edit-2
          1 day ago

          I rip the wifi card out and if that’s not available all things can be solved with the proper application of an angle grinder.

          • ZMonster@lemmy.world
            link
            fedilink
            arrow-up
            2
            ·
            15 hours ago

            Careful, the last person with a modicum of common fucking sense almost got flamethrowered by Beef Supreme.

            “Hey, I just thought of something crazy, maybe DON’T kick yourself in the balls?”

            one-of-usone-of-usone-of-usone-of-usone-of-usone-of-usone-of-usone-of-usone-of-us^one-of-us

    • GreenKnight23@lemmy.world
      link
      fedilink
      arrow-up
      34
      arrow-down
      1
      ·
      1 day ago

      pro tip

      I get it. I hate it, but I get it.

      another pro tip from someone else in IT: see that appliance with the digital screen? fuck it. don’t get it. get the old shitty one that’s $800 less that doesn’t have WiFi or non-tactile buttons. you know what doesn’t need firmware updates? a charcoal Weber grill.

    • MonkderVierte@lemmy.zip
      link
      fedilink
      arrow-up
      5
      arrow-down
      1
      ·
      edit-2
      22 hours ago

      Thanks, but i prefer most utilities without wifi and need of patching. Each wifi device is running a full blown OS, for which the (cheapest possible) hardware will start to fail after 5 to 10 years. Experience from a wifi capable HP printer; wifi was the first that failed. Not to talk about never patched security holes.

    • Zorsith@lemmy.blahaj.zone
      link
      fedilink
      English
      arrow-up
      4
      ·
      edit-2
      1 day ago

      Tuesday is the perfect day for it. Finish up the update on Friday, review it Monday and fix where you probably fucked up something and didn’t notice, push it the next day.