Server is slow for me in the past few days. Anyone else feeling the same?

  • Salamander@mander.xyzM
    7·
    2 days ago

    I can see that since October 13 the CPU use spiked to 100%…

    I just reset the server to check if that would fix it quickly. It don’t think it did.

    I can see the processes that are taking up a lot of CPU (parallell database queries it seems), but I still don’t understand what is causing these and why they have spiked since Oct 13. I will need to investigate.

    • Salamander@mander.xyzM
      3·
      2 days ago

      So far, I have been able to ‘control’ the CPU use by setting limits to the process that pulls stuff from the database (pool size, CPU, memory).

      This does release some of the CPU for other tasks, but I think that that what creates the lag might actually be the clogged database queries. So, constraining those resources might not solve the lag problem.

      • Admiral Patrick@lemmy.world
        3·
        2 days ago

        Check your web access logs for these 3 IPs:

        • 134.19.178.167
        • 213.152.162.5
        • 134.19.179.211

        And see if they’re making repeated un-scoped (no page paramater) requests to /api/v3/comment/list. If they are, block them in your firewall.

        Those used to hit my instance constantly with requests like /api/v3/comment/list?sort=Old&page=16514 (yes, page 16,414). When I blocked those IPs making those requests, problem solved.

        • Salamander@mander.xyzM
          2·
          2 days ago

          Thanks!

          I don’t see those specific IPs, nor 16514. But now I see what scrapers tend to look like in the logs :)

          I am now pretty sure that the cause was scraper-like activity coming from the Mlmym front-end that I am serving over an onion site. I am not sure if it randomly started mis-behaving or if a tor scraper was using it.

          After blocking this, federation was restored, performance increased, and CPU use came down:

      • acockworkorange@mander.xyz
        3·
        2 days ago

        Did you upgrade Lemmy on the 13th or something? Could be a new bug.

        Also, did site traffic increase at all? Maybe there’s an AI scraper messing up.

        • Salamander@mander.xyzM
          6·
          2 days ago

          I did not update or change anything in the past few days.

          But, now that you mentioned an AI scrapper I looked into the logs and noticed some heavy requests to the API from a specific IP.

          Requests look consistent to scraping - just consistently and continuously issuing GET requests to different API endpoints.

          XX.XX.XX.XX - - [14/Oct/2025:21:28:54 +0000] "GET /api/v3/community/list?limit=20&sort=TopAll HTTP/2.0" 403 107 "-" "Mlmym"

          I have started denying their requests and it is the first thing that seems to have actually helped!

          I don’t want to speak too early but I think you may have identified the cause. Thanks!

          • Salamander@mander.xyzM
            5·
            2 days ago

            I just realized that it is not a ‘scraper’, the requests came from the server that I am using to provide an interface to the site as an Onion site. The amount of requests was suspiciously high so maybe a bot is scraping through Tor. I will leave it off for a few days and see if I can turn it back on later.

            • acockworkorange@mander.xyz
              3·
              2 days ago

              That’s a level of sophistication I hadn’t seen - using Tor to scrape. Hopefully they give up soon. I’m glad we got some respite. Thank you!

    • Bonus@mander.xyz
      1·
      2 days ago

      It’s loading way faster than it was this morning.

      and, I may have spoken too soon.

      Earlier today, it was hanging for some seconds with every click. Seemed faster for a moment but still laggy.

      • Salamander@mander.xyzM
        3·
        2 days ago

        Hmm, thanks!

        I just made another change. I am not sure why the database suddenly started taking up so many resources, so I have reduced the amount of resources available to it. It did reduce CPU use to 80% as opposed to 100%, but it is still somewhat laggy for me.