Does anyone know of any off the shelf tool (online or offline) to find duplicates in several DNS blocklists and merge them into one?

Context: I am running AdGuard on one GL.iNet router with ~10 blocklists some of them pretty huge and most of the times the lists are updated the router comes to one halt while doing so, having to often times reboot it through the old power-off-and-on.

I would rather download the lists myself from time to time and merge them into one file but with duplicates extracted somehow.

  • easeKItMAn@lemmy.world
    link
    fedilink
    English
    arrow-up
    7
    ·
    edit-2
    1 year ago

    If I’m understanding you correctly, you could make use of a shell script for this. Use WGET to download lists, then combine them into a single large file, and finally create a new file with no duplicates by using “awk ‘!visited[$0]++’”

    wget URL1 URL2 URL3
    cat *.txt > all.txt (This overwrites all.txt)
    awk ‘!visited[$0]++’ all.txt > no_duplicates.txt

    • BinaryUnit@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      1 year ago

      When no tool is available bash to the rescue, thank you for this it seems actually simpler then I thought :)