Hi! I would like to host a transparent proxy for cache.nixos.org on my local kubernetes cluster.

I took the following NGINX config https://nixos.wiki/wiki/FAQ/Private_Cache_Proxy and created all the folders on the mounted storage.

This is the kubernetes deployment:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nix-cache-volume
spec:
  capacity:
    storage: 500Gi
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/k8s/nix-cache" # Needs exists before PV is created!
  persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nix-cache-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: manual
  resources:
    requests:
      storage: 500Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nix-cache
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nix-cache
  template:
    metadata:
      labels:
        app: nix-cache
        name: nix-cache
    spec:
      volumes:
        - name: nix-cache-storage
          persistentVolumeClaim:
            claimName: nix-cache-pvc
        - name: nix-cache-config
          configMap:
            name: nix-cache-config
      containers:
        - name: nix-cache
          image: nginx:1.27.0 
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nix-cache-storage
              mountPath: /data
            - name: nix-cache-config
              mountPath: /etc/nginx/sites-available/default
          resources:
            limits:
              memory: "512Mi"
              cpu: "300m"
            requests:
              memory: "256Mi"
              cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
  name: nix-cache
spec:
  selector:
    app: nix-cache
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nix-cache-ingress
  annotations:
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  rules:
    - host: "nix-cache.raspi.home"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: nix-cache
                port:
                  number: 80
  tls:
    - secretName: nix-cache-raspi-home-tls
      hosts:
        - "nix-cache.raspi.home"
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nix-cache.raspi.home
spec:
  commonName: nix-cache.raspi.home
  dnsNames:
    - "nix-cache.raspi.home"
  secretName: nix-cache-raspi-home-tls
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nix-cache-config
data:
  nginx.conf: |
    server {
      listen 80;
      server_name nix-cache.raspi.home;

      location ~ ^/nix-cache-info {
        proxy_store        on;
        proxy_store_access user:rw group:rw all:r;
        proxy_temp_path    /data/nginx/nix-cache-info/temp;
        root               /data/nginx/nix-cache-info/store;

        proxy_set_header Host "cache.nixos.org";
        proxy_pass https://cache.nixos.org;
      }

      location ~^/nar/.+$ {
        proxy_store        on;
        proxy_store_access user:rw group:rw all:r;
        proxy_temp_path    /data/nginx/nar/temp;
        root               /data/nginx/nar/store;

        proxy_set_header Host "cache.nixos.org";
        proxy_pass https://cache.nixos.org;
      }
    }

To use the cache I added it to the substituters.

  nix.settings.substituters = [
    "https://nix-cache.raspi.home/"
  ];

But when I try to use it, get the error:

# Trigger a download
nix develop nixpkgs#just
# Error message
warning: 'https://nix-cache.raspi.home' does not appear to be a binary cache

In the logs of the NGINX I see the following error:

2024/08/03 12:09:30 [error] 31#31: *3 open() "/usr/share/nginx/html/nix-cache-info" failed (2: No such file or directory), client: 10.42.2.7, server: localhost, request: "GET /nix-cache-info HTTP/1 │
│ 10.42.2.7 - - [03/Aug/2024:12:09:30 +0000] "GET /nix-cache-info HTTP/1.1" 404 153 "-" "curl/8.8.0 Nix/2.18.5" "10.42.2.1"                                                                             │
│ 10.42.2.7 - - [03/Aug/2024:12:09:30 +0000] "PUT /nix-cache-info HTTP/1.1" 405 157 "-" "curl/8.8.0 Nix/2.18.5" "10.42.2.1"    

Any ideas whats wrong? I’m neither an nix nor an nginx expert, so maybe it is something really simple but I cannot figure it out.

  • onlinepersona@programming.dev
    link
    fedilink
    English
    arrow-up
    2
    arrow-down
    3
    ·
    3 months ago

    /usr/share/nginx/html/nix-cache-info

    That indicates your nginx is probably still using the default config.

    This is the default config in nginx.conf

    user  nginx;
    worker_processes  auto;
    
    error_log  /var/log/nginx/error.log notice;
    pid        /var/run/nginx.pid;
    
    
    events {
        worker_connections  1024;
    }
    
    
    http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;
    
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
    
        access_log  /var/log/nginx/access.log  main;
    
        sendfile        on;
        #tcp_nopush     on;
    
        keepalive_timeout  65;
    
        #gzip  on;
    
        include /etc/nginx/conf.d/*.conf;
    }
    

    Reading the dockerhub page for your image it states

    Customize configuration

    You can mount your configuration file, or build a new image with it.

    If you wish to adapt the default configuration, use something like the following to get it from a running nginx container:

    $ docker run --rm --entrypoint=cat nginx /etc/nginx/nginx.conf > /host/path/nginx.conf

    And then edit /host/path/nginx.conf in your host file system.

    Basically

    volumeMounts:
                - name: nix-cache-storage
                  mountPath: /data
                - name: nix-cache-config
                  mountPath: /etc/nginx/sites-available/default # this should be /etc/nginx/nginx.conf
    

    You might want to take some configuration options from the default config like the user, worker connections, etc.

    Good luck.

    Anti Commercial-AI license

    • secana@programming.devOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      3 months ago

      Thanks for the response. You are right, the config was at the wrong path. Unfortunately, the config itself does not work, too. After a bit of testing around this worked for me:

      apiVersion: v1
      kind: PersistentVolume
      metadata:
        name: nix-cache-volume
      spec:
        capacity:
          storage: 500Gi
        storageClassName: manual
        accessModes:
          - ReadWriteOnce
        hostPath:
          path: "/mnt/k8s/nix-cache" # Needs exists before PV is created!
        persistentVolumeReclaimPolicy: Retain
      ---
      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: nix-cache-pvc
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: manual
        resources:
          requests:
            storage: 500Gi
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: nix-cache
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: nix-cache
        template:
          metadata:
            labels:
              app: nix-cache
              name: nix-cache
          spec:
            volumes:
              - name: nix-cache-storage
                persistentVolumeClaim:
                  claimName: nix-cache-pvc
              - name: nix-cache-config
                configMap:
                  name: nix-cache-config
            containers:
              - name: nix-cache
                image: nginx:1.27.0 
                ports:
                  - containerPort: 80
                volumeMounts:
                  - name: nix-cache-storage
                    mountPath: /data
                  - name: nix-cache-config
                    mountPath: /etc/nginx/nginx.conf
                    subPath: nginx.conf
                    readOnly: true
                resources:
                  limits:
                    memory: "512Mi"
                    cpu: "300m"
                  requests:
                    memory: "256Mi"
                    cpu: "200m"
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: nix-cache
      spec:
        selector:
          app: nix-cache
        ports:
          - protocol: TCP
            port: 80
            targetPort: 80
      ---
      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: nix-cache-ingress
        annotations:
          traefik.ingress.kubernetes.io/router.tls: "true"
      spec:
        rules:
          - host: "nix-cache.raspi.home"
            http:
              paths:
                - pathType: Prefix
                  path: "/"
                  backend:
                    service:
                      name: nix-cache
                      port:
                        number: 80
        tls:
          - secretName: nix-cache-raspi-home-tls
            hosts:
              - "nix-cache.raspi.home"
      ---
      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: nix-cache.raspi.home
      spec:
        commonName: nix-cache.raspi.home
        dnsNames:
          - "nix-cache.raspi.home"
        secretName: nix-cache-raspi-home-tls
        issuerRef:
          name: ca-issuer
          kind: ClusterIssuer
      ---
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: nix-cache-config
      data:
        # See: https://www.channable.com/tech/setting-up-a-private-nix-cache-for-fun-and-profit
        nginx.conf: |
          events {
              worker_connections 1024;
          }
          http {
            proxy_cache_path /data/nginx/cache max_size=500G keys_zone=cache_zone:50m inactive=365d;
            proxy_cache cache_zone;
            proxy_cache_valid 200 365d;
            proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_504 http_403 http_404 http_429;
            proxy_ignore_headers X-Accel-Expires Expires Cache-Control Set-Cookie;
            proxy_cache_lock on;
      
            server {
                listen 80;
      
                server_name nix-cache.raspi.home;
      
                location /nix-cache-info {
                    return 200 "StoreDir: /nix/store\nWantMassQuery: 1\nPriority: 41\n";
                }
      
                location / {
                    proxy_set_header Host $proxy_host;
                    proxy_pass https://cache.nixos.org;
                }
            }
          }
      
      

      The config is an adaption from this blog post: https://www.channable.com/tech/setting-up-a-private-nix-cache-for-fun-and-profit