I am using the official Nginx image with modified config to cache origin responses. The configuration below is includeed in the http block inside the /etc/nginx/nginx.conf (The main config file for Nginx)

proxy_cache_path /var/nginx-cache
                 levels=1:2               # Directory levels when persisting cache (this is faster than one level)
                 keys_zone=fe_cache:500m  # Memory zone and its space to speedup cache (800 keys/1m)
                 max_size=10g             # All disk space if not set
                 inactive=30d             # How long to keep things without being accessed
                 use_temp_path=off;       # Don't know what is it, but it is recommended as off

server {
    listen 80;
    server_name _;
    set_real_ip_from  0.0.0.0/0;
    real_ip_header    X-Forwarded-For;
    real_ip_recursive on;

    location / {
        # Proxy_pass configuration
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_max_temp_file_size 0;
        proxy_pass http://my.origin/;

        # The cache zone we are using
        proxy_cache fe_cache;
        proxy_cache_min_uses 1;
        proxy_cache_use_stale error timeout updating http_500 http_502
                              http_503 http_504;
        #proxy_cache_background_update on;
        proxy_cache_lock on;

        # Add this header to the response
        add_header X-Cache-Status $upstream_cache_status;

        # If request contains token cookie, then don't save it to cache
        proxy_no_cache $cookie_token;

        # If request contains token cookie, then don't take it from cache
        proxy_cache_bypass $cookie_token;

        proxy_redirect off;
        proxy_read_timeout 240s;
    }
}

This Nginx image is running as a Kubernetes pod that has multiple replicas. To enhance performance, I'm trying to have the same cache volume shared between these replicas. But the problem I'm facing is that the cached content is getting deleted after some time. I noticed this by monitoring the output of this script (the number printed here decreases sometime)

while true; do
    kubectl exec -it nginx-pod -- bash -c '
        find /var/nginx-cache/ -type f | wc -l
    ' >> counts
done

To make sure that I don't have some kind of race condition, I tried reducing the total number of replicas to 1. However, I still got the same problem.

Finally, I tried to store data directly in the pod without mounting the volume. When I did this, the data was not getting deleted.