RustFS console behind nginx: broken login and 429 errors, fixed

I put the RustFS web console behind nginx on its own subdomain and added two ordinary things: a redirect from / to the console path, and a rate limit. Both broke the console.

The redirect broke sign-in.
The rate limit returned 429s right after a successful login.

Both fixes are an nginx map. The working config is below, with the reason for each part.

The setup

RustFS runs in Docker with two ports bound to localhost: 9000 for the S3 API and 9001 for the console. nginx serves the console at s3-console.example.com and proxies to 127.0.0.1:9001.

Two facts about the console explain everything that follows:

The console’s real path is /rustfs/console/. Opening the bare address does not land you on it.
Port 9001 serves the console pages, the admin API and S3. The console calls all of them on its own origin. That is convenient, because it needs no CORS setup. It also means the root path / on the console host is a real API endpoint, not just a landing page.

I wanted https://s3-console.example.com/ to open the console, so I added a plain location = / block that redirected to /rustfs/console/.

After that, I could not sign in.

The cause: the console signs in with POST /, and it lists buckets with a signed GET /. Those are requests to exactly the path I was redirecting. A location = / block matches on the path alone. It does not care about the method or the headers, so it caught the console’s own API calls and sent them a redirect to an HTML page instead of passing them to RustFS.

The fix: redirect only a browser opening the bare address

The redirect should apply to one case only: an unsigned GET / with no query string. That is what a browser sends when someone types the address. A map can express that by joining four request properties into one string:

# Only a browser opening the bare console address: the console itself signs in with POST / and lists buckets
# with a signed GET /, so those must reach RustFS
map "$request_method:$uri:$args:$http_authorization" $s3_console_home {
    default 0;
    "GET:/::" 1;
}

The string is method:path:query:authorization. The only value that maps to 1 is GET:/::, which means:

method is GET
path is exactly /
the query string is empty
there is no Authorization header

A POST / sign-in produces a string starting with POST:, so it does not match. A signed GET / has a non-empty Authorization value at the end, so it does not match either. A request with query arguments fails on the third field.

The server block then uses the variable:

server {
    listen 80;
    listen [::]:80;
    server_name s3-console.example.com;
    client_max_body_size 0;
    if ($s3_console_home) {
        return 302 /rustfs/console/;
    }
    location / {
        # proxy to 127.0.0.1:9001, shown below
    }
}

The if sits at server level and does nothing but return. Everything that is not a bare browser visit falls through to location / and reaches RustFS.

The console checks the login itself, so the only brute-force protection I can add in front is a rate limit. I put a limit_req on the console site, keyed on the client IP.

Sign-in then worked, and the console immediately started getting 429 responses.

The cause: right after login the console preloads its own page files. It is about 20 pages, all requested in a burst from one IP. A blanket limit counts those the same as login attempts, so a normal login looked like an attack.

The fix: an empty key for the console’s own files

The requests I want to limit are the ones that check the key: sign-in, the admin API and S3. The static page files under /rustfs/console/ do not need limiting.

nginx does not count a request against a limit_req_zone when the zone’s key is empty. So the key itself can be a mapped variable:

# Rate-limit only requests that check the key (sign-in, admin API, S3); the console's own page files
# are left out, since it preloads dozens of them right after login
map $uri $s3_console_limit_key {
    ~^/rustfs/console/ "";
    default $binary_remote_addr;
}
limit_req_zone $s3_console_limit_key zone=s3_console_api:1m rate=10r/s;

For any path under /rustfs/console/ the key is an empty string and the request is not counted. For everything else the key is the client address, as usual.

The zone is applied in the proxy location:

    location / {
        limit_req zone=s3_console_api burst=100 nodelay;
        limit_req_status 429;
        proxy_pass http://127.0.0.1:9001;
        proxy_http_version 1.1;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_buffering off;
        proxy_request_buffering off;
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
    }

The limit is 10 requests per second per IP with a burst of 100. That leaves room for normal console use and still slows down password guessing.

Other details in that proxy block

Host $http_host. S3 request signatures include the Host header. nginx must pass it through unchanged, or signed requests fail. This applies to the console site too, because the console sends signed requests to its own host.
client_max_body_size 0 and request buffering off. Uploads from the console stream straight to RustFS with no size limit.
The limit is the main defence if the password is weak. I changed the console login from the generated secret to a short one I chose. With a short password, this rate limit is what stands between the login and a guessing script. A long random secret would be safer.

One more gotcha: certbot owns the site file

My deploy script writes the nginx site once and then lets certbot add HTTPS to it. Because certbot edits that file, the script never rewrites it:

# Nginx site (HTTP for now; certbot adds HTTPS once DNS points here, so it's only written once)
if [ ! -f /etc/nginx/sites-available/s3 ]; then

This matters when fixing problems like the two above. Changing the config in the script and re-running it changes nothing on the server, because the file already exists. Fixes to a live site go into the live file, followed by nginx -t and a reload. Deleting the file and re-running the script also works, but it drops HTTPS until certbot runs again.

How to check the redirect rule

The script already probes the console path locally:

echo "== console local: HTTP $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9001/rustfs/console/)"

For the redirect logic, the useful test is the real one: open the bare address in a browser, confirm it lands on /rustfs/console/, then sign in and open the bucket list. If sign-in fails with the redirect in place, the redirect is catching API calls.

The general lesson

Both bugs came from treating the console like a static site with a login page. It is a single-page app whose API lives on the same host, including on /. Rules that match on the path alone will hit API calls too.

When a proxy rule should apply only to “a person opening this URL”, match on what makes that request different: the method, the missing query string, the missing Authorization header. And when a rate limit should protect a login, key it so that static files are not counted.

Reference: the nginx limit_req module docs describe the empty-key behaviour.