JWT Auth Started Failing in Production — It Was Supabase Key Rotation

The Slack channel was completely quiet until 9:02 AM on a Monday. Then the alerts hit: "Protected dashboard pages are throwing 401 Unauthorized errors." A minute later, a user reported getting kicked out to the login screen immediately after uploading a file.

On my screen, the server terminal was printing hundreds of lines of auth exceptions: JWT Signature verification failed: Signature verification failed.

I checked the client's token generation logic. No changes. I checked the frontend cookie storage settings. Perfect. I checked the backend middleware to see if an update had broken our token decoding scripts. Nothing had changed in our code repository in 6 days.

It took me two hours of high-stress digging, network log tracing, and JWT decoding to find the root cause. The culprit wasn't our application code at all. It was a silent Supabase JWT Secret rotation that instantly made our backend's environment variables stale.

Here is exactly how I diagnosed the issue, fixed the auth loop, and implemented guards to prevent it from ever happening again.

🔬 The Initial Suspects: Where I Looked First

When a JWT-based authentication system suddenly starts throwing 401 errors in production across all requests, you usually look for these standard failures:

1. Token Expiration (TTL): I checked the expiration claims (exp) inside the active tokens. The tokens were freshly issued and well within their active window.
2. CORS and Cookie Policies: I verified that the Authorization: Bearer <token> header was still attaching correctly to outbound HTTP calls. The payloads were completely intact.
3. Database Connectivity: I checked if the auth database had locked up or run out of connection pools. The Postgres database was completely responsive.

🔍 The Discovery: The Stale Signature Verification Key

To understand the actual failure, you have to look at how our FastAPI Python backend verifies incoming tokens issued by Supabase Auth.

To avoid making a slow network request to Supabase on every single API request, the backend decodes and verifies the signatures of the JWTs locally using a shared SUPABASE_JWT_SECRET key:

# backend/utils/auth.py
import jwt
import os
from fastapi import HTTPException, Security
from fastapi.security import HTTPBearer

security = HTTPBearer()
JWT_SECRET = os.getenv("SUPABASE_JWT_SECRET")

def verify_supabase_token(credentials = Security(security)):
    token = credentials.credentials
    try:
        # Decode and verify token signature locally using the shared secret
        payload = jwt.decode(
            token, 
            JWT_SECRET, 
            algorithms=["HS256"], 
            options={"verify_aud": False}
        )
        return payload
    except jwt.ExpiredSignatureError:
        raise HTTPException(status_code=401, detail="Token has expired")
    except jwt.InvalidSignatureError:
        # This exception was silently catching all valid tokens in production!
        raise HTTPException(status_code=401, detail="Invalid auth signature")
    except jwt.PyJWTError as e:
        raise HTTPException(status_code=401, detail=f"Authentication failed: {str(e)}")

I copied a raw token from one of the active browser sessions and pasted it into jwt.io. The token header was correct, and it listed the signature algorithm as HS256.

Then, I manually ran a local script using Python to decode the token using the SUPABASE_JWT_SECRET stored in our production environment variables.

It failed with a jwt.exceptions.InvalidSignatureError.

This meant only one thing: the JWT was signed with a completely different secret than the one we had stored in our environment configuration. The keys had split.

💡 The Cause: Silent Supabase Key Rotation

After searching through the Supabase status pages and project logs, I discovered the cause. The client had initiated a database restoration/reset from their Supabase dashboard during the weekend to recover some mistakenly deleted mock assets.

What they didn't realize is that resetting, pausing, or changing key settings on a Supabase project automatically triggers a JWT Secret rotation.

Supabase instantly updated its internal authentication server with a brand-new cryptographically random secret to sign outgoing user tokens. Because our FastAPI backend ran in an isolated Render container, our .env configuration file was still using the stale, old SUPABASE_JWT_SECRET key.

Every single incoming token was treated as forged because the old secret couldn't verify the new signature!

🛠️ The 5-Minute Fix

Once identified, resolving the issue was simple.

Step 1: Copying the New JWT Secret

I logged into the Supabase Dashboard, selected our project, and navigated to Project Settings → API.

Under JWT Settings, I located the JWT Secret field, unrevealed the hash, and copied the new string.

Step 2: Updating Production Environment Variables

I logged into the Render dashboard (where our FastAPI backend was running), navigated to the Environment Variables settings panel for our web service, and updated the old value of SUPABASE_JWT_SECRET with our newly copied key:

SUPABASE_JWT_SECRET=new_cryptographically_secure_rotated_secret_here...

I clicked "Save Changes." Render automatically initiated a rolling update, spinning up a new container with the fresh environment variables and terminating the old instance with zero downtime.

Within 45 seconds, all dashboard authentication routes went from throwing 401s to responding with clean 200 OK payloads. The system was completely healed.

🛡️ Prevention Checklist for Future Deploys

To guarantee that a secret rotation never causes production downtime again, I updated our deployment configuration and developer documentation:

1. Keep .env.example in absolute sync: Added clear warnings inside the template file so any new developer is aware that Pausing/Resuming a Supabase project resets the token values:

   # CRITICAL: Retrieve this from Supabase Dashboard -> Settings -> API.
   # Note: Pausing or restoring your project rotates this secret automatically!
   SUPABASE_JWT_SECRET=your_supabase_jwt_secret
   

2. Setup alert channels: Connected a Slack webhook directly to the Supabase project notifications channel so that any critical setting updates, restorations, or key changes alert the engineering team instantly.

💡 Key Takeaway

Treat your database and auth provider secrets like dynamic variables, not static constants. Supabase, Firebase, and Auth0 can and will rotate keys—either due to security policies, database pauses, or developer actions. Never hardcode keys, keep your production environment variables isolated, and verify signature match states immediately when 401 exceptions start showing up across all authenticated endpoints!