[SEO / Indexing] Google Search Console - Page is not indexed: Redirect error

[Karelaking]

Issue: [SEO / Indexing] Google Search Console - "Page is not indexed: Redirect error"

Issue Identifier: ISSUE-2026-SEO-001
Status: RESOLVED (Pending Production Deployment)
Severity: High (Impacts Search Engine Indexing & Organic Search Visibility)
Affected Service: Web Application / Next.js 16 Front-end (<your-app-domain>.com)

---

1. Executive Summary

Google Search Console reported a critical indexing failure for the web application: "Page is not indexed: Redirect error" (Page fetch: Failed: Redirect error).

When Googlebot Smartphone attempted to crawl the site, it encountered HTTP redirection failures, preventing Google from fetching and indexing the domain's landing page and public routes.

---

2. Evidence & Inspection Metrics

Google Search Console URL Inspection Report

• Indexing Status: URL is not on Google (Page is not indexed)
• Primary Error: Page is not indexed: Redirect error
• Page Fetch Status: Failed: Redirect error
• Crawler Agent: Googlebot smartphone
• Crawl Authorization (Crawl allowed?): Yes
• Discovery:
  • Sitemaps: No referring sitemaps detected
  • Referring page: None detected
• Canonical Selection:
  • User-declared canonical: N/A
  • Google-selected canonical: N/A

---

3. Technical Root Cause Analysis

Detailed investigation of the codebase and hosting configuration identified two primary root causes contributing to the redirect error:

A. Hosting Provider Edge Router vs. Next.js Router Conflict (vercel.json)

• Mechanism: The repository's vercel.json contained:

  "cleanUrls": true,
  "trailingSlash": false

• Failure Mode: When deployed, cleanUrls: true and trailingSlash: false instruct Edge Infrastructure to intercept incoming HTTP requests and issue forced 308 Permanent Redirect responses prior to passing the request to the Next.js runtime.
• Conflict: Next.js has built-in routing logic for trailing slashes and clean URLs. When the edge layer and Next.js's internal router both process trailing slash rules, they generate competing, recursive redirect chains (e.g. https://<your-app-domain>.com/ $\leftrightarrow$ https://<your-app-domain>.com). Googlebot aborts crawl requests after exceeding 5 redirect hops, triggering the Redirect error status.

B. Next.js 16 Proxy Convention & Authentication Bypass (proxy.ts)

• Mechanism: Next.js 16 deprecated middleware.ts in favor of the network boundary convention proxy.ts.
• Failure Mode:
  1. Unauthenticated crawlers (such as Googlebot) hitting protected routes were redirected to authentication (/sign-in?redirect_url=...), causing multi-hop 302/307 redirect chains.
  2. Public routes (such as /, /sitemap.xml, /robots.txt, /cookies, /og-image.png, and /favicon.ico) needed explicit route matching to guarantee an immediate 200 OK HTTP response without auth challenges.

---

4. Remediation & Code Changes

Step 1: Cleaned Up vercel.json Edge Routing Rules

Removed "cleanUrls": true and "trailingSlash": false from vercel.json to allow Next.js to manage URL normalization natively.

File: vercel.json

 {
   "$schema": "https://openapi.vercel.sh/vercel.json",
   "framework": "nextjs",
-  "cleanUrls": true,
-  "trailingSlash": false,
   "buildCommand": "pnpm build",
   "installCommand": "pnpm install",

Step 2: Implemented Next.js 16 Proxy with Public Route Safeguards

Standardized proxy.ts using Next.js 16 proxy conventions with authentication middleware. Public endpoints are matched via createRouteMatcher so crawlers receive direct 200 OK responses.

File: proxy.ts

import { clerkMiddleware, createRouteMatcher } from "@clerk/nextjs/server";

/**
 * Route matcher defining public routes that do not require authentication.
 * Ensures search engine crawlers (Googlebot) and public visitors access landing,
 * legal, auth, sitemap, and robots pages without authentication redirect loops.
 */
const isPublicRoute = createRouteMatcher([
    "/",
    "/cookies(.*)",
    "/sign-in(.*)",
    "/sign-up(.*)",
    "/api/(.*)",
    "/sitemap.xml",
    "/robots.txt",
    "/favicon.ico",
    "/og-image.png",
]);

/**
 * Next.js 16 Proxy enforcing authentication on protected routes without redirect loops.
 */
export const proxy = clerkMiddleware(async (auth: any, req: any) => {
    if (!isPublicRoute(req)) {
        await auth.protect();
    }
});

export default proxy;

export const config = {
    matcher: [
        "/((?!_next|[^?]*\\.(?:html?|css|js(?!on)|json|png|jpg|jpeg|webp|gif|svg|ttf|woff2?|ico|csv|docx?|xlsx?|zip|webmanifest)).*)",
        "/(api|trpc)(.*)",
    ],
};

Step 3: Configured Middleware Alias Compatibility

Updated middleware.ts to re-export proxy.ts so any legacy tool or backward-compatible environment safely resolves to proxy.ts.

---

5. Verification & Testing Protocol

To verify that the redirect error is completely eliminated:

1. HTTP Redirect Chain Check:
   Run a curl trace on the domain:

   curl -IL "https://<your-app-domain>.com/"

   Expected Result: A single HTTP/2 200 OK response (or maximum 1 hop from HTTP to HTTPS) without circular 308 or 302 redirects.

2. Robots & Sitemap Verification:
   Verify https://<your-app-domain>.com/robots.txt and https://<your-app-domain>.com/sitemap.xml return HTTP 200 OK with valid XML/plain text content.

3. Google Search Console Re-indexing:

   • Deploy code updates to production environment.
   • Open Google Search Console $\rightarrow$ URL Inspection tool.
   • Click "TEST LIVE URL" to confirm Page fetch: Successful.
   • Click "REQUEST INDEXING".

---

6. Action Items Checklist

• Remove cleanUrls and trailingSlash from vercel.json
• Configure Next.js 16 proxy.ts with public route exemptions
• Re-export proxy.ts from middleware.ts
• Push changes to git repository (main branch)
• Verify deployment build success
• Perform curl -IL live response header validation
• Submit REQUEST INDEXING in Google Search Console