Automating IndexNow submissions with Vercel cron jobs

TL;DR

• IndexNow lets you notify participating search engines about changed URLs immediately.
• Automate submissions with a scheduled Vercel cron hitting a protected Next.js route.
• Submit URLs from your sitemap to https://api.indexnow.org/indexnow using your key and verification file.

Instead of waiting for search engine crawlers to discover new content, this site proactively notifies them using IndexNow. The entire process is automated via a Vercel cron job that runs every 6 hours.

How IndexNow works

IndexNow is a protocol that lets websites instantly notify search engines about content changes. When you submit a URL, the receiving engine (like Bing) automatically shares it with all other participating engines (Microsoft Bing, Naver, Seznam.cz, Yandex, and Yep). This means you only need to submit once.

The protocol requires three things:

1. An API key - A unique identifier (8-128 alphanumeric characters and dashes) that you generate
2. A verification file - A text file at /{your-key}.txt proving domain ownership
3. A submission endpoint - Where you POST your URLs (api.indexnow.org/indexnow)

The architecture

Here's how the pieces fit together:

Three components work together:

• Vercel cron job - Triggers the process on a schedule
• Next.js API route - Handles authentication and orchestrates the submission
• IndexNow utility - Fetches the sitemap and submits URLs with retry logic

Setting up the cron job

Vercel supports cron jobs via vercel.json. The configuration is minimal:

The schedule 0 */6 * * * means "at minute 0 of every 6th hour" - so the job runs at midnight, 6am, noon, and 6pm UTC.

Vercel automatically sends an Authorization: Bearer ${CRON_SECRET} header with cron requests, which we verify in the API route.

The API route

The cron endpoint at app/api/cron/indexnow/route.ts handles three responsibilities:

The maxDuration = 60 export allows the function up to 60 seconds to complete, accommodating potential retries.

The submission logic

The core utility in utils/indexnow.ts handles the actual submission. First, it fetches URLs from the sitemap:

Before submission, URLs are validated to ensure they belong to the correct domain and use HTTPS:

The submission itself follows the IndexNow API spec, with retry logic for transient failures:

The utility implements exponential backoff for rate limits (HTTP 429) and server errors, waiting 2, 4, then 8 seconds between retries (up to 3 attempts).

Domain verification

IndexNow requires proof of domain ownership via a text file. The file must be named {your-key}.txt and contain the key itself:

The keyLocation parameter in the API request tells IndexNow where to find this file. When placed at the root (Option 1 in IndexNow docs), all URLs on the domain can be submitted.

Environment variables

Three environment variables are required:

Testing locally

You can trigger a manual submission by calling the cron endpoint directly:

Or use the POST endpoint to submit specific URLs:

The POST endpoint at /api/indexnow accepts an optional urls array in the request body. If omitted, it fetches all URLs from the sitemap.