Shopify Web Bot Auth: Set Up Crawler Access (2026)

Your SEO crawl just returned 200 pages of 429 errors and zero useful data. Screaming Frog timed out. Sitebulb pulled back incomplete results. Your store looks fine in a browser, but your audit tools can't get past the front door. The fix is Shopify's Web Bot Auth — a crawler access system that authenticates your tools so Cloudflare lets them through.

Shopify routes all storefront traffic through Cloudflare's bot protection layer. Without authentication, your crawler looks identical to a scraper or an attack bot. Shopify blocks it accordingly. Setting up Web Bot Auth takes about five minutes.

Why Your SEO Crawler Gets Blocked on Shopify

Shopify uses Cloudflare to filter traffic before it reaches your store. Legitimate search engine bots like Googlebot and Bingbot are automatically whitelisted — they'll never have access issues. But third-party tools like Screaming Frog, Sitebulb, Ahrefs Site Audit, and custom scripts don't get the same treatment.

When these tools send dozens or hundreds of requests per second, Cloudflare flags them as suspicious. The result is a wall of 429 (Too Many Requests) or 403 (Forbidden) errors. Your crawl either fails outright or returns partial data that makes your audit useless.

Shopify also updated its robots.txt across all stores to explicitly restrict automated scraping and end-to-end buying flows that skip human review. AI shopping bots, price scrapers, and unauthorized crawlers are increasingly blocked by default. If you're running SEO audits or accessibility checks, your tools get caught in the same net.

What Is Shopify Web Bot Auth?

Shopify Web Bot Auth is an authentication system that lets merchants authorize specific automated tools to access their store. It generates cryptographic HTTP message signatures that your crawler includes in every request. When Shopify's Cloudflare layer sees those signatures, it recognizes the traffic as authorized and lets it through.

Each signature consists of three HTTP headers:

• Signature-Input — defines the signature parameters and algorithm
• Signature — the cryptographic signature itself
• Signature-Agent — identifies the requesting agent (typically set to "https://shopify.com")

You create these in your Shopify admin. Shopify generates all three values for you — no cryptography knowledge required. You copy them into your crawler's custom header settings, and your next audit runs clean.

How to Create a Crawler Access Signature

1. In your Shopify admin, go to Online Store → Preferences
2. Scroll to the Crawler access section
3. Click Create signature
4. Enter a descriptive name (e.g., "SEO Audit — Screaming Frog" or "Monthly Sitebulb Crawl")
5. Select the domain you want this signature to apply to — each signature works for one domain only
6. Set an expiration period — options range up to a maximum of 3 months
7. Click Create

Shopify generates your three header values immediately. Copy each one using the copy button next to it. Don't try to type these manually — they're long cryptographic strings, and a single wrong character breaks the authentication.

Store these values somewhere safe. You'll need them every time you set up a new crawl or configure a new tool.

Configure Screaming Frog With Web Bot Auth

Screaming Frog added native support for Shopify's Web Bot Auth signatures. Here's how to set it up:

1. Open Screaming Frog and go to Configuration → HTTP Header
2. Add a new header with the name Signature-Input and paste the corresponding value from Shopify
3. Add another header named Signature with its value
4. Add a third header named Signature-Agent with its value
5. Save the configuration and start your crawl

You should see clean 200 status codes instead of 429 errors. If you're still getting blocked, double-check that you copied all three values exactly — no trailing spaces, no missing characters.

Configure Sitebulb With Web Bot Auth

Sitebulb handles Web Bot Auth through its custom headers setting:

1. Start a new audit or edit an existing project
2. Go to Crawler Settings → Custom HTTP Headers
3. Add all three headers (Signature-Input, Signature, Signature-Agent) with their respective values
4. Save and run your audit

The same approach works for any crawler that supports custom HTTP headers — Jet Octopus, ContentKing, custom Python scripts using the requests library, or any tool where you can inject headers into outbound requests.

Which Bots Need Signatures and Which Don't

Not every bot needs manual authorization. Here's how it breaks down:

Automatically whitelisted (no signature needed):

• Googlebot
• Bingbot
• Other major search engine crawlers

Need a Web Bot Auth signature:

• SEO audit tools (Screaming Frog, Sitebulb, Ahrefs Site Audit)
• Accessibility checkers
• Custom monitoring scripts
• Performance testing tools
• Any automated tool you build or hire someone to run

Blocked by default with no override:

• Price scrapers
• Unauthorized AI shopping bots that attempt end-to-end purchases
• Content scrapers without proper identification

AI crawlers from companies like OpenAI and Anthropic fall into a grey area. Shopify's updated robots.txt restricts automated buying flows, but AI discovery crawlers (like those powering ChatGPT product recommendations) may still access product data depending on how Shopify's policies evolve. The Shopify GEO feature and Universal Commerce Protocol (UCP) are designed to make stores discoverable by AI — but on Shopify's terms, not through uncontrolled scraping.

Three Mistakes That Break Your Signature

Manually typing values instead of copying. The Signature and Signature-Input fields are long, complex strings. One wrong character and the authentication fails silently — you'll just get 429 errors with no explanation. Always use the copy button in Shopify admin.

Forgetting signatures expire. Every signature has a maximum lifespan of 3 months. When it expires, your crawls start failing again. Set a calendar reminder to regenerate before expiration. There's no auto-renewal — you create a new signature and update your tools manually each time.

Using one signature for multiple domains. Each signature is tied to a single domain. If your store has both a primary domain and a redirect domain, you need separate signatures for each one you want to crawl.

When to Run Authenticated Crawls

With a 3-month maximum signature lifespan, plan your auditing schedule around it:

• After any major theme change — check for broken internal links, missing meta tags, and crawlability issues
• Monthly technical audits — catch 404s from deleted products, orphaned pages, and redirect chains before they compound
• Before seasonal campaigns — verify that landing pages, collection pages, and product pages are all indexable and loading correctly
• After bulk product uploads or deletions — confirm that new pages are crawlable and old ones redirect properly

A single authenticated crawl can surface problems that Google Search Console won't show you for weeks. Broken internal links, duplicate content issues, thin content pages, missing canonical tags — these problems are invisible until you audit, and they silently erode your rankings every day they go unfixed.

If you haven't crawled your Shopify store in 2026, there's a good chance your tools were getting blocked and you didn't realize it. Go to Online Store → Preferences → Crawler access, create your first signature, and run an audit this week. Five minutes of setup buys you months of clean SEO data.