Authorizing Custom Crawlers on Shopify with Web Bot Auth: Implementation and Importance

Implementing Shopify’s Crawler Access Signatures (Web Bot Auth)

What Are Crawler Access Signatures? Shopify recently introduced Web Bot Auth (crawler access signatures) to let merchants securely authorize automated crawlers, scripts, and tools to access their storefront without being blocked changelog.shopify.com. This system leverages HTTP Message Signatures – essentially cryptographic headers – to verify that each request comes from an authorized source help.shopify.com. When you create a signature in Shopify Admin, you get two unique cryptographic tokens (a "Signature-Input" and a "Signature" value) and are instructed to include them, along with a fixed agent identifier, on every HTTP request your tool makes help.shopify.com. Shopify (and its underlying Cloudflare protection) recognizes these headers and treats the traffic as verified rather than malicious, ensuring your tool isn’t rate-limited or blocked changelog.shopify.comhelp.shopify.com.

Key Features of Shopify Signatures: Shopify’s crawler access keys come with a number of built-in features designed for security and manageability help.shopify.com:

• Cryptographically Secure: The signatures are generated using secure cryptographic methods (aligned with HTTP message signing standards) help.shopify.com. In practice, this means each request carries a digital signature that Shopify/Cloudflare can verify as coming from you. (Cloudflare’s Verified Bot system uses Ed25519 public–private key pairs and HTTP signatures under the hood to prevent spoofing searchenginejournal.com, so you can trust that fake bots can’t simply guess your signature.)

• Expiration Control: You set an expiration for each signature, up to a maximum of 3 months help.shopify.com. Once a signature expires, it automatically becomes invalid. This time limit mitigates risk (you wouldn’t want a lost token to work forever) and means you’ll periodically rotate credentials.

• Custom Names: You assign a descriptive name to each signature for easy identification help.shopify.com. For example, if you run multiple crawlers (SEO audits, accessibility checkers, etc.), you might name one “SEO ScreamingFrog Q4” and another “Perf Monitoring Bot”.

• Domain-Scoped: Each signature is tied to one specific domain (one of your Shopify store’s connected domains) help.shopify.com. This scoping means the signed requests are only valid for that domain’s storefront – an extra safeguard so you can’t accidentally use a token on the wrong site.

• Central Management: All your active signatures can be viewed and managed in one place in the Shopify Admin. You can see the name, the associated domain, the first few characters of the tokens, the agent, and the expiration date/status help.shopify.com.

Creating a Signature in Shopify Admin: To use this feature, you’ll generate a signature pair via your Shopify admin. Follow these steps:

1. Navigate to Preferences: In your Shopify admin, go to Online Store → Preferences → Signatures (at the bottom of the page) help.shopify.com.

2. Create Signature: Click “Create signature.” In the form that appears, fill in:

   • Name: A clear label (e.g. the name of your crawler or tool).

   • Domain: Select which connected domain this signature will authorize help.shopify.com.

   • Valid for: Choose an expiration period (1 week, 1 month, 3 months, etc.). Remember, the max is 90 days help.shopify.com.

3. Save and Copy Values: Click Create. Shopify will generate the signature and display two cryptographic strings: Signature-Input and Signature. Use the “Copy” button next to each to grab their values help.shopify.com. (You won’t be able to view the full values again later, for security reasons – so copy and store them securely now help.shopify.com!)

You’ve now got your keys. In summary, Shopify did the heavy lifting by generating a public/private key pair and an HTTP message signature for you behind the scenes. From your perspective, you just need to take those provided header values and use them in your crawler’s requests.

Using the Signature in Your HTTP Requests: To present your crawler as authorized, every HTTP request it sends to your storefront must include three specific HTTP headers:

• Signature-Input: set this header’s value to the Signature-Input string you copied.

• Signature: set this to the Signature string you copied.

• Signature-Agent: set this to the exact string "https://shopify.com" (including the quotes in the header value) help.shopify.com.

For example, a raw HTTP request might look like:

Shopify’s servers (and Cloudflare’s bot management in front of Shopify) will see these headers and verify the signature against the public key on file for your token, confirming your bot’s identity developers.cloudflare.com. This tells them “this is a trusted bot belonging to the store owner, let it through.” As long as the headers are present and valid on each request, your crawler will be treated as an authorized agent and should not be blocked or throttled by bot protection measures changelog.shopify.com. (If you ever see HTTP 429 rate-limit errors or unexpected CAPTCHA challenges, it usually means the signature was missing, expired, or incorrect on those requests help.shopify.com.)

Most SEO auditing tools allow custom headers. Shopify’s documentation links to guides for popular tools like Screaming Frog, JetOctopus, Sitebulb, OnCrawl, etc., which show how to include custom HTTP headers in their crawl configurations help.shopify.com. In those tools, you’ll typically find a setting to add HTTP headers to each request – simply add the three headers above with the values from Shopify. After that, you can run the crawl as usual, and Shopify will serve your store’s pages to the tool just like it would to a normal browser (no more blocked requests or incomplete data).

Tools Without Custom Header Support – Proxy Workaround: What if you use a platform that doesn’t let you set custom headers for crawling? For example, some online SEO services or older tools might not offer this option. In such cases, a workaround is to insert a proxy between the tool and your Shopify store. The idea is: your tool hits the proxy (with no special headers), and the proxy forwards the request to Shopify with the signature headers injected.

Two developer-friendly ways to do this are using a Vercel serverless function or a Cloudflare Worker:

• Vercel Serverless Function (Node.js): You can deploy a simple Node.js function on Vercel that forwards requests. In the code, you’ll add the Signature-Input, Signature, and Signature-Agent headers to whatever incoming request it receives before fetching your real Shopify store URL. For example:

  
  

  // Vercel Serverless API Route (Node.js) import fetch from 'node-fetch'; export default async function handler(req, res) { const targetUrl = 'https://YOUR-SHOP-DOMAIN.myshopify.com' + req.url; const response = await fetch(targetUrl, { headers: { ...req.headers, // forward all original headers 'Signature-Input': process.env.SF_INPUT, // use env vars to store your secrets 'Signature': process.env.SF_SIGNATURE, 'Signature-Agent': '"https://shopify.com"' } }); // Relay the response back to the client res.status(response.status); response.headers.forEach((value, key) => res.setHeader(key, value)); const text = await response.text(); res.send(text); }

  In this example, you’d set your actual Signature-Input and Signature strings as environment variables (SF_INPUT and SF_SIGNATURE) in Vercel, instead of hardcoding them (treat them like passwords!). The proxy simply takes any request, adds the necessary auth headers, and fetches the content from your Shopify storefront. To use it, point your crawler to the Vercel function’s URL instead of your actual store URL.

• Cloudflare Worker: If you’re comfortable with Cloudflare Workers, you can achieve a similar result at the edge. A Cloudflare Worker script can intercept requests and rewrite them with the required headers:

  
  

  addEventListener('fetch', event => { const { request } = event; // Construct new URL pointing to your Shopify domain const url = new URL(request.url); url.hostname = 'YOUR-SHOP-DOMAIN.myshopify.com'; // Create a new request with the additional headers const modifiedRequest = new Request(url, { method: request.method, headers: { ...request.headers, 'Signature-Input': SIGNATURE_INPUT, // your signature input value 'Signature': SIGNATURE, // your signature value 'Signature-Agent': '"https://shopify.com"' } }); event.respondWith(fetch(modifiedRequest)); });

  Here, SIGNATURE_INPUT and SIGNATURE would be secrets you store via Cloudflare’s interface. You’d then configure your tool to crawl via the Worker’s public URL. The Worker will call your Shopify store with the proper headers and return the content to the tool.

Using a proxy does add complexity and a bit of latency, but it’s a robust solution if you’re locked into a crawler that doesn’t support custom headers. In either case, the principle is the same: inject the necessary headers on all requests somewhere upstream of Shopify.

Best Practices & Maintenance: Once your custom crawler is up and running with Web Bot Auth, keep these best practices in mind:

• Secure the Signature Values: Treat the Signature-Input and Signature strings like passwords or API keys. Do not hard-code them in public code or share them. Store them in secure config files or environment variables, as they grant access to your site help.shopify.com.

• Track Expiry and Rotate: Note when each signature will expire and be ready to generate a new one before that date. Shopify will not let you renew a signature or extend its date help.shopify.com – after 90 days (or whatever period you set) it’s done. You’ll need to create a fresh signature and update your tool with the new values when the old one expires. (The old signature will continue to work until its expiry, but you can’t change an existing token’s date help.shopify.com.)

• Use Descriptive Names: Especially if you manage multiple crawlers or have a team, name each signature clearly help.shopify.com (e.g., include the tool name and purpose). This will help you identify which token is which in the Shopify admin list.

• One Domain per Signature: Remember that a signature is only valid for the domain you chose. If you have multiple store domains (or subdomains), you’ll need separate signatures for each help.shopify.com. Make sure your crawler is hitting the correct domain that matches the signature, otherwise Shopify will reject it.

• Recovering Lost Keys: If you lose the saved token values, you can always go back to Online Store → Preferences → Signatures in admin to copy the Signature-Input and Signature again help.shopify.com. Shopify displays the first few characters for identification and provides a copy button to retrieve the full values. Keep in mind this is only available for active (unexpired) signatures – once expired, you’d have to create a new one anyway.

• Remove or Replace as Needed: If you stop using a certain crawler, it’s good hygiene to go into admin and delete the signature or let it expire. That way you don’t have unused credentials floating around. And if you suspect a token was compromised, delete it early and generate a new one.

By following the above, you ensure that your custom scripts and tools can reliably crawl your Shopify store with full access, without tripping bot defenses. Next, we’ll explore why going through this trouble is worth it – especially given that some third-party SEO tools or AI like ChatGPT might already crawl sites without any special setup.

Why Web Bot Auth Matters: Authorized vs Unauthenticated Crawling

It’s reasonable to ask: “Why do I need to set up these custom crawler signatures at all? Tools like Ubersuggest or even ChatGPT seem to scan my Shopify site just fine without them.” The short answer is that Shopify’s Web Bot Auth provides a reliable, secure, and officially sanctioned way to crawl your store, whereas generic crawlers might be working partially or by luck. Let’s break down the differences from a developer’s perspective.

The Role of Web Bot Auth: The primary goal of Shopify’s Web Bot Auth system is to positively identify your bot and treat its traffic as first-party/authorized. Under the hood, it uses cryptographic verification – the Signature headers we discussed – to prove the bot is genuine developers.cloudflare.com. When Shopify (and its CDN firewall) sees a valid signature, it knows “this request is from an approved tool, not a random scraper,” and lets it through unimpeded. This gives you, as a developer or merchant, reliable access for important tasks like SEO audits, accessibility scans, automated testing, etc., without fear of being blocked. changelog.shopify.com

By contrast, if you don’t use an authorized signature, your bot is indistinguishable from any other unknown scraper out there. Shopify and Cloudflare employ various bot-detection and rate-limiting measures. Unrecognized bots might get served cached content, slowed down, or outright blocked if they hit too many pages too quickly or trigger protection rules. The upshot is that an unauthorized crawl might not get a true picture of your site. It could miss pages that were hidden behind a challenge, see outdated content, or have data gaps due to being throttled.

In technical terms, Web Bot Auth is like giving your crawler a passport and visa to access the site. Without it, the crawler is sneaking over the fence – it might succeed here and there, but it’s not guaranteed or legal. With proper credentials, it walks through the front door every time.

Why Generic SEO Tools Appear to Work: Tools like Ubersuggest, Ahrefs, Semrush, etc., crawl the web as well-known third-party bots. They often identify themselves with a custom user-agent string, but they do not use your Shopify-generated signature, since you have no way to plug that into their service. Typically, such tools rely on the fact that Shopify (like most sites) allows standard search engine crawlers and might not aggressively block well-behaved bots to avoid harming SEO. However, these tools can still be caught in Shopify’s net of bot protection:

• No Custom Headers: Ubersuggest and similar SEO SaaS cannot include your Shopify auth headers even if you wanted – they simply fetch pages like a regular browser or bot would, with no special authentication. This means they’ll be treated as unknown bots. If Shopify/Cloudflare decides to challenge or limit them, the tool will not know how to respond (it won’t solve CAPTCHAs or bypass a block).

• Risk of Blocking or Throttling: Shopify may block or rate-limit bots by default if they make rapid or high-volume requests. For example, Cloudflare’s bot management might assign a low bot score to an unfamiliar crawler, causing some requests to get challenged or served a static page. The result could be incomplete data – e.g., the crawler only gets a portion of your pages before being slowed, or it receives a “please enable cookies” page instead of your content. In the best case, the bot might get through but at a much slower pace than an authorized bot.

• Cached or Stale Data: In some cases, an untrusted crawler might be served cached content (to reduce load) or less priority. This could mean the SEO tool isn’t seeing the latest version of your site, undermining the “actionable insights” it provides. Without explicit authentication, there’s no guarantee the data is fresh or fully accurate.

• No Visibility or Control: Perhaps most importantly, when an external SEO tool crawls your site without auth, you as the site owner have no log or control over that crawl in Shopify. With Web Bot Auth, your authorized requests are a known quantity – you named them, you know which tool is running, and Shopify even logs that it was an authorized signature access. This transparency can be useful for debugging (“did our SEO tool actually hit the site successfully?”) and for accountability.

What About ChatGPT or AI “Crawling”? It’s worth noting that ChatGPT and similar AI aren’t crawlers in the traditional sense – they don’t roam the web on their own at your command. When you ask ChatGPT about your Shopify store, it’s either retrieving info from its training data (which might be outdated) or using some web browser plugin or API (that itself might be a generic, unauthenticated web fetch). In either case, ChatGPT isn’t using your Shopify credentials, so it falls in the same camp as any random script hitting your site. If it does manage to fetch your site content, it’s subject to the same potential blocking or throttling. Moreover, an AI’s interpretation can be two steps removed from real-time data – you’re relying on what the AI thinks is on your site, rather than directly auditing the live site content. This is no substitute for a purpose-built crawler that you control.

In summary, authorized crawlers ensure fidelity. When you use Web Bot Auth, you know the crawler is seeing exactly what a normal user would see, and doing so efficiently. Without it, you’re hoping nothing interferes with the crawl – which might not hold true as Shopify tightens bot security.

Let’s compare the approaches:

Bottom Line: Even if third-party SEO platforms or AI tools seem to scan your Shopify store successfully, they do so without an official blessing – meaning they’re one step away from hitting a wall. Shopify’s new Web Bot Auth signatures give you a guaranteed, secure pathway for crawling. For any serious development, SEO auditing, or automated testing work, you’ll want that assurance of unfettered access and accurate data. It’s a bit of upfront setup, yes, but it pays off by preventing headaches like incomplete audit results or false negatives caused by blocked requests. In a world where bot traffic is increasingly scrutinized (Cloudflare’s recent moves to curb unverified crawlers underscore this searchenginejournal.com), having your own whitelisted crawler is invaluable.

In short: If you care about your crawler’s results, authenticate it. Shopify has provided the mechanism – and as we’ve detailed, implementing it is quite straightforward for developers. You’ll gain a dependable window into your store’s online presence, which means better SEO analysis, more reliable QA automation, and ultimately a smoother experience maintaining your shop.

Need help or more info? Feel free to reach out or consult Shopify’s documentation on crawler signatures for more details help.shopify.comhelp.shopify.com. And if you’re using a particular tool and aren’t sure how to add headers (or need to build a custom solution), there’s a growing community of developers sharing tips on making the most of Web Bot Auth – don’t hesitate to ask for guidance!

Add-On Comment

While Web Bot Auth signatures focus on authorizing crawlers for secure and reliable access, Shopify SEO apps like SearchPie SEO (SEO Booster & Speed) address a different layer: on-page optimization and performance. These apps automate SEO tasks such as meta tag management, schema markup, sitemap generation, and speed improvements.

The distinction is important:

• Web Bot Auth ensures your crawlers and audit tools are never blocked, so you can trust the data they collect.

• SEO apps ensure the content itself is optimized for search engines once crawlers access it.

Together, they form a complementary strategy—Web Bot Auth guarantees visibility, and SEO apps maximize what that visibility delivers.