How to scrape a Shopify store

The five public endpoints

All of them answer on the store's own domain, logged out, the same way they answer a shopper's browser.

Reading the whole catalog

products.json returns at most 250 products per request: asking for limit=500 still returns 250. Page with ?page=N until a page comes back empty. Past the last page you get {"products": []} with a 200, not an error: on Allbirds, page 4 is empty because 692 products fit in three pages.

const store = "https://www.allbirds.com";
const products = [];
for (let page = 1; ; page++) {
  const res = await fetch(`${store}/products.json?limit=250&page=${page}`);
  if (!res.ok) throw new Error(`HTTP ${res.status} on page ${page}`);
  const { products: batch } = await res.json();
  if (batch.length === 0) break; // past the last page: an empty list, not an error
  products.push(...batch);
  await new Promise((resolve) => setTimeout(resolve, 400)); // be polite
}
console.log(`${products.length} products`);

It printed 692 products, exactly the published_products_count that /meta.json announces for the store.

How long it takes

The per-product Ajax endpoint answered in a median 0.49 s. One request per product is what makes stock and barcodes the slow part of any Shopify scraper: 300 products with a polite pause take about a minute and a half.

The traps

• Cents in one place, decimals in the other. The same Allbirds variant is "25.00" in products.json and 2500 in /products/{handle}.js. Pick one source per field and convert.
• No stock, no barcode in products.json. Both live only in the Ajax endpoint, and inventory_quantity is only meaningful when inventory_management is set. Allbirds tracks stock on every variant; Kith publishes no counts at all, though it does publish a barcode on every variant.
• The placeholder option. A product with a single variant still has an option called Title with the value Default Title. It is not a real option; drop it before you show variants to anyone.
• Compare prices as integers. Comparing 19.99 and 19.98 as floats to detect a one-cent price drop fails in more than one language. Store cents.
• An empty catalog is not a broken scraper. tattly.com answers products.json with an empty list, and its /meta.json says published_products_count: 0. Check the count before you debug.
• Headless stores serve nothing. gymshark.com runs Shopify behind a Next.js storefront: /products.json answers 403 and /meta.json 404, whatever the user agent. There is no public catalog to read there.
• Big catalogs are big. Kith announces 25,001 products: 101 requests of about 1.4 MB each. Set a cap, and remember that a capped read cannot tell you a product was removed, only that you did not reach it.

Or skip the plumbing

Our Shopify and WooCommerce scraper does all of the above, reads WooCommerce stores through their public Store API in the same run, returns 62 normalized fields per product, and tracks price and stock changes between runs. It runs on the Apify Store at $1.50 per 1,000 products.

Questions