How ProteinDeals Tracks UK Protein Prices: Where Our Data Comes From

There is not one generic script pretending every shop has the same website. ProteinDeals has retailer configurations and retailer-specific parsers because Shopify stores, custom storefronts and marketplace listings expose products differently.

An admin run can discover retailer categories, select the sections to collect, scrape product pages and produce structured variant records. The pipeline keeps the source URL and page evidence used during extraction so questionable size or nutrition data can be audited later.

1. 1

   Category discovery: finds the retailer sections relevant to protein, foods, supplements and other supported categories.

2. 2

   Product discovery: gathers unique product URLs rather than relying on a hand-written list of individual products.

3. 3

   Variant extraction: separates flavours, sizes, bundles and purchase options where the retailer exposes them.

4. 4

   Page extraction: reads current price, stock, serving information, ingredients, nutrition, images and descriptive evidence.

5. 5

   Import: matches or creates the corresponding product, retailer and variant records in the database.

A product name is not a price. The same powder can have several flavours, bag sizes, multipacks and subscription options, each with a different URL or price. ProteinDeals stores the sellable variant using retailer identity, flavour and canonical size so a 500g pouch is not silently compared with a 2.5kg bag as if they were the same item.

Product-level information such as the name, brand, category and core nutrition is linked to variant-level information such as retailer, flavour, size, stock, URL and last-scraped time. This separation is what allows related options to appear together while retaining their real prices.

Each imported variant updates its current stock state and last-scraped timestamp. When the record contains price data, the importer creates a new timestamped price record rather than overwriting the only historical value.

A price record can hold the single-purchase price, price per 100g, subscription price, subscription price per 100g, subscription saving and whether the retailer marked the item as on sale. The comparison uses the newest record by scrape time, while the stored series supports the internal 90-day price-history view.

A subscribe-and-save price is not presented as though every buyer can pay it once. ProteinDeals stores single and subscription prices separately and defaults the comparison to the single-purchase value.

Where a retailer publishes a genuine subscription price or percentage saving, the importer records it as a separate purchase mode. Switching to subscription mode recalculates the normalised price metrics from that subscription value.

The raw pack price is retained, but the useful comparison depends on the product category. Powder sold by weight can be compared per 100g; products with reliable serving information can be compared per serving; protein products can also be compared by the amount of protein they actually contain.

Price per 100g = pack price ÷ pack size in grams × 100. Price per serving = pack price ÷ servings per pack. Price per 1g protein = pack price ÷ total grams of protein in the pack.

For protein powder, total protein is pack size multiplied by protein per 100g and divided by 100. Missing nutrition is kept missing rather than converted to zero, so incomplete rows cannot appear artificially cheap.

ProteinDeals prefers explicit values from the retailer or manufacturer's published nutrition table. It looks for the relevant protein and energy rows and handles both per-100g and per-serving columns.

When a site supplies protein per serving and a valid serving size but omits protein per 100g, the project can derive the per-100g value. It can likewise derive serving count from pack size and serving size. These fallbacks only run when the inputs are valid; implausible macro values and missing sizes are not guessed.

Stock is stored on each product variant. In-stock variants power the normal comparison and retailer counts, while out-of-stock rows are hidden unless the user asks to see them.

Imports reconcile the latest retailer data with existing database variants. This prevents a stale option from continuing to look current when a retailer removes, replaces or changes it, while preserving the price records already collected for audit and history.

Amazon and eBay offers are not treated as interchangeable copies of a direct retailer listing. Marketplace matchers compare candidate titles and sizes with the ProteinDeals variant, store match scores and reasons, and support approval or rejection before questionable matches are shown.

Marketplace records keep item price and delivery price as separate fields where available. They also retain the marketplace product URL, availability and match timestamp. This extra matching layer reduces the chance of attaching the price of a sample, multipack or wrong-size product to the main listing.

ProteinDeals has a separate voucher-code collection and import flow. Codes can be displayed beside an eligible retailer, but a promotional code is not automatically deducted from every comparison row as though it is guaranteed to work for every product and customer.

A live sale price published directly on the product page can be collected as the current price. Checkout-only, account-specific, minimum-spend and limited-category codes remain visible offers rather than hidden assumptions inside the base calculation.

Retailer imports run through the admin scraper tools rather than a single hard-coded weekly clock. Different retailers can be refreshed independently, which is useful when one catalogue changes without requiring every other source to be collected again.

The public comparison API maintains a versioned 24-hour summary cache for speed. A successful import clears the in-memory and persisted comparison caches immediately, so the next request rebuilds them from the newly imported database state. The comparison response itself is not placed behind a separate browser cache that could remain stale after invalidation.

Retailer pages are moving targets. A price can change after its last scrape, a flavour may inherit the wrong default option from a storefront, a promotion may require an account, or the retailer may alter its page structure. Delivery can also change the final amount.

ProteinDeals uses retailer-specific parsing, source evidence, validation and marketplace review tools to reduce these errors, but no comparison site controls the retailer's checkout. Always confirm the exact product, size, purchase type, delivery charge and final total before paying.

The database is no longer limited to whey powder. The same pipeline supports protein products, foods, general supplements and selected accessories, with category-specific size and value fields. Direct suppliers and marketplace alternatives can both appear where reliable records exist.

Coverage grows retailer by retailer. A supplier being listed does not mean every historical product, flavour or temporary offer has been collected, and a missing value means the current source did not provide enough trustworthy information to calculate it.