Free shipping, a discount, a lower price: each will usually raise conversion. That’s what makes them dangerous in a conversion program. A test that reports conversion rate will declare each of them a winner, and the business can be worse off.

Free shipping

Extra costs are the most common reason US shoppers give for abandoning a checkout: 40% in Baymard’s survey named shipping, tax or fees, and 12% said they couldn’t see or calculate the total up front Reported. So showing the shipping cost early is a fix, not a test.

Making shipping free is a different decision. Michael Lewis, Vishal Singh and Scott Fay studied an online retailer that tried many shipping-fee schedules and found shoppers very sensitive to shipping charges, in whether they ordered and how much. Free shipping and free-shipping thresholds generated extra sales. But the lost shipping revenue, and the segments that didn’t respond, made those promotions unprofitable for that retailer Published. A 2020 study by Edlira Shehu, Dominik Papies and Scott Neslin found that free-shipping promotions led people to buy riskier products and return more of them, which also made the promotions unprofitable Published.

A conversion test on a free-shipping offer will almost always say yes. Ask the contribution question instead.

If you set a free-shipping threshold, put it a little above your typical order, show shoppers in the cart how far they are from it, and judge it on contribution per visitor, returns included. The Whole Machine has a contribution calculator, and The First Offer covers discounts and offers in depth.

Price tests

Random price tests aren’t generally illegal in the US, but the rules vary by state and country and are changing. They’re also easy to get wrong in ways customers remember. In September 2000, Amazon was found to be showing different customers randomly different discounts, 20% to 40% off, on 68 DVD titles over five days. After the backlash it refunded 6,896 customers an average of $3.10 each, and Jeff Bezos said, “We’ve never tested and we never will test prices based on customer demographics.” Amazon also said any future random price test would give buyers the lowest test price automatically Reported.

The law has moved since. New York’s Algorithmic Pricing Disclosure Act, in force since November 2025, requires a disclosure when a price is set by an algorithm using the customer’s personal data Published. Whether a random price test that uses no personal data falls under it hasn’t been settled; ask counsel if you sell into New York. At the federal level, the FTC published initial findings in January 2025 from a study of how companies use personal data to set individual prices, without declaring the practice illegal Published. One Shopify price-testing app, Intelligems, sets the store’s list price to the highest price in the test and applies test prices at checkout, so ads and shopping feeds never show a lower price than the store Reported.

Four rules keep a price test defensible:

1. Randomize; never targetAssign prices at random, not by device, location or anything you know about the person.
2. One price per personA visitor who comes back sees the same price.
3. Honor the lowestBe ready to refund the difference to anyone who paid more than the lowest tested price, and consider doing it automatically when the test ends, as Amazon promised in 2000.
4. Judge on contribution per visitorNot conversion rate. A higher price can lose orders and still win.

And be realistic about size. A price test is a revenue test, and chapter 3 showed that revenue needs several times the traffic of conversion. Most stores should test prices in bold steps, across a product line at once so the test has enough traffic, and only with visitor-level randomization that keeps each person on one price. Don’t use on-off weeks for price: returning shoppers would see the price change from week to week.

Do this

• List every offer and shipping change of the last year and ask what it did to contribution per visitor, not conversion rate. If nobody can say, re-read the two you’re most confident about with the before-and-after guards.