How to test whether your program makes a difference

People who already like a shop are the ones who take its card. Compare them a month later with the people who did not, and you are mostly measuring that liking.

A study of grocery shopping in the Netherlands put a size on the error. From July 1998 to July 2000, 1,909 households scanned every supermarket receipt at home and reported which of the country's seven grocery loyalty programs they belonged to, across its 20 largest chains. On the raw figures, a chain's members gave it 36% of their grocery spending and non-members 7%, a gap of 29 points. Once the authors allowed for loyal shoppers choosing to join, membership itself added 4.1 points: "a small positive yet significant effect", which "is seven times smaller than is suggested by a naïve model that ignores the endogeneity of program membership". These are Dutch supermarkets in the late 1990s, read in the authors' manuscript of the paper. What travels is the direction of the error: the simple comparison overstated the program seven times over.

An example with made-up numbers. A café has 40 customers who come most mornings, about 20 times a month, and 160 who drop in twice a month. It launches a card that changes nobody's habits. Thirty of the morning crowd take it, because it pays them for what they do anyway, and 20 of the others do. A month later the 50 members have averaged 12.8 visits and the 150 non-members 3.2. "Members visit four times as often" is true, and the card caused none of it.

native.cards counts only members, so its dashboard cannot make this comparison. The nearest thing is on Home's Retention view: What the first month predicts sets members with one visit in their first 30 days beside those with two, and those with three or more. It shows who stayed. It does not show that a second visit makes a regular, because the keener member does more of everything.

Sound studies use these designs. The café study behind How soon should customers earn their first reward? used the third: research assistants "randomly offered customers either a 10-stamp or a 12-stamp coffee card", 108 customers in all.

In native.cards the first is partly built in. You run the other two by hand.

Before and after. For a notification, the column Came in, 7 days does this for you. For a dated change, such as adding a second, closer reward, use Retention by join month on Home's Retention view ("Share of each month’s new members who visited in later months"). Read down one column, such as +1, and compare months that joined well before the change with months that joined after it. Leave out the month just before: its +1 already falls after the change. Skip each row's outlined last figure, which is the month still running. Months differ, so this is the weakest of the three.

Alternate weeks. Send the reward reminder in weeks one and three and nothing in weeks two and four. The Visits per week chart on Home's Overview shows the last 12 weeks, and each column carries that week's visits and the number of members who visited. Here the weeks are the sample, not the members. Two of each is only a hint. The chart's 12 columns include the week still running, so it holds five or six of each.

A split by chance. native.cards has no setting that keeps a random group out of a notification or a reward, but you can split what happens at your counter. In QR Materials, under Ready-made QR codes, use Your own twice to add two categories, named, say, Test A and Test B. Each code "is live at once" and is counted on its own. Print a sign with each. Put the first on the counter, keep the second behind it and hand it to every second new customer, along with whatever you are testing: a different way of asking, or a reminder of how close the first reward is. Campaigns gives each QR code its own line, with Joined and Came back (two or more recorded visits in a member's first 30 days). The rule has to be mechanical. Once staff choose who gets the second sign, the keen customers land in one group and you are back to the first mistake.

Run any of the three for at least one full reward cycle, because pace changes along a card. The café study had a second test, separate from the one above, which timed finished ten-stamp cards. Customers bought faster as the free coffee neared and slowed once they had it, and a card took 24.6 days on average. Your own cycle is Time to earn it in the program's Rewards view.

Every notification you have sent has this column beside it in Notifications. The screen describes it as "Members who visited within 7 days of receiving it, beside the same members' visits the week before."

The same people are on both sides, so the problem of who joins has gone. The figure shows whether more of those members came in the week after than the week before. It does not show that the notification brought them:

• The two weeks are different weeks. A notification sent just before payday has payday in its second week. A public holiday, school holidays or a week of rain do as much, in either direction.
• The audience can bend the earlier week. With New members ("Joined in the last 30 days."), some joined inside that week. Those who joined while paying at your counter have a visit in it if staff recorded the purchase. Those who joined from a link could not have visited before they were members.
• "Receiving it" means Google accepted the notification, or Apple accepted the card update that carries it. It does not mean the member read it.

The cure is repetition. Send the same kind of notification three or four times, in dull weeks as well as good ones, and see whether the week after beats the week before every time. Read each one only when Still counting has gone.

Chance alone makes two groups differ. Split 100 new members in half, treat them identically, and their return rates will still not match. The table shows roughly how far apart two identical groups can drift. It is worked out with the standard formula for comparing two shares, for return rates near one half, where chance has most room, and it covers 19 cases in 20.

With 50 members a side, 44% against 58% tells you nothing yet. With 400 a side the same gap is worth acting on. A shop that gains 40 members a month needs five months to fill two groups of 100.

Holding back only a few, say every tenth new member, does not work. A tenth of 300 is 30 against 270, the small group sets the limit, and a gap of up to about 19 points can be luck. The same 300 split in half can show 11.

So a small shop can find out whether a change makes a large difference. Whether it makes a small one is beyond its numbers, and a small gap seen once is not a finding.

A study of a men's hair salon chain shows where an effect is likely to appear. By Rice University's account, it covered more than 5,500 clients over 30 months, the program arrived ten months in, and it gave US$5 off for every US$100 spent. The authors had records from before the program existed. That let them compare how members and non-members changed, where a simple comparison sees only how they differ. The paper's abstract reports that the program raised customer value "by almost 30% over a five-year horizon", that its "reduction in attrition accounts for more than 80% of the program’s total lift", and that its effect "on spending per visit is negligible". This is one chain, in the United States, and the five-year figure comes from the authors' model. For your own check it settles what to measure: the share of members who come back, not the size of the bill.

A second study shows when to look. It covered 322 listed companies that introduced a program between 2000 and 2015. Sales and gross profit rose within the first year, but the effect on gross profit did "not become significant until the second quarter" and lagged "substantially behind sales". These were public companies, not shops, and the abstract gives no percentages. So a launch judged on its first month's sales is judged too early and on the wrong line.

native.cards records what members spend and knows nothing of what your goods cost you. Gross profit comes from your own books: sales less the cost of what you sold and of what you gave away. For the program as a whole that is the only check a shop has, and it is the weakest kind, a before and after: the two quarters after launch against the two before.