Is comparing members to non-members fair?

Not entirely, and it is worth knowing why: people who join are more engaged to begin with. Cohort comparison over time is the more honest method.

How long before I can tell?

Several times your typical visit gap. For a weekly cafe regular, a couple of months. Anything sooner measures signup, not loyalty.

How to tell whether your loyalty programme is actually working

Compare members against comparable non-members, or compare joining cohorts against each other over time. A programme's total activity always rises as people join, so any measure without a comparison group will tell you it is working whether or not it is.

The awkward truth about loyalty measurement is that the obvious numbers all go up regardless.

Why the obvious numbers lie

Members joined, so member activity rises. Total stamps issued rises. Redemptions rise. None of that tells you whether anybody visited more than they would have.

Selection bias, named

People who join a loyalty programme are already more engaged than people who do not. Comparing the two groups and finding members visit more proves that engaged people join things.

It is not worthless, but it is not the evidence you think it is.

The comparison that works

Cohorts over time. Take everyone who joined in March and follow their visit frequency. Then April, then May. If later cohorts behave better after a change you made, that is a real signal.

Before and after, per person. For members who joined long enough ago that you have history both sides, did their own visit frequency change? This is the closest a small business gets to a controlled comparison.

A held-back group. If you run a campaign, leave a comparable group out of it. The difference between them is the effect. This is the single most valuable habit in the whole article and costs nothing but discipline.

Change one thing at a time

Two changes in a month means learning nothing from either. Loyalty programmes move slowly, so the temptation to adjust several things at once is strong and should be resisted.

What a working programme looks like

Visit frequency among members rising over time relative to earlier cohorts. Completion rates healthy. Lapsed regulars falling. Redemption rate and visit frequency moving together rather than redemption alone.

What to do if it is not working

Read the diagnosis in order: is anyone mentioning it, is the code where people wait, is the threshold reachable, is the reward worth having. In that order, because that is the order of likelihood.