This new feature allows businesses to perform a set of actions on two populations at the same time in
order to measure the difference between their performance (the business can use it in order to learn
about the effect of a campaign on a small group before implementing it fo all members).
NOTE: A/B testing is supported only for the following actions: Send Benefit, Send SMS, Send Push.

📹 Learn more on how to perform A/B testing with our tutorial video!

STEP 1: Filter the population

Filter the population on which you would like to run an A/B testing.
NOTE: In order to use this feature, the filtered population must include more than 500 members.

STEP 2: Define and Activate the Campaign

Define for each test a name, population size and actions. Default population for each test is 50% of the filtered members. It can be changed manually as desired for test A (any value from 0 to 100%). 
In order to be able to compare test A and test B, try to have only one changing parameter between the two. For example: same gift for both groups but different communication, or same communication but different gifts.

NOTE: Currently only 2 populations comparison is available.

STEP 3: Analyze Your Campaign

Once you post the one-time activation, you can analyze the campaign and its result. All metrics will be updated daily and become permanent after a fixed time period of one month.

The metrics in the A/B testing analysis will be divided into two categories:

1. General metrics - presented regardless of the selected actions:

a. Sales: the total of all the purchases of the test members.
b. Number of Members: the actual number of members that were included in the test.
c. Average Spend Per Visit: the average spend of test members, in the time period after
the one-time was executed and up to one month.
d. Visited Members: the percentage of members that performed purchase in the time
period after the one-time was executed and up to one month.

2. Action related metrics - presented only when specific actions were performed, in addition to
the general metrics.

The actions are: Send benefit, send push, send SMS. For other actions, only the general
metrics will be presented.

a. Send benefit:
i. Redeems - the number of redeemed test benefits 
ii. Conversion rate (%) – the number of members who redeemed the test benefits
from all members that were included in the test
iii. Average Spend Per Redeem - the average sales of purchases with redemption of
the test benefit (from the members that were included in the test and
redeemed the test benefit) in the time period after the one-time was
executed and up to one month
iv. Revenue - the total estimated revenue from the benefit. The estimation is based
on a model that predicts whether a member would have made a purchase
regardless of the campaign

b. Communication (currently only push and SMS are presented in the
metrics):
i. Received – the number of members that actually received the push/SMS (not
necessarily equal to the number of push/SMS that have been sent)
ii. App opens – the number of app opens by members that were included in the
test, in the time period after the one-time was executed and up to one
month
iii. Average Days to Purchase - the average days between the action execution and
purchase (given there was a purchase)

NOTE: It might take up to 1 day until you will be able to review the A/B testing analysis.