One-Time: A/B Testing

This new capability allows businesses to perform a set of actions on two populations at the same time in
order to measure the difference between the performance (the business can use it in order to learn
about the campaign in a small group before implementing it for all members).
Note: A/B testing supported only for the following actions: Send Benefit, Send SMS, Send Push.



STEP 1: Filter the population
Filter the population you would like to run an A/B testing.
Note: In order to use this capability, the filtered population must include more than 500 members



STEP 2: Define and Activate the Campaign
Define for each test its name, population size and actions. Default population for each test is 50% of
the filtered member. It can be changed manually as desired for test A (any value from 0 to 100%), test
B is disable and complete the test for the entire filtered population.
In order to be able to compare between A to B test, try to have only one variable parameter, for
example: same gift for both groups but different communication, or same communication but
different gits.



Note: currently only 2 population comparison are available.

STEP 3: Analyze Your Campaign
Post the one-time activation, you can analyze the campaign and its result. All metrics will be updated
daily and becomes permanent after a fixed time period of one month.



The metrics in the A/B testing analysis will be divided into two categories:
1. General metrics – presented regardless of the selected actions:

a. Sales – the total of all the purchases of the test’s members
b. Number of Members - the actual number of members that were included in the test
c. Average Spend Per Visit - average spend of test members, in the time period after
the one-time was executed and up to one month.
d. Visited Members – the percentage of members that performed purchase in the time
period after the one-time was executed and up to one month, from all members that
were included in the test.

2. Action related metrics – presented only when specific actions were performed, in addition to
the general metrics.
The actions are: Send benefit, send push, send SMS. For other actions, only the general
metrics will be presented.

a. Send benefit:
i. Redeems - number of redeemed benefits from the member’s test (of the
test’s benefit)
ii. Conversion rate (%) – the number of members who redeemed the text’s benefit
from all members that were included in the test.
iii. Average Spend Per Redeem - average sales of purchases with redemption of
the test’s benefit (from the members that were included in the test and
redeemed the test’s benefit) in the time period after the one-time was
executed and up to one month.
iv. Revenue - total estimated revenue from the benefit. The estimation is based
on a model that predicts whether a member would have made a purchase
regardless of the campaign.

b. Communication (currently only send push and send SMS are presented in the
i. Received – number of members that actually received the push/SMS (not
necessarily equal to the number of push/SMS that have been sent)
ii. App opens – number of app opens by members that were included in the
test, in the time period after the one-time was executed and up to one
iii. Average Days to Purchase - average days between the action execution and
purchase (given there WAS a purchase)

Note: It might take up to 1 day until you will be able to review the A/B testing analysis.

Was this article helpful?
1 out of 1 found this helpful