Landing Page Test: "Where Your Donation Goes" Pie Chart

For additional documentation on the testing methodology please see the following pages:

Landing Pages

Result

Test Time: 2010-12-30 01:00:00 UTC - 2010-12-30 01:50:00 UTC
Sampling Interval = 2 minutes
Testing Interval = 10 minutes
Total Number of Samples per Class = 25

Donations / View*: "No Pie Chart" landing page won by 2.37%.
Amount50 / View**: "No Pie Chart" landing page won by 9.54%.

WINNER: "No Pie Chart" Landing Page

(*) The rate of donations per banner view over a fixed time interval.

(**) The rate of amount50 per banner view over a fixed time interval. Amount50 is the dollar amount raised from donations initiated under a given banner where all donations of more than $50 are recorded as $50 donations. This counters the skewing effect of outlier donations.

Data Analysis

This section analyzes and interprets the results of the tests. It should be noted that the data is analyzed over a period at least as large as the full testing period and that the testing period was chosen based on the period of time where significant hits and donations were observed.

Data Consistency and Cleaning

Analyzing the above plots the donation and impression data appear to be quite regular over the interval 2010-12-30 01:00:00 UTC - 2010-12-30 01:50:00 UTC. Therefore, two minute intervals will be used for sampling over this period as a source for the paired t-test to assess confidence in the winner.

Modelling and Hypothesis Testing

The landing page "No pie chart" won in each case for donations/view and amount50/view with increases of 2.37% and 9.54% respectively. The student's t-test was used to assess confidence over each metric and the confidence in the winner for donations/view and amount50/view were at least 95.0% and 99.0% respectively. Although the the increases are not large they are significant and it is notable that the increase in amount50/view is more than donations/view. There are two potential causes here, (1) the variability of amounts tends to be much greater than donations (since donations are binary and amounts are values subject to a relatively wide range), (2) the winner actually had an affect on engaging higher amount donors. However, since both results are significant (ie. the cumulative variance was small enough) this may point to the latter explanation.


TOTAL DONATIONS "No pie chart": 348
TOTAL DONATIONS "Pie chart": 309

TOTAL AMOUNT50* RAISED "No pie chart": $3873.73
TOTAL AMOUNT50* RAISED "Pie chart": $3362.00

* AMOUNT50 indicates the total amount raised where all donations greater than $50 are taken to be a donation of $50.


DONATIONS PER VIEW:

Between 95.0% and 97.5% confident about the winner.

Pie chart -- 2010-12-30 01:00:00 - 2010-12-30 01:50:00

item 1  = Pie chart
item 2  = No pie chart

The winner No pie chart had a 2.37% increase.

interval	mean1		mean2		stddev1		stddev2

0		0.03043		0.03778		0.00988		0.02216
1		0.03239		0.04426		0.00886		0.00472
2		0.05725		0.04604		0.02748		0.01531
3		0.05320		0.05948		0.02602		0.03636
4		0.05524		0.04636		0.00410		0.00290


Overall Parameters:

mean1		mean2		stddev1		stddev2
0.04570		0.04678		0.01803		0.02039


AMOUNT50 PER VIEW:

Between 99.0% and 99.5% confident about the winner.

Pie chart -- 2010-12-30 01:00:00 - 2010-12-30 01:50:00

item 1  = Pie chart
item 2  = No pie chart

The winner No pie chart had a 9.54% increase.

interval	mean1		mean2		stddev1		stddev2

0		0.52351		0.51283		0.24953		0.30449
1		0.47199		0.86516		0.29552		0.18720
2		0.83610		0.66408		0.06229		0.01991
3		0.76068		1.09058		0.16285		0.82047
4		0.89079		0.68263		0.09534		0.08091


Overall Parameters:

mean1		mean2		stddev1		stddev2
0.69662		0.76306		0.19447		0.40196

Endnotes

Campaign = "20101229JA083_US"
"No pie chart" landing page = "WMFJA064"
"Pie chart" landing page = "WMFJA068"