Landing Page Test: "Ask String Compare"

For additional documentation on the testing methodology please see the following pages:

Landing Pages

The difference here is subtle, the selections in the radio-button "Ask" strings are different.

Result

Test Time: 2010-12-30 16:00:00 UTC - 2010-12-30 16:48:00 UTC
Sampling Interval = 2 minutes
Testing Interval = 48 minutes
Total Number of Samples per Class = 24

TEST RESULT INCONCLUSIVE

(*) The rate of donations per banner view over a fixed time interval.

(**) The rate of amount50 per banner view over a fixed time interval. Amount50 is the dollar amount raised from donations initiated under a given banner where all donations of more than $50 are recorded as $50 donations. This counters the skewing effect of outlier donations.

Data Analysis

This section analyzes and interprets the results of the tests. It should be noted that the data is analyzed over a period at least as large as the full testing period and that the testing period was chosen based on the period of time where significant hits and donations were observed.

Data Consistency and Cleaning

Analyzing the above plots the donation and impression data appear to be quite regular over the interval 2010-12-30 16:00:00 UTC - 2010-12-30 16:48:00 UTC. Therefore, two minute intervals will be used for sampling over this period as a source for the paired t-test to assess confidence in the winner.

Modelling and Hypothesis Testing

The plots below depict the mean and standard deviation of each banner over four ten minute testing intervals. Samples were taken over two minute periods. The hypothesis testing was performed over the average mean and variance of the test intervals. The landing page "ask string 5,20,30,50,100,200,300" won in each case for donations/view and amount50/view with increases of 6.48% and 2.49% respectively. The student's t-test was used to assess confidence over each metric and the confidence in the winner for donations/view and amount50/view were at least 75.0% and 60.0% respectively. This is not a significant result and it cannot be determined if the different ask strings make any difference in donations from this data.


TOTAL DONATIONS "ask string 5,20,30,50,100,250,500": 406
TOTAL DONATIONS "ask string 5,20,30,50,100,200,300": 401

TOTAL AMOUNT50* RAISED "ask string 5,20,30,50,100,250,500": $4620.66
TOTAL AMOUNT50* RAISED "ask string 5,20,30,50,100,200,300": $4672.00

* AMOUNT50 indicates the total amount raised where all donations greater than $50 are taken to be a donation of $50.


DONATIONS PER VIEW:

Between 75.0% and 90.0% confident about the winner.

Ask String -- 2010-12-30 16:00:00 - 2010-12-30 16:48:00

item 1  = ask string 5,20,30,50,100,250,500
item 2  = ask string 5,20,30,50,100,200,300.

The winner ask string 5,20,30,50,100,200,300. had a 6.48% increase.

interval	mean1		mean2		stddev1		stddev2

0		0.07933		0.08100		0.00933		0.04019
1		0.06992		0.07791		0.00957		0.00390


Overall Parameters:

mean1		mean2		stddev1		stddev2
0.07462		0.07946		0.00945		0.028


AMOUNT50 PER VIEW:

Between 60.0% and 75.0% confident about the winner.

Ask String -- 2010-12-30 16:00:00 - 2010-12-30 16:48:00

item 1  = ask string 5,20,30,50,100,250,500
item 2  = ask string 5,20,30,50,100,200,300.

The winner ask string 5,20,30,50,100,200,300. had a 2.49% increase.

interval	mean1		mean2		stddev1		stddev2

0		1.26806		1.31983		0.31806		0.60554
1		1.18943		1.19896		0.11184		0.07377


Overall Parameters:

mean1		mean2		stddev1		stddev2
1.22874		1.25940		0.23840		0.43135

Endnotes

Campaign = "20101230JA091_US"
"ask string 5,20,30,50,100,250,500" landing page = "WMFJA069"
"ask string 5,20,30,50,100,200,300" landing page = "WMFJA071"