Jump to content

Fundraising 2010/Report/Draft Test 7

From Meta, a Wikimedia project coordination wiki

Banner Test - Progress Meter

[edit]

For additional documentation on the testing methodology please see the following pages:


Banners

[edit]
Animated Progress Meter. This is a static banner and features the text "Please read: An urgent appeal from Wikipedia founder Jimmy Wales.". The progress meter below the text is animated.


Static Progress Meter. This is a static banner and features the text "Please read: An urgent appeal from Wikipedia founder Jimmy Wales.". The progress meter below the text is a static graphic.

Result

[edit]

Test Time: 2010-12-17 19:00:00 UTC - 2010-12-17 22:00:00 UTC
Sampling Interval = 2 minutes
Testing Interval = 180 minutes
Total Number of Samples per Class = 90


TEST RESULT INCONCLUSIVE


(*) The rate of donations per banner impression over a fixed time interval.

(**) The rate of amount50 per banner impression over a fixed time interval. Amount50 is the dollar amount raised from donations initiated under a given banner where all donations of more than $50 are recorded as $50 donations. This counters the skewing effect of outlier donations.

Data Analysis

[edit]

This section analyzes and interprets the results of the tests.

Data Consistency and Cleaning

[edit]

The plots below display the counts of the data sources over the testing period as verification of the consistency of the donation pipeline data used in testing. It should be noted that a certain amount of natural variance is expected in donation counts and is no cause for alarm. The bottom two figures depict the total views and donations with those coming from each banner. This illustrates when the campaign becomes active and is also a useful tool for determining where anomalous behaviour may be visible in the data. It is of note that there are a couple of intervals where the impressions dipped down to zero, however these were sparse enough not to effect the quality of the data significantly. It should be noted that the data is analyzed over a period at least as large as the full testing period and that the testing period was chosen based on the period of time where significant hits and donations were observed.

Impressions broken out in two minute intervals over the test period.


Donations broken out in two minute intervals over the test period.


Donations/Impression broken out in two minute intervals over the test period.


Amount50/Impression broken out in two minute intervals over the test period.

Analyzing the above plots the donation and impression data appear to be quite regular over the interval 2010-12-17 19:00:00 UTC - 2010-12-17 22:00:00 UTC. Therefore, two minute intervals will be used for sampling over this period as a source for the paired t-test to assess confidence in the winner.

Modelling and Hypothesis Testing

[edit]

"Animated Progress Meter" won in each case for donations/impression and amount50/impression with increases of 5.30% and 6.22% respectively. The student's t-test was used to assess confidence over each metric and the confidence in the winner for donations/impression and amount50/impression is at least 75.0% and 95.0% respectively. This is a very slight and non-significant win for the "Animated Progress Meter". Both banners effectively performed the same however this may merit more testing as additional data could lead to a conclusive result.

Mean and standard deviation over test intervals over Donations / Impression.


Mean and standard deviation over test intervals over Amount50 / Impression

TOTAL DONATIONS "Animated Progress Meter": 397
TOTAL DONATIONS "Static Progress Meter": 384

TOTAL AMOUNT50* RAISED "Animated Progress Meter": $7435.00
TOTAL AMOUNT50* RAISED "Static Progress Meter": $7096.81

* AMOUNT50 indicates the total amount raised where all donations greater than $50 are taken to be a donation of $50.


DONATIONS PER IMPRESSION:

Between 75.0% and 90.0% confident about the winner.

Progress Meter -- 2010-12-17 19:00:00 - 2010-12-17 22:00:00

item 1  = Animated Progress Meter
item 2  = Static Progress Meter

The winner "Animated Progress Meter" had a 5.30% increase.

interval	mean1		mean2		stddev1		stddev2

0		0.00015		0.00014		0.00004		0.00007
1		0.00014		0.00014		0.00004		0.00003


Overall Parameters:

mean1		mean2		stddev1		stddev2
0.00015		0.00014		0.00004		0.00005


AMOUNT50 PER IMPRESSION:

Between 95.0% and 97.5% confident about the winner.

Progress Meter -- 2010-12-17 19:00:00 - 2010-12-17 22:00:00

item 1  = Animated Progress Meter
item 2  = Static Progress Meter

The winner "Animated Progress Meter" had a 6.22% increase.

interval	mean1		mean2		stddev1		stddev2

0		0.00299		0.00252		0.00190		0.00050
1		0.00251		0.00266		0.00077		0.00050


Overall Parameters:

mean1		mean2		stddev1		stddev2
0.00275		0.00259		0.00145		0.00050

Endnotes

[edit]
  1. Campaign = "20101217JA033"
  2. "Animated Progress Meter" utm_source = "20101217_JA022A_US"
  3. "Static Progress Meter" utm_source = "20101217_JA022B_US"