Research:Donor engagement/Thank You campaign
Overview
[edit]The campaign ran 3 different pairs of banners tested concurrently. During each test, visitors randomly received one of the two banners up to a maximum of 5 impressions, after which the banner was automatically disabled. Visitors also had the ability to manually disable banners via the close button. Each banner included a direct link to the account registration landing page as well as thumbnails pointing to a series of landing pages with editor stories, which in turn pointed to the account creation page (indirect link). Referrals from the direct link where logged with a fr12b-{banner_id} campaign id, while referrals from the indirect links embedded in editor stories where logged with a fr12c-{page} campaign. The campaign was launched at 2012-12-27T21:15:00Z and ended approximately at 2013-01-01T00:10:00Z.
Banners
[edit]- 2012-12-27
- B12_1227_ThankYou_5pillars
- B12_1227_ThankYou_Short
- 2012-12-28
- B12_1228_ThankYou_Nonprofit
- B12_1228_ThankYou_Thrilled
- 2012-12-31
- B12_1231_ThankYou_Createacct
- [1]
Results
[edit]How many new users registered as a result of this campaign?
[edit]The campaign resulted in a total of 1,107 successful account registrations referred from the banners. The vast majority of these registrations (N=1,057, 95.4%) were referred via the direct link, while only a tiny fraction originated from the landing pages (N=50, 4.5%) . During the period of the campaign (as defined by the time of the first and the last account registration referred by the banners), a total of 20,958 users registered an account via the regular Create Account link, in line with the expected signup rate for this period of the year.
How productive are these users compared to organic registrations?
[edit]
To compare the productivity of the two groups of referred and organic registration we looked at the proportion of users in each group who completed at least 1 edit in the main article namespace within 24 hours of registrations. A total of 183 users (17.3%) from the referred group completed 1+ main namespace edits in 24 hours compared to 4,633 users (22.1%) from the control group of organic registrations (+4.8%). A chi-square test rejects the null hypothesis that the true proportion of editing accounts is the same in both groups (X² = 13.24, N = 22,015, p < .001). Referred users were significantly less likely to complete an edit than the control.
Which banner had the highest conversion rate?
[edit]Banners had fairly low conversion rates (account creations by raw account impressions), with mean conversions ranging from 0.00037% to 0.00056%. We compared the raw clickthrough rate, completion and conversion rate for each pair of banners that ran concurrently (we didn't include the last pair of banners (B12_1231_*) as they ran for a substantially shorter amount of time compared to the previous ones). The plots below compare the campaigns against these three metrics.
-
Click-through -
Signup completion -
Conversion -
Click-through -
Signup completion -
Conversion
Conversion rates for each banner campaign (with binomial proportion CI)
Estimate Lower Upper Campaign
1 0.0005489544 0.0004825809 0.0006244566 B12_1227_ThankYou_5pillars
2 0.0005655095 0.0004927165 0.0006490568 B12_1227_ThankYou_Short
Estimate Lower Upper Campaign
1 0.0004575854 0.0004052429 0.0005166885 B12_1228_ThankYou_Nonprofit
2 0.0003737361 0.0003332572 0.0004191317 B12_1228_ThankYou_Thrilled
Conclusions
[edit]Funnel conversion data
[edit]The following data was collected at the end of the test (events logs imported as of 2013-01-02T23:30:00Z, raw event counts, non-deduplicated).
Banner impressions
[edit](1/100 sampled impressions)
420800 B12_1227_ThankYou_5pillars 357200 B12_1227_ThankYou_Short 568200 B12_1228_ThankYou_Nonprofit 781300 B12_1228_ThankYou_Thrilled 76300 B12_1231_ThankYou_Createacct 23200 B12_1231_ThankYou_Getstrtd
Account creation: impressions
[edit](excluding error impressions, i.e. form impressions logged when an error is returned after submitting the form)
Banner
[edit]2298 fr12b-B12_1228_ThankYou_Thrilled 2145 fr12b-B12_1228_ThankYou_Nonprofit 1517 fr12b-B12_1227_ThankYou_Short 1332 fr12b-B12_1227_ThankYou_5pillars 623 fr12b-B12_1231_ThankYou_Getstrtd 288 fr12b-B12_1231_ThankYou_Createacct
Landing page
[edit]110 fr12c-Poongothai-Balasubramanian 92 fr12c-Thank-You 84 fr12c-Ken-Thomas 73 fr12c-Mei-Jiun 57 fr12c-Jimmy-Wales 55 fr12c-Ruben-Hilare-Quispe 53 fr12c-Oarabile-Mudongo 15 fr12c-Abhishek-Hingnikar 12 fr12c-Melisa-Parisi 12 fr12c-Benjamin-Chen 8 fr12c-Peter-Gallert 7 fr12c-Ravan-Jaafar 6 fr12c-Zinaida-Tebaykina 6 fr12c-Peter-Coti 6 fr12c-Noopur-Raval 6 fr12c-Dumisani-Ndubane 6 fr12c-Aliona-Bogdanova 5 fr12c-Q-Miceli 5 fr12c-Netha-Hussain 4 fr12c-Viswa-Prabha 4 fr12c-Elena-Strelnikova 4 fr12c-Anastasia-Lvova 4 fr12c-Amina-Duale 3 fr12c-Susanna-Mkrtchyan 3 fr12c-Ganesh-Paudel 3 fr12c-David-Shankbone 3 fr12c-Andrea-Zanni 2 fr12c-Tony-Santiago 2 fr12c-Judy-Tuan 2 fr12c-Juan-Ignacio 2 fr12c-Ivan-Meetin 2 fr12c-Howard-Morland 2 fr12c-Chanitra-Bishop 2 fr12c-Benoit-Rochon 2 fr12c-Amit-Kapoor 2 fr12c-Alejandro-Linares 1 fr12c-Srikeit-Tadepalli 1 fr12c-Mina-Nagy 1 fr12c-Leslie-Carr 1 fr12c-Gideon-Digby 1 fr12c-Gereon-Kalkuhl 1 fr12c-Emmanuel-Engelhart 1 fr12c-Deryck-Chan 1 fr12c-David-Thomsen 1 fr12c-Brandon-Harris 1 fr12c-Adrianne-Wadewitz
Account creation: error impressions
[edit](form impressions logged when an error is returned after submitting the form)
Banner
[edit]1735 fr12b-B12_1228_ThankYou_Thrilled 1314 fr12b-B12_1228_ThankYou_Nonprofit 981 fr12b-B12_1227_ThankYou_5pillars 900 fr12b-B12_1227_ThankYou_Short 239 fr12b-B12_1231_ThankYou_Getstrtd 164 fr12b-B12_1231_ThankYou_Createacct
Landing page
[edit]52 fr12c-Poongothai-Balasubramanian 40 fr12c-Oarabile-Mudongo 31 fr12c-Thank-You 31 fr12c-Amina-Duale 25 fr12c-Ken-Thomas 19 fr12c-Ruben-Hilare-Quispe 17 fr12c-Jimmy-Wales 15 fr12c-Abhishek-Hingnikar 11 fr12c-Mei-Jiun 4 fr12c-Ganesh-Paudel 4 fr12c-Deryck-Chan 3 fr12c-Viswa-Prabha 3 fr12c-Peter-Gallert 3 fr12c-Dumisani-Ndubane 3 fr12c-Amit-Kapoor 2 fr12c-Zinaida-Tebaykina 2 fr12c-Ravan-Jaafar 2 fr12c-Noopur-Raval 2 fr12c-Benjamin-Chen 2 fr12c-Adrianne-Wadewitz 1 fr12c-Elena-Strelnikova 1 fr12c-Andrea-Zanni
Account creation: submits
[edit](excluding multiple submit attempts after the form returns an error)
Banner
[edit]814 fr12b-B12_1228_ThankYou_Thrilled 765 fr12b-B12_1228_ThankYou_Nonprofit 590 fr12b-B12_1227_ThankYou_Short 557 fr12b-B12_1227_ThankYou_5pillars 117 fr12b-B12_1231_ThankYou_Getstrtd 91 fr12b-B12_1231_ThankYou_Createacct
Landing page
[edit]35 fr12c-Poongothai-Balasubramanian 19 fr12c-Thank-You 17 fr12c-Ken-Thomas 12 fr12c-Jimmy-Wales 10 fr12c-Ruben-Hilare-Quispe 9 fr12c-Oarabile-Mudongo 7 fr12c-Mei-Jiun-Kwek 5 fr12c-Abhishek-Hingnikar 3 fr12c-Zinaida-Tebaykina 3 fr12c-Amina-Duale 2 fr12c-Viswa-Prabha 2 fr12c-Peter-Gallert 2 fr12c-Ganesh-Paudel 2 fr12c-Dumisani-Ndubane 2 fr12c-Benjamin-Chen 1 fr12c-Tony-Santiago 1 fr12c-Ravan-Jaafar-Altaie 1 fr12c-Peter-Coti 1 fr12c-Noopur-Raval 1 fr12c-Juan-Ignacio-Iglesias 1 fr12c-Howard-Morland 1 fr12c-Elena-Strelnikova 1 fr12c-Deryck-Chan 1 fr12c-Andrea-Zanni 1 fr12c-Anastasia-Lvova 1 fr12c-Amit-Kapoor 1 fr12c-Aliona-Bogdanova 1 fr12c-Adrianne-Wadewitz
Account creation: error submits
[edit](submit attempts after the form returned at least one error)
Banner
[edit]1035 fr12b-B12_1228_ThankYou_Thrilled 875 fr12b-B12_1228_ThankYou_Nonprofit 725 fr12b-B12_1227_ThankYou_5pillars 595 fr12b-B12_1227_ThankYou_Short 165 fr12b-B12_1231_ThankYou_Getstrtd 119 fr12b-B12_1231_ThankYou_Createacct
Landing page
[edit]31 fr12c-Poongothai-Balasubramanian 31 fr12c-Amina-Duale 29 fr12c-Oarabile-Mudongo 18 fr12c-Thank-You 15 fr12c-Jimmy-Wales 11 fr12c-Ken-Thomas 11 fr12c-Abhishek-Hingnikar 9 fr12c-Ruben-Hilare-Quispe 5 fr12c-Mei-Jiun-Kwek 3 fr12c-Viswa-Prabha 3 fr12c-Thank-You 3 fr12c-Ganesh-Paudel 3 fr12c-Deryck-Chan 3 fr12c-Amit-Kapoor 2 fr12c-Poongothai 2 fr12c-Peter-Gallert 2 fr12c-Noopur-Raval 2 fr12c-Benjamin-Chen 2 fr12c-Adrianne-Wadewitz 1 fr12c-Zinaida-Tebaykina 1 fr12c-Ravan-Jaafar-Altaie 1 fr12c-Dumisani-Ndubane
Account creation: success
[edit]Banner
[edit]292 fr12b-B12_1228_ThankYou_Thrilled 260 fr12b-B12_1228_ThankYou_Nonprofit 231 fr12b-B12_1227_ThankYou_5pillars 202 fr12b-B12_1227_ThankYou_Short 38 fr12b-B12_1231_ThankYou_Getstrtd 33 fr12b-B12_1231_ThankYou_Createacct
Landing page
[edit]9 fr12c-Poongothai-Balasubramanian 7 fr12c-Jimmy-Wales 6 fr12c-Thank-You 4 fr12c-Oarabile-Mudongo 4 fr12c-Ken-Thomas 3 fr12c-Amina-Duale 2 fr12c-Viswa-Prabha 2 fr12c-Ruben-Hilare-Quispe 2 fr12c-Mei-Jiun-Kwek 2 fr12c-Abhishek-Hingnikar 1 fr12c-Zinaida-Tebaykina 1 fr12c-Ravan-Jaafar-Altaie 1 fr12c-Peter-Gallert 1 fr12c-Noopur-Raval 1 fr12c-Ganesh-Paudel 1 fr12c-Deryck-Chan 1 fr12c-Benjamin-Chen 1 fr12c-Amit-Kapoor 1 fr12c-Adrianne-Wadewitz
Raw signup completion rates
[edit]| Referral | Raw account creation impressions | Account creations | Raw completion rate |
|---|---|---|---|
| fr12b-B12_1227_ThankYou_Short | 1517 | 202 | 13.3% |
| fr12b-B12_1227_ThankYou_5pillars | 1332 | 231 | 17.3% |
| fr12b-B12_1228_ThankYou_Nonprofit | 2145 | 260 | 12.1% |
| fr12b-B12_1228_ThankYou_Thrilled | 2298 | 292 | 12.7% |
| fr12b-B12_1231_ThankYou_Getstrtd | 623 | 38 | 6.1% |
| fr12b-B12_1231_ThankYou_Createacct | 288 | 33 | 11.4% |
| fr12c (landing pages) | 674 | 50 | 7.4% |
Notes
[edit]This data is interesting to compare to the results of our previous account creation experiments, where conversion rates were between 28-32% for both the control and test versions of a new account creation interface. It should not, however, be necessarily used to decide which of the two banners was more successful. To do so, the conversion rate would be account creations over total banner and landing page impressions.