June 23, 2011 Brandon & Ryan Appeal Tests

Last week, we tested a new appeal from WMF developer Ryan Lane against the Brandon Harris appeal we introduced in our previous test. We wanted to isolate the effect of the letter from the effect of the banner. We did that by using one banner with Brandon’s name and photo leading to two different appeals – Brandon’s original letter and Ryan's letter (both signed by Brandon).

Separately, we tested a Ryan banner-letter combo against a Brandon banner-letter combo. The pictures in the two banners were very different: For Ryan a plain head shot; For Brandon, a Rock and Roll Hall of Fame glam shot!

From these two tests we learned that:

Ryan's appeal letter is another breakthrough performer. It did about 85% as well as Brandon's, which is still far, far better than any non-Jimmy letter we've ever had. This further endorses the method we're now using of interviewing subjects on tape and then extracting an appeal made of their own thoughts and words.

Brandon's "rock and roll" photo used in the banner had a significantly higher click rate than Ryan's plain head shot. This suggests that we should spend more energy getting visually interesting, striking photos of subjects.

It's possible that the effects of Brandon's picture and his letter, when put together, were greater than the sum of their parts. In other words, the two variables of appeal and banner may not be independent variables. But our hour long test didn't quite give us large enough sample to say that with confidence.

For our upcoming tests, we’re working on personal appeals from editors and donors. We’re still refining the interviewing process, and will be sharing some basic interviewing principles that facilitate effective appeal writing for our fundraising purposes. For a more in depth explanation of our testing methodology, please see the 2010 Fundraiser Report.

Test Report Key

US Test

Brandon Original Appeal - L11_BR2_Junetest_1
Ryan Appeal text - L11_BR2_Junetest_2

EN Test

Brandon Banner - B_BRvRL_Junetest_BR_EN-L11_BRvRL_Junetest_BR
Ryan Banner - B_BRvRL_Junetest_BR_EN-L11_BRvRL_Junetest_RL
Brandon Appeal -L11_BRvRL_Junetest_BR
Ryan Appeal - L11_BRvRL_Junetest_RL

Meganhernandez 18:22, 29 June 2011 (UTC)[reply]

June 16, 2011 Brandon Appeal Test

During the 2010 campaign, we put a lot of effort into finding a compelling message to perform at the same levels as the appeal from our founder, Jimmy Wales. Throughout the fundraiser, we tested messages from editors around the world. While editor banners had similar click through rates as Jimmy banners, their letters performed about half as well as Jimmy’s letter for both number of donations and total amount.

Last week, we tested an appeal from Brandon Harris in the tech department at the Wikimedia Foundation against the Jimmy appeal. We’re extremely excited to report that Brandon’s appeal performed at nearly the same level as Jimmy’s appeal. This is the first time any message other than Jimmy’s has compelled readers to donate at this level.

Jimmy Banner

Brandon Banner

We actually conducted two tests simultaneously, one in the United States and the other on English Wikipedia, excluding the US and chapter countries. In both tests, Brandon’s banner had a slightly higher click through rate than Jimmy’s banner. This finding wasn’t too surprising, last year the editor banners also had similar click through rates as Jimmy banners. The big finding here is that Brandon’s message performed almost as well as Jimmy’s message. Combined with Brandon’s higher click through rate, Brandon actually brought in more donations than Jimmy with a higher rate of donations per banner impression.

So, why did Brandon perform so well? Brandon is incredibly passionate about Wikipedia and we tried something new with the process of writing the letter. With the editor appeals last year, editors wrote drafts and the fundraising team edited them to include some principles that performed well in previous letters. This time, we interviewed Brandon in person and recorded the words he actually spoke. We took out the most powerful parts from the interview and used the exact words he spoke in the letter about why Wikipedia is incredibly important. The message was authentic and Brandon’s passion for Wikipedia really came through.

For the 2011 fundraiser, we’ve brought in extra help to create new appeals so we can move away from running Jimmy's appeal for the majority of the fundraiser. The three Storytellers, Aaron, Victor, and Matthew, are interviewing a wide range of individuals with unique Wikipedia experiences to capture passionate messages that can be used in fundraising appeals.

The positive results from the Brandon test give us encouragement to find more appeals that can help break the fundraising dependence on the appeal from our founder. This breakthrough helps guide our direction as we work toward gathering and testing messages from readers, editors, donors, staff members, and more. Updates to come as we continue our weekly testing.

Key Findings

Brandon banner had a slightly higher click through rate than the Jimmy banner
Jimmy’s appeal performed slightly better than Brandon’s appeal in terms of donations per landing page view
Combining Brandon’s higher click through rate with his high donation rate, Brandon actually beat Jimmy in overall donations
Messages must authentically convey personal passion for Wikipedia

Test Reports

Please see the test reports below for detailed results.

Banner name JM_Junetest = Jimmy Banner

Banner name BR_Junetest = Brandon Banner

For a detailed explanation of our testing methodology, please see the 2010 Fundraiser Report.

Meganhernandez 20:43, 22 June 2011 (UTC)[reply]

June 3, 2011 Banner Size Test

In June, the fundraising team started testing the banner, landing page and analytics systems to be used in the 2011 fundraiser. We wanted to start testing several months before the fundraiser launches so that staff are well trained and systems function effectively during the fundraiser. This first testing session was really useful to start training more people to code banners and landing pages than last year.

During the 2010 fundraiser, we received several comments about the size of our banners, so we decided to start off our testing this year with an A/B test of banner size. We started with one of our banners from the last campaign and tested it in the US for one hour against a banner that was 67% smaller. These two banners were identical in message and style, with the only difference being the size of the banner. The larger banner had a height of 172px and the smaller banner had a height of 115px.

The resultｓshow that the larger banner had an increase of about 49% over the smaller banner, in terms of both number of donations and total amount raised. This shows that if we ran the smaller banner during the fundraiser, we would have to run the banners for nearly twice the time. In future tests on banner size, we'll want to use the larger banner as the starting point.

For our next round of testing, we're working on gathering an appeal from the WMF tech staff to test against the Jimmy appeal. The letter from our founder, Jimmy Wales, was the most effective message in the past fundraiser. This year, we'd like to work on other types of effective letters such as appeals from editors, readers, donors, staff, and more. Our donors support the technology and staff that keep Wikipedia running, so we definitely want to test an appeal that focuses on those key points. As we test, we'll continue posting regular updates and asking for your feedback.

For a recap on testing results from the last fundraiser, you can see the testing section of the 2010 Fundraiser Report