Learning patterns/Asking the right questions

Asking the right questions

problemYou need to make sure that you don't accidentally leave important questions out of your survey.

solutionUse different types of survey questions to ask for different kinds of information. To make sure you cover all of your bases, frame your survey around three or four types of question: background questions, activity questions, experience questions, and opinion questions.

creator• Jtmorgan

discuss

endorse

What problem does this solve?

The most important step in performing a survey is figuring out what it is you want to learn. This sounds obvious, but many researchers realize after they have already deployed their survey that they failed to ask an important question. If you want to know how many people on different Wikimedia projects subscribe to public mailing lists, you will need to ask people which project they currently contribute to most right now. If you want to know how many people migrate from one project to another during their editing careers, you need to ask which project they joined first and which one they contribute to most right now.

What is the solution?

To make sure you cover all of your bases, frame your survey around three or four types of question:

background questions
activity questions
experience questions
opinion questions

You don't need to have an equal number of questions in these four categories, and you might not need them all, but many surveys will be composed of a mix of all four types.

Background questions

Questions about the participant themselves: who they are, where they're from, what they do off Wikipedia, etc.

Example

Teahouse is intended to be a friendly, supportive environment for all new Wikipedians, but we make a special effort to welcome female editors, who are underrepresented within the Wikipedia community. If you're comfortable telling us, what is your gender? (Teahouse Pilot project guest survey)

Activity questions

Questions about what they do now, or have done in the past.

Example

Have you ever sought help on any of the following topics on Wikipedia? (Help Project survey)

Experience questions

Questions about how they feel (or felt), or what they think, about the specific activities they participated in.

Example

Was there anything in particular you liked about the review process or the feedback? Tell us about it. (IEG submitter survey)

Opinion questions

Questions about what they think in general about the subject you're interested in, beyond their own specific experiences.

Example

What ideas do you have to make dispute resolution overall more effective and efficient? (Dispute resolution survey)

Considerations

When deciding which questions to ask, and how many, it is also helpful to keep these considerations in mind:

Keep your survey as short as possible. People have short attention spans. If you ask them to fill out a 50 question survey, many of them will get bored or frustrated and quit partway through, leaving you with gaps in your data and lingering ill will. Don't ask every possible question; figure out which questions are necessary and only ask those. Realistically, keep the length of time it takes to fill out your survey (in the amount of detail you desire) under 10 minutes.
Make multiple choice answers required and free text answers optional. No one likes being forced to write. Making basic questions multiple choice allows respondents to give you critical information quickly and easily. Making answering multiple choice questions required to finish the survey (and most online survey tools will allow this) gets you good data, as long as your questions are clear and the choices you provide make sense. Free response questions can be very useful, and most surveys will contain at least a few. If people have strong opinions about something, you can be sure that they will take advantage of this—no need to force them.
Use opinion questions sparingly. Opinion questions are useful for some surveys, but in many cases it is better to ask primarily Experience questions and Activity instead--people's opinions don't always reflect their behaviors!^[1]

Examples

Example background questions

When did you start editing (year)?	Former contributors survey
How did you find out about the IEG program?	IEG reviewer survey
Teahouse is also intended to provide a friendly, supportive environment for editors from all nations and cultures. If you're comfortable telling us, what country do you live in?	Teahouse Pilot project guest survey
How old are you?	survey from Brochure "Editing Wikipedia" Chinese localization

Example activity questions

Today, how often do you participate in resolving disputes?	Dispute resolution survey
Have you interacted with any other Teahouse guests outside of the Q&A board? If so, please briefly describe one of these interactions (details like who, where, and what happened are great, if you want to provide them!)	Teahouse Pilot project guest survey
Did you participate in the IdeaLab?	IEG submitter survey
How does the event use the brochure?	survey from Brochure "Editing Wikipedia" Chinese localization

Example experience questions

In your average dispute, what techniques do you use to resolve it?	Dispute resolution survey
Did the difficulty of the work have an impact on your decision to stop contributing? (select up to three reasons)	Former contributors survey
What are some things that have been challenging for you on Wikipedia? (check all that apply)	Teahouse Pilot project guest survey
The brochure is suitable for self-study (scale from 1 to 5, 1 is totally disagree, and 5 is totally agree)	survey from Brochure "Editing Wikipedia" Chinese localization

Example opinion questions

Finally, do you have any other comments on Wikipedia's help pages?	Help Project survey
Do you think you will participate in the Teahouse in the future? (for instance, by asking or answering questions, or by creating or browsing host & guest intro boxes)	Teahouse Pilot project guest survey
Please tell us about anything you can think of in the review guidelines that should be clarified, improved, or added.	IEG submitter survey
According to the previous question, why do you think the brochure is suitable / not suibable for discussion usage?	survey from Brochure "Editing Wikipedia" Chinese localization

Endorsements

J-mo
Rastus Vernon
Libcub
Reke
Liang(WMTW)
We conducted a survey in South Africa, which was the first of it's kind, and we felt it very important to ask the right questions so that we can find out more about our community and how we can support them. Humetheresa (talk) 09:29, 23 July 2015 (UTC)[reply]