Wiki Education Foundation/Quarterly reviews/2014-Q2 Digital Infrastructure

Digital Infrastructure Quarterly Review, 1 December 2014.pdf

The following are notes from the Quarterly Review Meeting for the Wiki Education Foundation's Product Manager Digital Services Sage Ross on December 1, 2014

Present: LiAnna, Jami, Frank, Samantha, Sara, Helaine, Bill, Eryk
Participating Remotely: Sage, Ian, Adam

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases, and other official material.

Introduction

Frank: Why we're here. Don’t want people to work in silos. This is a good opportunity for feedback and to flag roadblocks etc.

Agenda (Sage)

Assignment Design Wizard and our work with WINTR

Launched the Assignment Design Wizard - program now live and useable for professors. The project can be considered a success because it met our criteria that

• it would be an improvement over the old system
• we would begin directing professors to use the new product
• built fully functional system by the deadline

Product Walk Through

• Not yet live (available on wizard-testing.wikiedu.org): ability to specify course dates, meeting times, length, days off etc for professors
• Assignment type: professors get the chance to choose what type of assignment they will assign (research and write an article, for example) or have the opportunity to choose a la carte options like assigning students to copyedit or upload an image.
• Milestones: milestones can be made mandatory or optional
• Selecting Student Assignments: would the professor like to prepare the list of student assignments themselves or would they like the students to find their topics themselves? Also an option for the professor flag that they would like to contact Adam and/or Ian for support.
• Outline: professors are given the choice to choose between a traditional classroom outline and a "Wikipedia Lead Article" style outline. Previously piloted the Wikipedia Lead outline and had success, as it helped familiarize students with Wiki formatting from the start.
• Drafts and Mainspaces: Asks instructors if they would like their students to work live or from their sandboxes.
  • Discussion: Frank asked how many students we had working from sandboxes and if this was something that we preferred? Ian guesses that 60% of students begin working in a sandbox, and LiAnna offered that professors have different reasons for using one over the other and there is not a one-size-fits-all option. Will table for further programs team discussion.
• Peer Feedback: asks professors if student editors will review each others work. Suggests that students review each others work twice throughout the semester, but there is a scale from 1-5 available for selection. Professors also receive suggestions for managing collaboration among students.
• Supplementary Assignments: what other items will be included in the course plan?
  • Examples: reflective essay, in class presentation, blog, student portfolio of work, original analytical paper (usually grad school option)
  • Discussion: Frank asked if we could track this data and see how often professors include supplemental assignments and which type. Sage responded that yes this was possible but is not currently being done. He will consider options for implementation.
• DYK & Good Article Nominations: informs instructor what these terms represent and gives option to include as ungraded option in the course assignment
• Grading: allows professors to change the grading scale for each portion of the assignment and has advice on grading
• Assignment Overview: allows professor to review all previously made decisions and edit each one individually. Also asks them to write a description of the course for the Wiki community and then prompts them to publish their selections to their course page as an assignment plan.
• Discussion
  • Professors must do this in one sitting, cannot begin a course design and return later.
  • Sara asked what our measure of success is. Sage/Frank/LiAnna: Main measure of success is that instructors find it useful and see the wizard as an improvement over prior ways of getting assistance. Also measuring the quality of assignments - we will be gathering data each semester on if the course syllabi and resulting student assignments are good or bad. Will also measure how returning users rate the tool. Finally, we'll hope to see a significant lower workload on Jami and Helaine who help professors with assignment design and course pages.
  • December: so far about 5-8 professors have used it - the tool is ready for next semester!
  • Frank: This project matters because it shows we can successfully complete large, effective projects on time.
  • Frank: This project also shows that working successfully with outside developers is possible - we know we can involve those unfamiliar with OAuth and other aspects of Wikicoding and successfully develop products and expansions.
  • Frank: Wikimedia wants individual chapters and organizations to get into the software development game - having a variety of approaches to this (in house versus outside) is a good learning for other Wikimedia organizations as well.

Problems and Struggles

• Project came together only at the very end
• Took too long to create prototype, didn't have time for user testing until first phase of development was done
• In the future should focus on building one feature at a time. Development team took on too many tasks at once which resulted in unclear expectations. Sage will grow his capacity to communicate clear expectations and a future of incremental, testable changes into project timelines.
• Other roadblocks: unexpected delays with OAuth system. The system however doesn't have a large enough user base to have worked out all these bugs yet.

Learning Points

• Make sure user stories are focused on both plumbing tasks (ie, ensuring a solid fundamental framework) and user interface features
• Begin testing in ways that don’t depend on live prototype - create low-fi mockups, ask users to react, can be done without design or development, if we just want to test our assumptions on how users will use these tools
• Discussion
  • Sara: Do we have instructors available to test new features as they became available?
  • Sage / Jami: No, were connected with our target group of NEW instructors through Jami & Helaine as necessary

Overall Evaluation, Action Items & Discussion

• Project was a success: resulted in useable product which far exceeds old ways of providing assistance prior to each term.
• Very pleased with the design work; the UX is excellent and makes it easy for instructors to use the tool.
• Didn't make it quite as far down the project road map as we hoped, but overall feel really good about the timeline and output. Software development is often hard to scope for accurate timelines.
• WINTR is a good candidate for being a long-term partner.
• Code for this software is available under a free license on Github so others can re-use it.
• Will use the product for one term with new and current professors before scaling outward.

Our future work on course dashboards

• Dashboards are our first step toward a new course page system
• Our primary goal is to make grading easier, but the new dashboard will also help with collecting data
• Data
  • Student Contributions (in characters added)
  • Articles touched (cumulative for class)
  • Page views (cumulative for class)
• For instructors and Wiki Ed this means we can see
  • how many students are editing and when
  • what they are editing
  • what the impact has been
• The same numbers are collated system-wide for entire courses, and then totalled across entire system, making large data collection on these data points easy.
• WINTR will be consulted on how to collate & present this information well among other things
• Main target group is WikiEd & instructors but the hope is that these tools will be available for students to see and the community to see later down the road - first priority is making grading easier
• For individual course pages, our target audience is instructors
• For system wide stuff, we are the target audience

Discussion

• Jami: Love how useful this will be for instructor retention. Can we sort across disciplines (i.e. sociology courses?
  • Sage: eventually system-wide dashboard would be able to zoom in on specific groups of courses, but first priority is to get course specific dashboard
• LiAnna: starting this as small project until we have expanded funding. A large part of up front cost is design, how we want to store the data - new ways to pull and visualize the data will come later when project grows
• Frank: all this data will be on Wikimedia servers?
  • Sage: Data is coming from various locations, preferably Wikimedia, and would be pulled periodically and ready to release to users on a timeline we agree upon. The pulled information would be stored with us.
• Jami: where does the dashboard live?
  • Sage: links as part of header on course pages that will take you to course pages, will live on wikiedu.org, each class will have a url to their dashboard

First round of dashboards should be live for the spring term 2015 - hopefully in February when students are first beginning to edit. Goal is for editing to be on Wikipedia but everything else course related to live on WikiEdu.org

Updates on plagiarism prevention

Our goal is to automatically flag plagiarism or anything that looks like plagiarism in a students sandbox and alert the student and professor before it goes live. Can this be done?

• This is feasible, but how much will it cost?
• James Heilman's bot is doing a successful job of this but with huge amounts of labor and volunteers
• LiAnna: Plagiarism study showed that students plagiarize, more than experienced editors and less than other new editors. Want a solution based model. Sage will continue to work on this.

Open source software and mentorships

Sage is mentoring an intern from Germany who will work on graphic design for open source projects. There is a global need for funding research for improved course page system - we can hopefully incorporate some of this work into our future plans. Thanks Sage for staying connected to the community!

Long term tech strategies

• #1 longterm goal is developing our own course page system on wikiedu.org - this will be fleshed out soon
• The rest of our goals will be based on prioritizing needs
• Ideas
  • Q&A platform (curated with a database of answers)

Discussion

Jami: When will just-in-time system be ready? Sage: Requires knowing when students arrive at certain steps, i.e., course page system must be prepped and running. Tough to estimate that - and depends on funding as well. (<2 years rough guesstimate.)

Frank: What can we do to make work/life easier? Sage: Nothing in particular. Better webcam + Mic could be useful.

Sara: How is general IT/Admin stuff? Salesforce etc? Sage: Not too bad. Setting up e-mail forwards from expired domains; a few admin projects re: office wiki have been backburnered but will get to.

Frank: Thanks to Sage for all his hard work on this. The result is outstanding.

LiAnna: Writing the content for the assignment design wizard was mostly Sage, part of the broader project that may go overlooked. Re: the internship you also see Sage reaching out through the tech world to find opportunities. Sage has also been an excellent partner with outside organizations.