Research:Cloud Services Annual Survey/2017
The 2017 Toolforge survey collects feedback from Toolforge developers and maintainers on how the services offered can be improved to help their development and maintenance needs.
Participants
[edit]The participants of the survey are the members of the Wikimedia Toolforge project. 1286 participants were contacted via email. The emails were harvested from the Cloud Services' LDAP system and the Wikitech database. The final list was created by finding the intersection of the members of the Toolforge project gathered from LDAP and the Wikitech users with confirmed email addresses who had not opted-out of email contact by other users in their Wikitech preferences. Specific opt-outs for the Annual Survey can be made by users at wikitech:Annual Toolforge Survey/Opt out.
141 email recipients participated in the survey which represents 11% of those who were contacted.
Dates
[edit]The 2017 survey was active between 2017-11-20 and 2017-12-01.
Questions and answers
[edit]Basic demographics
[edit]How many years have you used Toolforge?
[edit]Help: If you were a Toolserver user, please include the total number of years that you have used Toolserver and Toolforge.
| Cohort | Percentage |
| 1 Year | 25.18% |
| 2-3 Years | 31.65% |
| 4+ Years | 43.17% |
How many tools have you developed in Toolforge?
[edit]Help: We do not have a canonical definition of "tools". We are trying to estimate how much burden is on you, so please use your best judgement in assessing how many tools you have developed.
| Cohort | Percentage |
| 0 Tools | 8.57% |
| 1 Tool | 28.57% |
| 2-3 Tools | 35.71% |
| 4+ Tools | 27.14% |
How many tools do you actively maintain in Toolforge?
[edit]Help: We do not have a canonical definition of "tools". We are trying to estimate how much burden is on you, so please use your best judgement in assessing how many tools you maintain.
| Cohort | Percentage |
| 0 Maintain | 15.83% |
| 1 Maintain | 33.09% |
| 2 Maintain | 23.02% |
| 3+ Maintain | 28.06% |
How many hours per week do you spend developing or maintaining tools on Toolforge?
[edit]
| Cohort | Percentage |
| 0 Hours | 22.96% |
| 1 Hour | 31.85% |
| 2-8 Hours | 36.30% |
| 9+ Hours | 8.89% |
Were you a developer and/or maintainer in the Toolserver as well?
[edit]
| Cohort | Percentage |
| No Toolserver | 59.57% |
| Toolserver | 40.43% |
Additional demographics
[edit]What programming languages do you prefer to program in?
[edit]For this question, the participants had the following choices: Perl, PHP, Python 2, Python 3, Ruby, NodeJS, or they could enter the language of their choice. Note that participants could report more than one language.
| Language | Percentage |
| Python3 | 57.45% |
| PHP | 44.68% |
| Python2 | 36.88% |
| NodeJS | 22.70% |
| Perl | 8.51% |
| Java | 7.09% |
| Ruby | 7.09% |
| Other | 16.31% |
Languages with fewer than 10 mentions were rolled into the "Other" category.
When you develop a tool, what percentage of your work is done locally (as opposed to remotely on Toolforge)?
[edit]Help: For example, enter 30 if 30% of your development work is done locally.
| Answer | Percentage |
| 0 | 4.35% |
| 10 | 7.25% |
| 20 | 5.80% |
| 25 | 2.17% |
| 30 | 2.17% |
| 40 | 1.45% |
| 50 | 10.14% |
| 60 | 5.80% |
| 70 | 4.35% |
| 75 | 2.17% |
| 80 | 10.14% |
| 85 | 4.35% |
| 90 | 15.94% |
| 95 | 3.62% |
| 100 | 20.29% |
Do you prefer using source control to manage your tool's source code?
[edit]
| Yes | No |
| 113 | 27 |
Qualitative questions
[edit]A series of qualitative questions were asked in order to gauge general satisfaction with Toolforge.
Services have high reliability (up time)
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 89.86% | 82.35% | 93.02% | 91.80% | 70.00% | 92.50% | 90.00% | 92.11% | 77.27% | 93.33% | 90.63% | 92.31% | 85.71% | 92.86% | 89.80% | 91.67% | 90.12% | 89.47% |
| Disagree/Strongly Disagree | 6.52% | 8.82% | 4.65% | 6.56% | 0.00% | 2.50% | 10.00% | 7.89% | 4.55% | 4.44% | 9.38% | 7.69% | 5.71% | 2.38% | 10.20% | 8.33% | 4.94% | 8.77% |
It is easy to write code and have it running on Toolforge
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 77.70% | 70.59% | 79.07% | 80.65% | 54.55% | 70.00% | 76.00% | 94.74% | 52.17% | 75.56% | 78.13% | 94.87% | 66.67% | 76.19% | 85.71% | 83.33% | 74.39% | 82.46% |
| Disagree/Strongly Disagree | 18.71% | 20.59% | 18.60% | 17.74% | 18.18% | 25.00% | 24.00% | 5.26% | 30.43% | 22.22% | 21.88% | 5.13% | 25.00% | 19.05% | 14.29% | 16.67% | 20.73% | 15.79% |
I feel I am supported by the Toolforge team when I contact them via cloud@lists.wikimedia.org, #wikimedia-cloudconnect IRC channel, or Phabricator.
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 58.65% | 53.13% | 52.38% | 66.10% | 40.00% | 48.65% | 58.33% | 73.68% | 38.10% | 52.38% | 61.29% | 74.36% | 54.55% | 43.90% | 70.21% | 75.00% | 57.14% | 60.71% |
| Disagree/Strongly Disagree | 5.26% | 3.13% | 2.38% | 8.47% | 10.00% | 5.41% | 2.08% | 7.89% | 4.76% | 4.76% | 3.23% | 7.69% | 3.03% | 7.32% | 6.38% | 0.00% | 2.60% | 8.93% |
I receive useful information via cloud-announce / cloud mailing list.
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 58.96% | 36.36% | 56.10% | 73.33% | 33.33% | 41.03% | 63.27% | 78.38% | 33.33% | 48.84% | 68.75% | 76.32% | 43.75% | 53.66% | 69.39% | 75.00% | 57.14% | 61.40% |
| Disagree/Strongly Disagree | 11.19% | 12.12% | 9.76% | 11.67% | 0.00% | 10.26% | 12.24% | 13.51% | 14.29% | 9.30% | 12.50% | 10.53% | 18.75% | 9.76% | 10.20% | 0.00% | 7.79% | 15.79% |
Documentation is easy-to-find
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 52.21% | 55.88% | 53.49% | 49.15% | 44.44% | 45.00% | 57.14% | 55.26% | 28.57% | 62.22% | 51.61% | 53.85% | 41.18% | 52.38% | 56.25% | 66.67% | 53.16% | 50.88% |
| Disagree/Strongly Disagree | 41.18% | 35.29% | 39.53% | 45.76% | 22.22% | 42.50% | 42.86% | 42.11% | 47.62% | 33.33% | 45.16% | 43.59% | 41.18% | 42.86% | 41.67% | 33.33% | 37.97% | 45.61% |
How do you compare the support you received when using Toolserver with the support you receive when using Toolforge?
[edit]Only the participants who had indicated previous Toolserver use were shown this question. Therefore, the responses are based on this subset of participants.
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | |
| Better | 54.55% | 33.33% | 14.29% | 64.29% | 100.00% | 40.00% | 50.00% | 62.50% | 40.00% | 38.46% | 46.67% | 72.73% | 44.44% | 31.58% | 72.73% | 80.00% |
| As good | 41.82% | 66.67% | 71.43% | 33.33% | 0.00% | 60.00% | 50.00% | 29.17% | 60.00% | 61.54% | 46.67% | 22.73% | 33.33% | 68.42% | 27.27% | 20.00% |
| Worse | 3.64% | 0.00% | 14.29% | 2.38% | 0.00% | 0.00% | 0.00% | 8.33% | 0.00% | 0.00% | 6.67% | 4.55% | 22.22% | 0.00% | 0.00% | 0.00% |
Toolforge documentation is comprehensive.
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 50.38% | 44.12% | 58.54% | 48.28% | 44.44% | 39.47% | 69.39% | 37.84% | 30.00% | 63.64% | 54.84% | 42.11% | 43.75% | 48.78% | 54.17% | 58.33% | 53.25% | 46.43% |
| Disagree/Strongly Disagree | 39.10% | 38.24% | 26.83% | 48.28% | 11.11% | 42.11% | 28.57% | 56.76% | 35.00% | 29.55% | 38.71% | 52.63% | 34.38% | 39.02% | 41.67% | 41.67% | 31.17% | 50.00% |
Toolforge documentation is clear.
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 52.59% | 52.94% | 57.14% | 49.15% | 44.44% | 43.59% | 69.39% | 42.11% | 33.33% | 63.64% | 61.29% | 43.59% | 41.18% | 51.22% | 56.25% | 75.00% | 51.28% | 54.39% |
| Disagree/Strongly Disagree | 39.26% | 38.24% | 30.95% | 45.76% | 22.22% | 41.03% | 30.61% | 52.63% | 38.10% | 31.82% | 35.48% | 51.28% | 38.24% | 41.46% | 41.67% | 25.00% | 38.46% | 40.35% |
Free form responses
[edit]The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form. The free form responses include comments broadly falling into these categories:
- Documentation (58 comments)
- Platform (48 comments)
- Workflow (48 comments)
- Community (17 comments)
- Support (6 comments)
Documentation
[edit]Comments on documentation included both positive recognition of work that has been done to improve our docs and areas that are still in need of additional work. Areas with multiple mentions include need for increased discoverability of current information, better getting started information, and more in depth coverage of topics such as wiki replica usage, Kubernetes, and job grid usage.
There were also comments asking for a self-service version control system and pre-installed pywikibot software. Both of these are offered currently in Toolforge, so these comments were classified as missing or difficult to find documentation.
Platform
[edit]Comments about the Toolforge platform have been subcategorized as follows:
- Software (26 comments)
- The majority of software comments were related to a desire for newer language runtime versions (PHP, Java, nodejs, Python) and more flexibility in the Kubernetes environment.
- Database (10 comments)
- Database comments include praise for the new Wiki Replica servers and multiple requests for a return of user managed tables colocated with the replica databases.
- Reliability (10 comments)
- Reliability comments included praise for good uptime, complaints of poor uptime, and requests to improve limits on shared bastion systems.
- Hardware (2 comments)
- (sample too small to summarize)
Workflow
[edit]- Deploy (12 comments)
- The major theme here was automation for software deployment including requests for full continuous delivery pipelines.
- Debugging (10 comments)
- People asked for better debugging tools and a way to create a more full featured local development environment.
- Monitoring (10 comments)
- Monitoring comments included a desire for alerting based on tracked metrics and tracking of (more) metrics for each tool.
- Setup (10 comments)
- Comments included praise for https://toolsadmin.wikimedia.org/ improvements and other work done in last year.
- Files (6 comments)
- Improved workflows for remote editing and file transfer are desired.
Community
[edit]Comments classified as community related broadly called for more collaboration between tool maintainers and better adherence to practices that make accessing source code and reporting bugs easier.
Support
[edit]Support related comments praised current efforts, but also pointed to confusion about where to ask questions (irc, email, phabricator).