Research:Cloud Services Annual Survey/2015
The 2015 Tool Labs survey collected feedback from Tool Labs' developers and maintainers on how the services offered can be improved to help their development and maintenance needs.
Participants
[edit]The participants of the survey were the members of the Wikimedia Tool Labs. 962 participants were contacted via email. The emails were harvested from Labs' LDAP system, which is available to everyone with a labs account. The list initially involved 977 participants. All those who had opted out of contacts via Wikitech were eventually excluded.
106 email recipients participated in the survey which represents 11% of those who are contacted.
Dates
[edit]The 2015 survey was active between 2015-09-25 and 2015-10-08.
Questions and answers
[edit]Basic demographics
[edit]Responses to the basic demographic questions were divided into roughly equal sized groups to provide "cohorts" of similar Tool Labs users. These groups were then plotted against each other in the qualitative responses to determine if there were significant differences in the opinions of the sub-groups.
How many years have you used Tool Labs?
[edit]Help: If you were a Toolserver user, please include the total number of years that you have used Toolserver and Tool Labs.
| Cohort | Percentage |
| 1 Year | 29.81% |
| 2-3 Years | 36.54% |
| 4+ Years | 33.65% |
Many of the respondents have been using Tool Labs for 2 or fewer years. In fact, nearly 30% of the respondents are using Tool Labs only for a year. This can be the sign of many young users or that young users are the ones that are more willing to participate in surveys. The distribution has a long tail indicating there are quite a few users who have been around for a long time.
How many tools have you developed in Tool Labs?
[edit]Help: We do not have a canonical definition of "tools". We are trying to estimate how much burden is on you, so please use your best judgement in assessing how many tools you have developed.
| Cohort | Percentage |
| 0 Tools | 8.57% |
| 1 Tool | 31.43% |
| 2-3 Tools | 28.57% |
| 4+ Tools | 31.43% |
40% of respondents have developed 0-1 tools. This may correlate with the amount of time that they have been active on Tool Labs. The distribution of the 4+ Tools cohort again has a long tail indicating that some tool developers are highly prolific.
How many tools do you actively maintain in Tool Labs?
[edit]Help: We do not have a canonical definition of "tools". We are trying to estimate how much burden is on you, so please use your best judgement in assessing how many tools you maintain.
| Cohort | Percentage |
| 0 Maintain | 16.19% |
| 1 Maintain | 27.62% |
| 2 Maintain | 26.67% |
| 3+ Maintain | 29.52% |
The maintenance numbers roughly correlate with the number of Tools developed although the raw numbers are skewed downward slightly. This seems to suggest that very few respondents have become maintainers of tools that they didn't build themselves and also possibly indicates that some tools have been abandoned or replaced over time.
How many hours per week do you spend developing or maintaining tools on Tool Labs?
[edit]
| Cohort | Percentage |
| 0 Hours | 24.51% |
| 1 Hour | 34.31% |
| 2-8 Hours | 29.41% |
| 9+ Hours | 11.76% |
Over 50% of respondents spend an hour or less per week on average working on their tools.
Were you a developer and/or maintainer in the Toolserver as well?
[edit]
| Cohort | Percentage |
| No Toolserver | 59.43% |
| Toolserver | 40.57% |
The majority of survey responses were received from users with no history using Toolserver. It is currently unknown how this distribution correlates with the full number of active Tool Labs users.
Additional demographics
[edit]What programming languages do you prefer to program in?
[edit]For this question, the participants had the following choices: Perl, PHP, Python 2, Python 3, Ruby, or they could enter the language of their choice. The following table shows the ranked list of languages in terms of the percentage of time they were chosen as the language of choice. Note that participants could report more than one language.
| Language | % |
| PHP | 58.49% |
| Python3 | 37.74% |
| Python2 | 36.79% |
| JavaScript | 9.43% |
| Perl | 8.49% |
| Ruby | 7.55% |
| Other | 7.55% |
| Java | 6.60% |
| C# | 4.72% |
High usage of PHP and Python were not unexpected results. MediaWiki itself is written in PHP and Python is both widely used for data analysis tasks and well supported by the Pywikibot community. Freeform responses were normalized to make this analysis easier. The JavaScript numbers include responses of "JavaScript", "node", and "nodejs". The "Other" grouping includes several common scripting tools and a few very esoteric languages.
When you develop a tool, what percentage of your work is done locally (as opposed to remotely on Tool Labs)?
[edit]Help: For example, enter 30 if 30% of your development work is done locally.
Answers were rounded up to the nearest 5% for analysis.
| Answer | Percentage reported |
|---|---|
| 0 | 8.65% |
| 5 | 3.85% |
| 10 | 9.62% |
| 20 | 2.88% |
| 25 | 0.96% |
| 30 | 3.85% |
| 50 | 9.62% |
| 60 | 3.85% |
| 70 | 10.58% |
| 75 | 2.88% |
| 80 | 16.35% |
| 85 | 1.92% |
| 90 | 9.62% |
| 95 | 5.77% |
| 100 | 9.62% |
Do you prefer using source control to manage your tool's source code?
[edit]
| Yes | No |
|---|---|
| 73% | 26% |
Qualitative questions
[edit]A series of qualitative questions were asked in order to gauge general satisfaction with Tool Labs. Since this was the first survey of this type given to the Tool Labs community these results will form a historical baseline to compare the results future surveys against.
Services have high reliability (up time)
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 63.81% | 67.74% | 54.05% | 70.27% | 70.00% | 65.63% | 76.67% | 48.48% | 66.67% | 65.52% | 70.37% | 54.84% | 68.97% | 57.14% | 63.33% | 72.73% | 68.25% | 57.14% |
| Disagree/Strongly Disagree | 34.29% | 29.03% | 43.24% | 29.73% | 20.00% | 31.25% | 23.33% | 51.52% | 22.22% | 34.48% | 29.63% | 45.16% | 24.14% | 42.86% | 36.67% | 27.27% | 28.57% | 42.86% |
It is easy to write code and have it running on Tool Labs
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 68.57% | 61.29% | 67.57% | 75.68% | 50.00% | 53.13% | 73.33% | 84.85% | 55.56% | 58.62% | 74.07% | 80.65% | 51.72% | 77.14% | 66.67% | 90.91% | 66.67% | 71.43% |
| Disagree/Strongly Disagree | 24.76% | 25.81% | 27.03% | 21.62% | 0.00% | 40.63% | 26.67% | 15.15% | 16.67% | 34.48% | 25.93% | 19.35% | 31.03% | 22.86% | 30.00% | 0.00% | 23.81% | 26.19% |
I feel I am supported by the Tool Labs team when I contact them via labs-l, wikimedia-labs IRC channel, or phabricator.
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 66.67% | 54.84% | 72.97% | 70.27% | 60.00% | 53.13% | 73.33% | 75.76% | 50.00% | 55.17% | 81.48% | 74.19% | 55.17% | 68.57% | 70.00% | 81.82% | 65.08% | 69.05% |
| Disagree/Strongly Disagree | 10.48% | 16.13% | 5.41% | 10.81% | 0.00% | 18.75% | 13.33% | 3.03% | 11.11% | 20.69% | 3.70% | 6.45% | 17.24% | 2.86% | 16.67% | 0.00% | 7.94% | 14.29% |
I receive useful information via labs-announce / labs-l mailing list.
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 52.88% | 40.00% | 54.05% | 62.16% | 20.00% | 45.16% | 56.67% | 66.67% | 17.65% | 44.83% | 74.07% | 61.29% | 32.14% | 48.57% | 63.33% | 90.91% | 49.21% | 58.54% |
| Disagree/Strongly Disagree | 16.35% | 6.67% | 18.92% | 21.62% | 10.00% | 9.68% | 13.33% | 27.27% | 11.76% | 13.79% | 11.11% | 25.81% | 10.71% | 17.14% | 26.67% | 0.00% | 11.11% | 24.39% |
Documentation is easy-to-find
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 45.71% | 58.06% | 43.24% | 37.84% | 50.00% | 43.75% | 43.33% | 48.48% | 44.44% | 51.72% | 40.74% | 45.16% | 55.17% | 45.71% | 26.67% | 72.73% | 50.79% | 38.10% |
| Disagree/Strongly Disagree | 51.43% | 35.48% | 56.76% | 59.46% | 40.00% | 53.13% | 56.67% | 48.48% | 50.00% | 44.83% | 55.56% | 54.84% | 41.38% | 54.29% | 66.67% | 27.27% | 44.44% | 61.90% |
How do you compare the support you received when using Toolserver with the support you receive when using Tool Labs?
[edit]Only the participants who had indicated previous Toolserver use were shown this question. Therefore, the responses in the following table are based on this subset of participants. This was an optional question, and 4% of the participants chose not to respond.
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Better | 34.15% | 33.33% | 0.00% | 41.94% | 0.00% | 12.50% | 50.00% | 31.58% | 33.33% | 28.57% | 46.15% | 27.78% | 50.00% | 31.25% | 35.71% | 0.00% | 0.00% | 34.15% |
| As good | 36.59% | 0.00% | 42.86% | 38.71% | 0.00% | 25.00% | 35.71% | 42.11% | 33.33% | 14.29% | 23.08% | 55.56% | 12.50% | 37.50% | 50.00% | 33.33% | 0.00% | 36.59% |
| Worse | 29.27% | 66.67% | 57.14% | 19.35% | 0.00% | 62.50% | 14.29% | 26.32% | 33.33% | 57.14% | 30.77% | 16.67% | 37.50% | 31.25% | 14.29% | 66.67% | 0.00% | 29.27% |
Tool Labs documentation is comprehensive.
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 46.67% | 54.84% | 43.24% | 43.24% | 60.00% | 40.63% | 46.67% | 48.48% | 50.00% | 44.83% | 40.74% | 51.61% | 51.72% | 51.43% | 33.33% | 54.55% | 49.21% | 42.86% |
| Disagree/Strongly Disagree | 49.52% | 41.94% | 54.05% | 51.35% | 20.00% | 56.25% | 53.33% | 48.48% | 38.89% | 51.72% | 55.56% | 48.39% | 37.93% | 48.57% | 63.33% | 45.45% | 46.03% | 54.76% |
Tool Labs documentation is clear.
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 49.52% | 58.06% | 37.84% | 54.05% | 50.00% | 46.88% | 43.33% | 57.58% | 38.89% | 48.28% | 51.85% | 54.84% | 58.62% | 45.71% | 43.33% | 54.55% | 49.21% | 50.00% |
| Disagree/Strongly Disagree | 47.62% | 38.71% | 59.46% | 43.24% | 30.00% | 53.13% | 56.67% | 39.39% | 50.00% | 51.72% | 48.15% | 41.94% | 34.48% | 51.43% | 56.67% | 45.45% | 47.62% | 47.62% |
Service usage
[edit]A section of questions on the frequency of use of various services offered by Tool Labs was included in the survey. The responses to these questions may be used to prioritize support and/or new service development by the Tool Labs administrators. These questions will not necessarily appear in future surveys.
How often have you used LabsDB access?
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Often | 44.90% | 29.63% | 47.06% | 54.05% | 11.11% | 44.44% | 43.33% | 56.25% | 12.50% | 44.00% | 51.85% | 56.67% | 28.00% | 38.24% | 67.86% | 45.45% | 39.66% | 52.50% |
| Seldom/Sometimes | 32.65% | 25.93% | 35.29% | 35.14% | 44.44% | 18.52% | 33.33% | 40.63% | 43.75% | 32.00% | 29.63% | 30.00% | 40.00% | 35.29% | 25.00% | 27.27% | 32.76% | 32.50% |
| Never | 18.37% | 37.04% | 11.76% | 10.81% | 22.22% | 33.33% | 20.00% | 3.13% | 25.00% | 24.00% | 18.52% | 10.00% | 20.00% | 23.53% | 7.14% | 27.27% | 24.14% | 10.00% |
How often have you used Webservices?
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Often | 59.60% | 46.43% | 67.65% | 62.16% | 11.11% | 59.26% | 53.33% | 78.79% | 35.29% | 64.00% | 57.69% | 70.97% | 52.00% | 64.71% | 62.07% | 54.55% | 55.17% | 65.85% |
| Seldom/Sometimes | 27.27% | 32.14% | 20.59% | 29.73% | 55.56% | 18.52% | 33.33% | 21.21% | 35.29% | 12.00% | 38.46% | 25.81% | 28.00% | 26.47% | 31.03% | 18.18% | 31.03% | 21.95% |
| Never | 11.11% | 14.29% | 11.76% | 8.11% | 11.11% | 22.22% | 13.33% | 0.00% | 17.65% | 24.00% | 3.85% | 3.23% | 12.00% | 8.82% | 6.90% | 27.27% | 10.34% | 12.20% |
How often have you used Cronjobs?
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Often | 53.61% | 53.57% | 54.55% | 52.78% | 11.11% | 40.74% | 73.33% | 58.06% | 11.76% | 52.00% | 73.08% | 62.07% | 20.83% | 54.55% | 72.41% | 72.73% | 56.14% | 50.00% |
| Seldom/Sometimes | 25.77% | 25.00% | 21.21% | 30.56% | 22.22% | 25.93% | 20.00% | 32.26% | 29.41% | 20.00% | 23.08% | 31.03% | 29.17% | 30.30% | 24.14% | 9.09% | 22.81% | 30.00% |
| Never | 18.56% | 14.29% | 24.24% | 16.67% | 44.44% | 33.33% | 6.67% | 9.68% | 47.06% | 28.00% | 3.85% | 6.90% | 41.67% | 15.15% | 3.45% | 18.18% | 17.54% | 20.00% |
How often have you used Redis?
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Often | 8.42% | 8.00% | 12.12% | 5.41% | 0.00% | 4.00% | 10.00% | 12.90% | 0.00% | 8.33% | 7.69% | 13.79% | 0.00% | 5.88% | 14.29% | 18.18% | 10.91% | 5.00% |
| Seldom/Sometimes | 12.63% | 8.00% | 12.12% | 16.22% | 11.11% | 4.00% | 10.00% | 22.58% | 6.25% | 4.17% | 15.38% | 20.69% | 4.55% | 17.65% | 14.29% | 9.09% | 10.91% | 15.00% |
| Never | 74.74% | 76.00% | 75.76% | 72.97% | 66.67% | 92.00% | 80.00% | 58.06% | 75.00% | 87.50% | 76.92% | 62.07% | 81.82% | 76.47% | 67.86% | 72.73% | 74.55% | 75.00% |
How often have you used GridEngine Continuous jobs?
[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Often | 35.05% | 51.85% | 27.27% | 29.73% | 11.11% | 30.77% | 43.33% | 37.50% | 12.50% | 32.00% | 50.00% | 36.67% | 16.67% | 29.41% | 46.43% | 63.64% | 36.84% | 32.50% |
| Seldom/Sometimes | 21.65% | 18.52% | 24.24% | 21.62% | 22.22% | 11.54% | 20.00% | 31.25% | 18.75% | 4.00% | 26.92% | 33.33% | 16.67% | 23.53% | 25.00% | 18.18% | 21.05% | 22.50% |
| Never | 41.24% | 22.22% | 48.48% | 48.65% | 44.44% | 57.69% | 36.67% | 31.25% | 56.25% | 64.00% | 23.08% | 30.00% | 58.33% | 47.06% | 28.57% | 18.18% | 38.60% | 45.00% |
Free form responses
[edit]The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form. The free form responses include comments broadly falling into these categories:
- Documentation (33 comments)
- Stability and performance (18 comments)
- Version control and Deployment (14 comments)
- Logs, Metrics, and Monitoring (12 comments)
- Package management (10 comments)
- SGE (8 comments)
- Database (7 comments)
- Account/Tool creation (5 comments)
- SSH (5 comments)
- Other (11 comments)
Documentation
[edit]In the responses mentioning documentation either directly or indirectly, generally the respondents were looking for more detailed how-to, getting started, and tutorial content. The need for more documentation for database contents was mentioned several times. Multiple respondents felt they did not know enough or have enough free time to contribute to the on-wiki documentation themselves.
Stability and performance
[edit]Stability and performance concerns often mentioned NFS. Some respondents are hoping for more proactive notification of planned and emergency downtime.
Version control and Deployment
[edit]Deployment comments mentioned the lack of a clear best practices and easy workflows for getting tools and their dependencies deployed to the servers. The version control comments were very similar in nature.
Logs, Metrics, and Monitoring
[edit]Monitoring for job failures and errors in user developed applications topped the comments related to logs, metrics and monitoring. Some facility for automated health checks and email alerts was mentioned several times.
Package Management
[edit]Package management comments either asked for specific system wide packages to be installed or lamented the difficulty in managing libraries needed for particular tools.
SGE
[edit]Sun Grid Engine comments generally mentioned the unexpected need to restart jobs and difficulty understanding how to pick the correct options for running jobs.
Database
[edit]Database comments ranged from complaints about using MySQL Workbench, to the naming of replica.my.cnf, to lack of EXPLAIN command support, and less open data than Toolserver provided.
Account/Tool creation
[edit]Account creation comments centered on the difficulty of registering for a Tool Labs account, largely mirroring the documentation concerns.
SSH
[edit]Comments on ssh mentioned the complexity of using ssh in general and frustration with bastions and tunneling.