Grants talk:Project/Accessibility of equation rendering: a comparative evaluation

Thanks for stopping by! We'd love to hear what you think about our proposal!

--- zorkow, pkra

What about adding a text browser e.g. Lynx, one old browser and one without JavaScript to the list? Of course you cannot expect a perfect rendering in those cases, but for me a good and accessible rendering solution means that the math is always readable somehow e.g. just printing the source code and not broken image icons.--Debenben (talk) 23:12, 26 November 2018 (UTC)

Thanks for your question (and your endorsement)! We have to limit ourselves to a small set of browsers to keep the scope reasonable. As such, we focus on browsers officially supported by MediaWiki. This could be an opportunity for (voluntary) community involvement along the way - once the test framework and evaluation matrix is ready, community members can contribute test results as well.--Pkra (talk) 11:39, 28 November 2018 (UTC)

with "This could be..." I meant to say that your idea of testing old and/or text browsers could be an opportunity for community involvement. We would of course value additional test data but we don't expect any such testing (i.e., the study will work if there's no community input). I hope that makes more sense :) --Pkra (talk) 11:50, 28 November 2018 (UTC)

All traffic on wikipedia, results for math rendering technique 1      Known to break page rendering / not be readable in any way (8%)     Tested and working (19%)     Not tested but assumed to work (34%)     Tested and has minor problems (17%)     Not tested but only minor problems expected (22%)

I will try to test a few devices and report what I get. Of course it is impossible to test all browser-version/operating-system/screen-reader-cominations. However I think the result should be a complete overview (e.g. like the pie chart for illustration of what I mean), and take into account if a rendering solution is known to completely break the page or not be readable in any way because someone else tested it or it is known to lack the requirements for the rendering solution.--Debenben (talk) 22:07, 29 November 2018 (UTC)

Thanks! A simple overview will definitely be in the results. But likely it will (have to) be more refined. We want to not only test the basics ("does something") but dive deeper, especially into the heuristics involved. Pkra (talk) 08:09, 6 December 2018 (UTC)

Thanks, I am happy with an overview like that. I wanted to make sure the investigation does not focus too much on details of e.g. native MathML rendering which works good on Firefox with add-on, but not for everyone else. The "lowest common denominator" is more important than awesome rendering solutions for readers that change configurations and install special fonts on their system etc. Mathematical editors choose to format their math how they think the presentation is best for "the readers" with the current setup. If the result of this investigation shows them what "the readers" see, they will choose the formatting accordingly. However I do not think they will consider much how it might be presented in the future (they are used to it taking 10 years until things might be working partially).--Debenben (talk) 14:41, 8 December 2018 (UTC)

Project Grant proposal submissions due 30 November!

Thanks for drafting your Project Grant proposal. As a reminder, proposals are due on November 30th by the end of the day in your local time. In order for this submission to be reviewed for eligibility, it must be formally proposed. When you have completed filling out the infobox and have fully responded to the questions on your draft, please change status=draft to status=proposed to formally submit your grant proposal. This can be found in the Probox template found on your grant proposal page. Importantly, proposals that are submitted after the deadline will not be eligible for review during this round. If you're having any difficulty or encounter any unexpected issues when changing the proposal status, please feel free to e-mail me at cschillingwikimedia.org or contact me on my talk page. Thanks, I JethroBT (WMF) (talk) 23:20, 27 November 2018 (UTC)

Noted!--Pkra (talk) 11:54, 28 November 2018 (UTC)

Eligibility confirmed, round 2 2018

This Project Grants proposal is under review!

We've confirmed your proposal is eligible for round 2 2018 review. Please feel free to ask questions and make changes to this proposal as discussions continue during the community comments period, through January 2, 2019.

The Project Grant committee's formal review for round 2 2018 will occur January 3-January 28, 2019. Grantees will be announced March 1, 2018. See the schedule for more details.

Questions? Contact us.

--I JethroBT (WMF) (talk) 02:25, 8 December 2018 (UTC)

Comments

In my opinion, the project as written is too vague: there are no clearly defined goals and success metrics. In addition, it is written in such a dense technical language that it is very difficult to understand for any person not directly involved with these MathML staff. I think that you should rewrite it in, ironically enough, more accessible language. Moreover, why do you think that your report will be that useful? Is not it true that such report are often quickly forgotten as soon as as they finished? Ruslik (talk) 19:36, 19 December 2018 (UTC)

@Ruslik0: I'll try to explain why I think such a report is needed: Some time ago volunteer developers spend a lot of time to integrate MathJax client-side rendering into the MediaWiki. It worked well as a user option and math editors and developers alike worked under the assumption that it will eventually become default for everyone. As opt-in for editors it was not useful because after all you could get the same behavior with a simple user-script or browser extension and you could not write stuff into articles that was not supported by the default rendering. Unfortunately (I was not involved in the discussions at that time) it was decided by WMF staff(?) that they did not want to spend money on servers for delivering the necessary webfonts and using a third party CDN was not acceptable. As a result of this all volunteer developers working on the math rendering which was state-of-the-art at the time left the project completely or at least did not work on the math extension any more. There was more frustration amoung editors later when the -then outdated- user option was removed (phab:T99369). E.g. the editors were told before that the HTML rendering for simple formulas had to be removed in order to be able to use MathJax client side rendering in the future and the German Wikipedia decided that they do not want to use the math templates like in the English Wikipedia because they would no longer be necessary with MathJax client side rendering and well... then they were told the MathJax client side rendering would not be necessary because people would get native MathML rendering in the future. All this drama would have been avoidable if WMF would have had a plan for the development of the math extension. Currently there is no plan at all and for creating one (hopefully with involvement of WMF staff) all options should be carefully examined and for this we need such a report. As you have noticed math rendering becomes quickly very technical and complicated and I doubt that there is anyone at WMF or any other volunteer that could keep up with the knowledge and experience of zorkow and pkra in this field.--Debenben (talk) 21:01, 22 December 2018 (UTC)

Or putting it into a specific question: "From our previous discussions with Wikimedia staff, we are aware that [...] webfonts-dependent CSS is not an option for Wikimedia at this time due to [...] bandwidth considerations[...]." - I know that this was the problem around 2012, but are those still an issue? Would it be possible to get an estimate on the additional bandwidth e.g. compared to other software components?--Debenben (talk) 00:17, 23 December 2018 (UTC)

@Ruslik0: Thanks for the comments, we appreciate them very much. We understand that proposing such a research project may seem a bit vague, especially since there is no prior work in this area. Consequently we are trying to create a baseline for an area that is poorly developed. Ultimately, the goal of this research project is to provide clarity in this area and help WMF and others to make implementation decisions based on real world evidence using a documented test framework.

Regarding your point on goals and success metrics. We outlined our outputs and outcomes at [1]. Would you have a suggestion on which points could be clarified or otherwise improved?

The irony of a inaccessible complex research proposal for an accessibility problem does not escape me. Unfortunately, equation layout is already a cluttered field on the web in terms of technologies; adding accessibility requirements makes it even more complex.

Regarding the futility of a report. Volker and I agree with you - a report on its own is not very useful as it will become outdated quickly as web technology improves. That's why the main output will be a evaluation methodology to allow for the examination of quality and accessibility of complex web construct together with the creation of a proper test framework. We will work directly with WMF developers to facilitate the use of the evaluation procedure to allow WMF (and anyone else) to continuously update results. As such, our goal is to provide precise metrics and a reliable first set of data for people to test against easily. In addition, we've discussed this problem space with WMF staff multiple times for a few years now. In particular thanks to Marko volunteering as an advisor to the project, we are confident that our outputs will directly help WMF to improve the implementations at WMF and on MediaWiki sites in general. Pkra (talk) 12:42, 9 January 2019 (UTC)

Than you for your answers. On the second reading I think that I better understand the projects and its goals. Ruslik (talk) 19:32, 27 January 2019 (UTC)

Thank you very much for the follow-up, @Ruslik0:. Please let us know if there's anything else you think we can improve.Pkra (talk) 11:56, 28 January 2019 (UTC)

Aggregated feedback from the committee for Accessibility of equation rendering: a comparative evaluation

Scoring rubric	Score
(A) Impact potential Does it have the potential to increase gender diversity in Wikimedia projects, either in terms of content, contributors, or both? Does it have the potential for online impact? Can it be sustained, scaled, or adapted elsewhere after the grant ends?	6.8
(B) Community engagement Does it have a specific target community and plan to engage it often? Does it have community support?	6.2
(C) Ability to execute Can the scope be accomplished in the proposed timeframe? Is the budget realistic/efficient ? Do the participants have the necessary skills/experience?	7.0
(D) Measures of success Are there both quantitative and qualitative measures of success? Are they realistic? Can they be measured?	5.6
Additional comments from the Committee: I'm somewhat familiar with the previous deployment of MathJax and the frustration surrounding it. Having a study like this to help shape development direction is a good idea. The project fits with Wikimedia's strategic priorities and can have a significant online impact if successful. While the final report will become outdated quickly, the methodology, that they are going to develop, may be useful in the future. In this sense the project is sustainable and scalable. not a top strategic priority but could be easily scaled - not only in Wikimedia but in the whole Internet as Wikipedia adopted practices will influence other sites and devices I love the proposal for what it aims to do with accessibility. More study should be done on how assistive technology interacts with MediaWiki software. My concern is for the broad impact. That has not been illustrated, and, in fact, the impact has been documented in the proposal to be rather narrow. Additionally, the methodology is experimental, which I appreciate must be done in some cases, but the methodology section leaves a bit to be desired. I would like to see more about which standards you would be tangentially applying in this assessment of quality, as well as other experts outside of the research team (people who use this software due to their disability, not just having experience with the software). I could also see something pulled from other accessibility technology standards to use as comparison (even if the standards themselves leave room for improvement, you could indicate these are a good base measure, but here is where they fail this scenario...). Having a standardized set of formulae will help. This could be especially true since much of MW development has seemed somewhat averse to planning that these standards would necessarily imply. That said, it is good to see that they are working with developers in the WMF and keeping an eye on non-WMF deployments of MW. The project is clearly innovative. The impacts and risks are evenly balanced. The measures of success are somewhat vague but for this kind of projects they can not probably be better. realistic measures and targets, new idea (haven't seen anything similar before) I think the possibility for learning about how the software interacts with accessibility software, and ways we can improve is great. I worry this does not have a broad community of users nor a broad swath of content to benefit from such innovation. The scope can probably be accomplished in 12 months and the budget is realistic. The participants have necessary skills/experience. participants certainly have skills but 100/120 EUR per hour seem excessive for me I feel comfortable about the applicants skill set, but what gives me pause is the rather vague nature of the methodology. I feel there should have been some foundation set before this proposal request by the applicants (having a more defined sample, identifying standards to which their sample will be measured). I understand this is experimental in nature, but I feel that leaves it too open-ended to be able to identify the project feasibility within a 12-month timeline. I see lots of engagement with the technical community and what looks like lots of experience with the assistive technologies, but I don't see much engagement of STEM users who would benefit. But maybe I missed it. The community engagement is limited but at the this stage it is not that necessary. I'd recommend to tell about that project on relevant 3rd party resources There is a target community. There is not, however, a community really engaged with this proposal, as indicated in the proposal, the wikis to be impacted are either defunct or small. The project is a study and an analysis but it will not produce e consistent and tangible impact. Interesting for a possible implementation but it doesn't seem to follow this goal. I would fund this but I would like a better description of the planned engagement with the communities that would benefit. I am willing to support this interesting research effort. I am not sure I can set a definite amount, just confused by 100-120 EUR per hour (as a CEO in local Russian company I earn times less :)). I'd better analyse benchmarks for similar jobs for non-for-profits I feel in order to receive the funding at the level the proposers are wishing to receive, more groundwork needs to be laid and with a broader impact.

This proposal has been recommended for due diligence review.

The Project Grants Committee has conducted a preliminary assessment of your proposal and recommended it for due diligence review. This means that a majority of the committee reviewers favorably assessed this proposal and have requested further investigation by Wikimedia Foundation staff.

Next steps:

1. Aggregated committee comments from the committee are posted above. Note that these comments may vary, or even contradict each other, since they reflect the conclusions of multiple individual committee members who independently reviewed this proposal. We recommend that you review all the feedback and post any responses, clarifications or questions on this talk page.
2. Following due diligence review, a final funding decision will be announced on March 1st, 2019.

Questions? Contact us.

I JethroBT (WMF) (talk) 19:14, 6 February 2019 (UTC)

Response

We would like to thank the reviewers for their work and their detailed feedback. We have collated the expressed concerns into four points below and interspersed our replies.

1) Reviewers have expressed the concern on breadth of impact and size of the community is will benefit.

The reviewers point out the catch-22 situation intrinsic to this problem. We are trying to open knowledge to users that have inadvertently been excluded from Wiki STEM content. While sources like Wikipedia is generally the first port-of-call for learners even of STEM subjects, for a sizable community with visually and print impairments, this is not the case and they use different starting sources for STEM content. Thus getting this content accessible is an important prerequisite before a broader community for Wikimedia can be grown. And while STEM content does indeed only comprise a fraction of all wiki content, it is still a highly important part to guarantee inclusive and equal education.

As estimated by the Wikispeech project (https://www.mediawiki.org/wiki/Wikispeech):

"The approximately 25% of people who find it easier to learn from spoken text could utilize this functionality [i.e., text-to-speech] as well as those who wish to learn at the same time as they do something else (e.g. driving). 25% of the readers of Wikipedia would mean that approximately 115–125 million people would benefit from the project in the long run."

While not all of these readers will be interested in STEM content we feel that this is not a reason exclude that content entirely. Already now simple tests with technology like Amazon Alexa or Google Assistant demonstrate that inquiries on mathematical topics that go beyond the quadratic equation, lead to answers that while taken from Wikipedia, are very unsatisfactory due to their lack of mathematical content.

2) There is not enough engagement with the STEM users concerned.

We appreciate the reviewers concern on this point and would like to address it during the project by assembling a group of interested STEM users with visual impairments at the beginning of the project that we will consult throughout the project at regular intervals.

Both proposers have long-standing experience and good connections into the relevant community. Both are Invited Experts on the W3C WAI-ARIA working group. Volker Sorge was a visiting faculty member of Google's accessbility team and he is a regular speaker at conferences like CSUN and Accessing Higher Grounds. Peter Krautzberger co-chairs the W3C MathOnWeb Community Group, managed the MathJax Consortium and consults a number of scientific publishers.

In addition to consultations we will also add a user testing session in the final month of the project, that will be organised by Volker Sorge. He has connections to the both students and teachers in the field via his University affiliation (The University of Birmingham) to the Victar Centre as well as the New College Worcester, a specialist college for visually impaired children.

3) Concerns about the methodology and its experimental nature.

a) One major concern is the lack of ground work or pre-defined sample set.

Both proposers have been involved in the development of various sample sets of equations. Examples on the technical side include

• the MathJax test suite, https://github.com/mathjax/MathJax-test
• the speech-rule-engine test suite, https://zorkow.github.io/speech-rule-engine/resources/www/
• the mathmlcloud test samples, https://mathmlcloud.org

Example on the authoring side include

• Wikipedia Formula help page https://en.wikipedia.org/wiki/Help:Displaying_a_formula
• the math stackexchange tutorial, https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference,

While these sets will provide a good starting point, as mentioned in the proposal, we will also use and intersect the existing sets with the list of formulas on WMF sites by usage, cf. (https://en.wikipedia.org/wiki/User:Salix_alba/maths2018)

b) It was also suggested applying already existing accessibility technology standards for quality assessment.

The starting point in this case is less fortunate as equations are somewhat of a conundrum for web accessibility standards today. Consider the following two examples:

On the one hand, there is the standard for accessible name computation (https://w3c.github.io/accname/). It defines an algorithm to calculate the so-called accessible name of any DOM object. This algorithm needs to be exact because heuristics have led to bad practices with both user agents and authors/developers in the past. Unfortunately, equational content is inherently ambiguous (cf https://github.com/w3c/mathonwebpages/wiki/%5Ba11y-TF%5D-ambiguous-notation), thus conflicting with this core assumption of web accessibility.

On the other hand, there is the WCAG https://www.w3.org/TR/WCAG20/, a set of recommendations for making web content accessible which is the foundation of many legal requirements worldwide (see https://www.w3.org/WAI/policies/). WCAG often appears vague (and in particular can not be fully evaluated programmically) because evaluating accessibilty usually depends highly on both content and author intent. Yet, WCAG makes no effort at describing any guidelines regarding STEM specific content such as equations, chemical notation, electric diagrams or other forms of domain-specific notation.

This is truly an unsatisfactory situation and our proposal aims to be following the direction of these standards to take a first solid step towards providing the community with a clearer picture of both what can be achieved with today's technologies and what should be achievable in the short, mid, and long term. As we are both invited experts on the W3C ARIA Working Group, we are working on this with a mindset to improve the overall situation of accessibility standards and help the WG make informed decisions.

c) Finally, a worry was expressed that the study is too open ended in its scope.

As a comparative study, we are bound by the set of available technologies. Currently there is only a handful of actively developed tools that can provide non-visual rendering of equational content. Although this is an unfortunate situation for the overall progress of accessibility, not only in STEM areas, it has the advantageous side effect that it gives our study automatically a tight focus while still covering all relevant technologies.

4) The project does not lead to tangible output

The main output will be an evaluation methodology to allow for the examination of quality and accessibility of complex web constructs together with the creation of a proper test framework. We regard this as tangible outcomes and while the web technologies underlying the test framework can and will change over time the evaluation methodology should not.

We will work directly with WMF developers to facilitate the use of the evaluation procedure to allow WMF (and anyone else) to continuously update results. As such, our goal is to provide precise metrics and a reliable first set of data for people to test against easily. A concrete implementation can be part of a follow up project, but is currently out of scope.

Thanks for your efforts @I JethroBT (WMF):. Zorkow (talk) 22:25, 17 February 2019 (UTC)

Round 2 2018 decision

This project has not been selected for a Project Grant at this time.

We love that you took the chance to creatively improve the Wikimedia movement. The committee has reviewed this proposal and not recommended it for funding. This was a very competitive round with many good ideas, not all of which could be funded in spite of many merits. We appreciate your participation, and we hope you'll continue to stay engaged in the Wikimedia context.

Next steps: Applicants whose proposals are declined are welcome to consider resubmitting your application again in the future. You are welcome to request a consultation with staff to review any concerns with your proposal that contributed to a decline decision, and help you determine whether resubmission makes sense for your proposal.

Over the last year, the Wikimedia Foundation has been undergoing a community consultation process to launch a new grants strategy. Our proposed programs are posted on Meta here: Grants Strategy Relaunch 2020-2021. If you have suggestions about how we can improve our programs in the future, you can find information about how to give feedback here: Get involved. We are also currently seeking candidates to serve on regional grants committees and we'd appreciate it if you could help us spread the word to strong candidates--you can find out more here. We will launch our new programs in July 2021. If you are interested in submitting future proposals for funding, stay tuned to learn more about our future programs.

Alex Wang (WMF) (talk) 17:25, 1 March 2019 (UTC)