IRC office hours/Office hours 2014-02-16

Chat on VisualEditor
Wednesday 16 February 2014
0000 - 0100 UTC

[23:59:59] <Maggie_Dennis> Hello, everyone. :) Welcome to our second VisualEditor office hours for the month of February. James Forrester is here to discuss VisualEditor; our earlier session is logged on office, here: https://meta.wikimedia.org/wiki/IRC_office_hours/Office_hours_2014-02-15
[00:00:28] <Maggie_Dennis> Does anyone have any questions to launch? I have a few from non-attendees but generally like to open the door here first. :)
[00:01:15] <Maggie_Dennis> Okay, then I'll start with one from an Italian Wikimedian.
[00:01:33] <Maggie_Dennis> This question is regarding <, which means "save and close" in dialogues, but which many evidently find confusing.
[00:01:46] <Maggie_Dennis> The question is if this is something being considered.
[00:01:56] <James_F> Hey everyone, and thanks Maggie.
[00:02:42] <James_F> This an interesting part of the design of VisualEditor's interface library that has come up a few times in different ways, and which we've changed before.
[00:02:51] <James_F> VisualEditor has three kinds of tools:
[00:03:18] <James_F> Instant tools that just have their effect (often toggling, like Bold, or doing something, like Cut).
[00:03:52] <James_F> Complex tools that open up a modal window with which you must interact, "dialogs" (these are things like transclusions, media settings, page settings, etc.).
[00:03:57] <James_F> And then there's the middle ground.
[00:04:36] <James_F> We call these "inspectors", and the idea is that they're much less heavy-weight and complex than full dialogs, and are something that users should just want to quickly use in the midst of their flow without breaking out of their actions.
[00:05:06] <James_F> Examples of this are adding or editing a link, a formula, galleries, hieroglyphics, and so on.
[00:05:53] <James_F> Because we want to keep these as close to "in line" as possible, the design is that there is no "OK" button – you just click (or touch if on mobile) away from them, and they apply.
[00:06:24] <James_F> We have a "<" button to give people a button, but it's not the only thing that makes them apply changes.
[00:06:43] <James_F> We also have a rubbish bin ('trash can') button to remove the item they're editing.
[00:06:56] <James_F> This is (as with all things in VisualEditor) up for discussion.
[00:07:07] <James_F> Perhaps we should just remove the "<" button.
[00:07:28] <James_F> We could add a confirm step, but this would be a regression in terms of flow and ease of use.
[00:07:58] <James_F> Perhaps we could have a dummy tick button, which applies and closes – but that clicking/touching outside the inspector also does the same?
[00:08:16] <James_F> I'm happy to discuss this if people are interested.
[00:08:28] <James_F> And sorry that it's not totally obvious yet.
[00:08:34] <James_F> Clearly we need to make it better.
[00:09:12] <Maggie_Dennis> Thanks, James_F.
[00:09:17] <James_F> (Hey again jayvdb.)
[00:09:27] <Maggie_Dennis> Unless anybody has questions about that (or something else), I'll pull up another from my list of "people who couldn't make it." :)
[00:09:37] <Keegan> Yes, I'm not a fan of the < at all :)
[00:09:57] <Maggie_Dennis> Okay, so...
[00:09:57] <James_F> Keegan: Do you think people will feel stranded if there's no button other than the "delete this" one, though?
[00:10:21] <Keegan> I just think we need a different design
[00:10:33] * James_F nods.�
[00:10:36] <Keegan> Heck, > might even work better. I just think < is regarded as back
[00:10:51] <James_F> Yes, that's what it was originally.
[00:11:12] <James_F> (As I said, we've changed the design a few times in the few years we've been developing VE.)
[00:12:03] <James_F> This also bleeds into the discussion about the link editor not having an "anchor" (link display text) editing box alongside the link target editing one.
[00:12:35] <jayvdb> Does the VE deploy process contain any testing before release? if so, what is it? and why are so many simple problems not being caught before deploy? lots of very basic bugs in every function release is costing VE team and the community time and wasting precious capital. What is being done to improve the process?
[00:12:37] <Maggie_Dennis> Okay, so, the next question on my list concerns the special character inserter. As you know, we've been asking people for feedback on this. The question is where we are in the design phase for SCI. Are you looking for refinement or are major overhauls a potential?
[00:13:32] <Maggie_Dennis> (jayvdb, we'll get yours first.)
[00:13:45] <James_F> jayvdb: In brief, yes.
[00:14:16] <James_F> jayvdb: When the code is written, it's tested and worked out before it's "merged" into master.
[00:14:31] <James_F> jayvdb: This is the principle way in which testing is meant to occur for functionality.
[00:14:43] <jayvdb> that is developer testing
[00:14:53] <jayvdb> is there any other testing done before deploy?
[00:14:59] <James_F> Yes. :-)
[00:15:17] <James_F> Post-merge, code sits on the pre-deployment server where the QA team test it to find issues that didn't come up before merge.
[00:15:46] <James_F> We notice in particular browser bugs here (Chrome and Firefox both have habits of following specifications until they don't).
[00:16:19] <jayvdb> Does the QA team have a set of functions to walk through (test plans) for each new major feature deployed?
[00:16:42] <Maggie_Dennis> (jayvdb, just to let you know, I can hear James_F busily typing. He's not finished. :))
[00:16:54] <jayvdb> (ta)
[00:17:14] <James_F> Then, each Wednesday, I go through every commit that's been merged in the previous week with QA and the lead developers, and we sanity-check whether things are ready to go out or if there are known bugs/etc. which means we'll pull that code, or make some quick fixes.
[00:17:52] <James_F> The next day (Thursdays), the Platform team take the current development version of VisualEditor ("master") and push it to MediaWiki.org (and test.wikipedia.org).
[00:18:33] <James_F> At that point we do additional testing (mostly quick verification that the deployment worked as expected, and more in-depth testing by the QA team of whether it works in production the same way it worked in testing, or if there are any new issues.
[00:18:37] <James_F> +)
[00:19:12] <James_F> The functional descriptions of new features are generally established in team discussion (and in code review) rather than written-out exhaustively.
[00:20:02] <James_F> This is mostly because the system that VisualEditor is trying to correspond to (MediaWiki) is… complex and ill- or even un-documented in large areas.
[00:20:12] <jayvdb> you dont have automated client unit tests to verify basic use cases?
[00:20:32] <James_F> Oh, of course, yes there are unit tests and browser tests, but those can only verify basic use cases, as you say.
[00:20:50] <James_F> All our code is required to pass all unit tests before it can merge, let alone get deployed.
[00:21:25] <James_F> That's not yet the case for our browser tests, sadly – mostly because our browser tests regularly don't pass due to the server falling over or other issues.
[00:21:46] <James_F> In the next few months we hope to get to a point where we can have browser tests blocking merge as well.
[00:22:21] <jayvdb> James_F: do the automated tests cover use cases - i.e. multi-step user interactions, or single function functional compliance tests ?
[00:23:33] <James_F> jayvdb: The unit tests are exactly that – automated (headless browser) tests of functional compliance. They don't simulate the UI, much less a full UX check, but make sure that (e.g.) rich-paste spits out the same HTML that it did last time given the same input.
[00:23:49] <James_F> The browser tests are full end-to-end UX multi-step user interaction tests.
[00:24:42] <James_F> They're currently run in "latest"(-ish) Firefox and Chrome on Linux (provided by SauceLabs), and we're about to expand that to more OSes and browsers.
[00:25:03] <James_F> We also have an automated test that checks all the wikis where VE is deployed by default to, to make sure that they work.
[00:25:10] <jayvdb> ok. is there a webpage that lists all of these browser tests (showing which are not passing)
[00:25:18] <James_F> Yes – https://wmf.ci.cloudbees.com/view/r-ve/
[00:25:20] <jayvdb> can the community help writing browser test plans
[00:25:27] <James_F> Note that quite a few of them don't pass right now.
[00:25:28] <jayvdb> i.e. use cases
[00:25:32] <James_F> Yes! We'd love that.
[00:25:40] <James_F> They're written in Cucumber mostly.
[00:25:46] <jayvdb> do you have a guide on how the community can do this ?
[00:25:53] <James_F> And they only cover a sub-set of functionality, sadly.
[00:25:58] <James_F> Good question.
[00:26:05] * James_F will hunt for such a guide.�
[00:26:18] <James_F> In general, if people join the qa mailing list they're very helpful.
[00:26:37] <Maggie_Dennis> I'll make a note to follow up with James_F on that one later, jayvdb, so it doesn't get overlooked. :)
[00:26:37] <James_F> See https://www.mediawiki.org/wiki/Quality_Assurance
[00:27:19] <jayvdb> ta Maggie_Dennis , and thx James_F
[00:27:48] <James_F> So, for example, the latest run of the Linux/Chrome test failed on the "formatting" tasks: https://wmf.ci.cloudbees.com/view/r-ve/job/VisualEditor-test2.wikipedia.org-linux-chrome/314/#showFailuresLink
[00:27:57] <James_F> All the others passed, but those broke.
[00:28:36] <James_F> I believe that this is because the code that the test was latching onto in VisualEditor's interface (the HTML structure) changed slightly, so now the script can't "find" the drop-down box to change the format.
[00:29:16] <James_F> Quite often the breakages are false negatives like this, but that's better than false positives (where we think that everything's working but instead it's not).
[00:29:33] <James_F> And yes, more tests (or even just ideas for tests) very much welcome.
[00:29:39] <James_F> We're a long way from 100% coverage.
[00:30:05] <jayvdb> (this is a Parsoid question, so feel free to skip it if it cant be answered by anyone here) What percentage of the wikipedia content (all language) has been tested as passing through parsoid and returning the same wikitext ?
[00:30:41] <James_F> Also, as a note, we've been exploring using automated unit tests for simulating browser, OS and software behaviour for "Input Method Editors" so that we can test those reliably.
[00:31:26] <James_F> One of the issues we've found is that not only does each IME have its own (totally different) way of interacting with the editing surface, but that these can change – not just between operating systems or browsers, but also between versions of the same IME - even sub-point releases.
[00:32:09] <James_F> So helping people insert text in Chinese or Korean or Thai is a particular challenge, and something that as far as we can tell there's no known expertise in doing automated testing for.
[00:32:12] <Maggie_Dennis> (jayvdb, I've got your new question on prompt for when he finishes your last one. :))
[00:32:41] <James_F> We feel that relying on manual testing – especially, relying on bug reports from end-users – just isn't good enough for this, which is why we've invested so much time building out such an framework for this.
[00:33:16] <James_F> (MediaWiki supports way, /way/ more languages than other tools, so VisualEditor needs to do the same – and because of this, we can't just steal other peoples' good efforts.)
[00:33:36] <James_F> Anyway, I've rambled on enough, jayvdb. :-) Hope I answered that question enough for you.
[00:33:40] <James_F> Now for your next one?
[00:33:42] <Maggie_Dennis> Jayvdb asks: What percentage of the wikipedia content (all language) has been tested as passing through parsoid and returning the same wikitext?
[00:34:40] <James_F> The original testing strategy for Parsoid was to take approximately 100k articles from the English Wikipedia, selected at random, and "round-trip" test against those.
[00:35:12] <James_F> This means taking the original wikitext, converting it to Parsoid's HTML+RDFa, and then converting it back, and looking for differences.
[00:36:42] <James_F> Results are classified into four groups – succeeded perfectly, succeeded but for some minor syntactic errors (e.g. whitespace differences), failed with some semantic errors (e.g. a table control character getting mis-read as a paragraph), and failed with a crash (no result returned).
[00:37:16] <James_F> By March last year Parsoid had got to (I believe) 75/23/2/0.1% for English Wikipedia.
[00:37:41] <James_F> Parsoid also contains a "selective serialisation" process which means that it only changes the wikitext if VisualEditor touched it.
[00:38:23] <James_F> This means that if Parsoid slightly muddles the wikitext in a non-semantic way (e.g. "*Foo[[bar]] " becomes "* Foo[[bar]]") it doesn't matter very much.
[00:38:46] <James_F> In March, Parsoid switched focus from enwiki-only to non-en languages too.
[00:39:07] <James_F> They changed to running the same process on 10k (rather than 100k) articles taken from 16 languages, including English.
[00:39:29] <James_F> So 160k articles from the biggest Wikipedias.
[00:40:14] <James_F> The results went down (as expected) but are now at 85/10.2/0.3/0.02% right now
[00:40:29] <James_F> You can see results live (triggered every day, I believe) at http://parsoid.wmflabs.org:8001/.
[00:41:09] <James_F> Languages tested are the "big 10" Wikipedias and a few selected others (like Arabic) that were thought to be tricky.
[00:42:01] <James_F> So, to summarise: Parsoid tests every night their code before it gets release against a large (but not complete) sample of wikitext from the big Wikipedias.
[00:42:49] <James_F> They don't test non-Wikipedia content, and they don't test smaller languages; this is mostly due to diminishing returns – we could add this but it'd just take up more time without adding more crashes, in our experience.
[00:42:56] <James_F> jayvdb: Is that helpful
[00:42:56] <James_F>  ?
[00:44:12] <Maggie_Dennis> (jayvdb, giving you another minute to type. :) Unfortunately, since I can't hear you, I don't know if you are!)
[00:44:21] <jayvdb> hehe. sorry I was afk
[00:44:29] <Maggie_Dennis> :D
[00:45:09] <James_F> Sorry, I mis-spoke – it's 84.98/14.74/0.26/0.02%
[00:45:17] <jayvdb> not really - I think it would be useful to have 100% of the pages tested, so uncover oddities to be investigated. 160K is tiny
[00:46:35] <James_F> Testing 20m pages a night would require spending quite a lot of donor funds for very little return.
[00:47:02] <jayvdb> one box slowly ploughing through all pages on all wikis. you dont need to test them all every night James :P
[00:47:19] <James_F> I agree that 160k won't catch everything (though you'd be surprised how little slips through), but it's very rare that we find new issues that weren't already in the logs.
[00:47:31] <jayvdb> bugs in Parsiod are not fixed that quickly
[00:47:38] <James_F> The code changes every day, several times a day.
[00:47:56] <James_F> You need a single benchmark to compare against to make decisions about whether a new version is an improvement.
[00:48:23] <jayvdb> James_F I wouldnt be surprised at how little slips through. I went through the parsiod errors in ~July 2013, and found many very large bugs which were not previously reported, and many still havent been fixed
[00:48:26] <James_F> "Today we're at 90%!" "Is that higher or lower than yesterday?" "Well, it's higher, but it's about different articles!".
[00:49:01] <James_F> There are currently 169 bugs in Bugzilla for Parsoid.
[00:49:31] <James_F> Some of which include back-end feature requests like "Add round-trip test mode using selser and web interfaces" (bug 56601).
[00:49:32] <jayvdb> e.g. the chess diagram bug was found in the Parsiod errors listed at http://parsoid.wmflabs.org:8001/
[00:49:39] <jayvdb> back in ~July 2013
[00:50:07] <Maggie_Dennis> (Guys, I have to pull this back on topic. We're at the 10 minute to go mark, and Parsoid _is_ a separate product. Clearly related, but I do still have at least one outstanding request for VE feedback.)
[00:50:24] <jayvdb> np Maggie
[00:50:34] <James_F> jayvdb: I'll pass your request onto the Parsoid team and ask them to increase the breadth of testing.
[00:50:40] <Maggie_Dennis> Happily, jayvdb covered many of the topics I had on list (mostly related to testing) and then some. :) But I still have this question: concerning the special character inserter and the feedback you request, can you indicate where we are in design phase? Are you looking for refinement or are major overhauls a possibility?
[00:50:55] <James_F> Maggie_Dennis: Thanks.
[00:51:37] <James_F> So, the "special character inserter" was built quickly as a proof-of-concept to ask for feedback about how such an inserter should work, and whether this design was a reasonable one to take forward.
[00:52:09] <James_F> I know that a lot of wikis are in languages that are very ill-served by the IT industry and so users need to insert characters that are inaccessible to them easily.
[00:52:36] <James_F> Most users can't just type - – — on their keyboards (like I can on a Mac) to get hyphen, endash and emdash, for example.
[00:53:05] <James_F> Similarly, naïve Operating System inserters leave people with a déluge of issues related to the rôle of characters.
[00:53:15] <Maggie_Dennis> LOL!
[00:53:18] <James_F>  ;-)
[00:53:51] <James_F> Vector's "WikiEditor" tried to replace the original "EditTools" block of 10 or so characters with a very comprehensive, but hard to use mass ranks of every character everywhere.
[00:54:18] <James_F> It also failed in that "EditTools" was never switched off. :-)
[00:54:52] <James_F> The needs of the special character inserter for VE are very close to, but not exactly the same as, the needs for wikitext editors.
[00:55:26] <James_F> For example, "helpful" blocks of wikitext to insert "<ref>Insert your reference here.</ref>" aren't appropriate (and won't help the user).
[00:55:40] <James_F> Which means that, unfortunately, we can't simply re-use an existing tool.
[00:56:47] <Maggie_Dennis> (oops! Missed my 5 minute warning. Sorry. 4 minute warning. )
[00:56:48] <James_F> Further, the WikiEditor inserter is extremely "heavy", making it very hard to find the few normally-needed characters on a wiki, because they're muddled up with all the other characters that could theoretically be used, over-whelming the user.
[00:57:17] <James_F> So we wanted to try something a bit different and ask if the concept was workable, or if it was going in the wrong direction.
[00:57:24] <Shangkuanlc> Hello
[00:57:32] <Maggie_Dennis> Hello, Shangkuanlc. :)
[00:57:41] <James_F> We've had a bunch of suggestions, which have been very helpful, and which we're starting to work on.
[00:57:56] <Shangkuanlc> Is LeAnna in this channel now?
[00:58:07] <Shangkuanlc> Hi Maggie_Dennis :D
[00:58:32] <Maggie_Dennis> Shangkuanlc, I'm afraid she is not right now.
[00:59:01] <James_F> We've not yet come to a conclusion about whether we should try a different approach for the overall design, or if we can accommodate the suggestions made so far in the existing design.
[00:59:35] <James_F> (Also, we're not hugely happy with the current design from an æsthetic perspective, but that's not unusual for first-passes at tools in VisualEditor.)
[00:59:46] <James_F> So… to answer your question simply, Maggie_Dennis, "the latter"
[00:59:54] <Maggie_Dennis> Okay, thank you. I will pass that along.
[01:00:02] <Maggie_Dennis> And we are at time for this session.
[01:00:12] <James_F> Aha.
[01:00:14] <Maggie_Dennis> We'll be announcing next month's soon - as soon as we calendar them. :)
[01:00:26] <James_F> (Sorry, Maggie!)
[01:00:46] <Maggie_Dennis> If anybody has questions about VE that were not answered, please let me know.
[01:00:54] <Maggie_Dennis> We can try to get them answered in the interim or queue them up.
[01:01:02] <Maggie_Dennis> jayvdb, thank you so much for coming. You asked some great questions. :)
[01:01:08] <James_F> Also, if people think that a different time of day / week or schedule would be better for the IRC office hours for VE, do shout. :-)
[01:01:13] <Maggie_Dennis> James_F, thank you also for being here to answer them. :D
[01:01:19] <Shangkuanlc> Sorry, just came. I am using Chinese Wikipedia. Just want to let you know although there's small glitches, but Visual Editor is really awesome and I use that a lot now!
[01:01:34] <Maggie_Dennis> Thank you, Shangkuanlc! It's wonderful to hear that you are liking it. :)
[01:01:44] <James_F> Shangkuanlc: Thanks! We've got lots of tools for Chinese planned to come in the future, so hopefully you'll like it more!
[01:01:48] <Maggie_Dennis> I hope you will let us know when and if you find problems or have suggestions.
[01:01:55] <James_F> Shangkuanlc: Also, if you have glitches, please do report them in Bugzilla.
[01:02:12] <Shangkuanlc> How to report on Bugzilla?
[01:02:27] <James_F> Shangkuanlc: (Or on MediaWiki.org/wiki/VisualEditor/Feedback if that's easier for you.)
[01:02:55] <Shangkuanlc> I see, thanks for the link :)
[01:03:39] <Maggie_Dennis> Shangkuanlc, there is also information on how to use bugzilla here: https://www.mediawiki.org/wiki/How_to_report_a_bug
[01:03:45] <Maggie_Dennis> I found it difficult at first, but have gotten used to it. :)
[01:04:10] <James_F> Thanks everyone! Sorry if I didn't answer a question you had fully – do shout so I can do better next time!
[01:04:41] <Shangkuanlc> Thank you!