Talk:Lua deployments

Adding more wikis to Batch #1

Thank you Rob for the news. Would it be possible to include other wikis on the 18th trial? I'd like to add it.wikisource, we can easily reach consensus in these 2 days. --Aubrey 16 Feb 2013 12:01:20 (UTC)

I think this could be possible. I expect Tim or someone from WMF to update the list of wikis for deployment.--Kozuch (talk) 13:11, 16 February 2013 (UTC)

Performance

At en.wiktionary, there are many concerns about the performance of Scribunto, and many editors are hesitant to use it because of this. This isn't surprising, but it can be a problem when people don't want to move "forward" and experiment because they'd rather play safe. Is there anywhere that we could ask about how Scribunto works or what best practices are? There doesn't seem to be any kind of central discussion forum for Scribunto-related questions on all wikis, even though I think that would be highly beneficial. CodeCat (talk) 00:16, 18 February 2013 (UTC)

You can use the TemplateSandbox feature to test the performance and correctness of a template migrated to Scribunto. For example:

• Go here and copy the text of the new template to the clipboard
• Go to the same page on en.wp
• In the box under "Preview page with this template", type "Canis Minor"
• Click the "show preview" button to the right of the box you just typed in
• When the page loads, click "view source" in your browser. Scroll to the bottom, it will have a comment like "<!-- Served by mw1093 in 9.300 secs. -->"
• Close the source view
• Replace the contents of the template with the text from test2.wikipedia.org
• Click the TemplateSandbox "show preview" button again, open the HTML source view again
• Check the "served by" comment. It will show you the amount of time the page took to render with the Lua version of the template.
• In the HTML source, search for "NewPP limit report". There will be a line telling you how long Lua execution took, for example "Lua time usage: 0.127s". This is the total for all Lua modules used on the page, including the invocation you just inserted.

-- Tim Starling (talk) 05:49, 19 February 2013 (UTC)

We have already run into speed issues with Scribunto. See wikt:Module talk:languages. It seems that the module that is meant to contain this data is too large and ends up slower than if we just put it in templates. I don't know if we are using the wrong approach, or whether there is a speed bottleneck in Scribunto that needs to be fixed. CodeCat (talk) 17:59, 19 February 2013 (UTC)

The main performance problem of Lua (not really the Scribunto extension itself encapsulating it) is that Lua code is not really compiled but concerted to some bytecode which is then very slow to interpret (because Lua still does not feature a JIT compiler like Javascript or Java, or Python, or even a "safe" version of PHP with a restricted set of API functions). Also the instanciation of the Lua emgine takes long time, and it requires a significant amoint of memory.

I would then advocate to adapt Scritunto to support Javascript (because Javascript engines are much faster and have now very effiicent engines featuring JIT compilers generating native code that does not need to be interpreted at a bytecode level). Templates are much more efficient, even if they are costly. This could be used to allow generating complex layouts or presenting data coming for example from Wikidata or from other dynamic requests from a database.

Lua would be better used if it was able to generate contents that can be stored in some allowed namespace where it will be rendered more directly (without needing to reinstanciate a Lua engine, with some caching options controling when that content needs to be regenerated. Templates are faster because they are implemented natively in code.

Lua is easy to port because it does not feature any JIT compiler (this is also why it is very easy to secure). The next candidate will certainly be Javascript whose JIT compilers are extremely safe today.

verdy_p (talk) 00:04, 3 June 2014 (UTC)

Actually there are JIT Lua compilers, like LuaJIT. I think the stock Lua compiler is used though because of a desire for installation of Scribunto to be easy to do without extra work like patching the Lua compiler, and because Scribunto wasn't designed/tested to work with any other Lua compiler. Scribunto is also using a Lua 5.1 compiler rather then a Lua 5.2 compiler. --darklama 07:18, 3 June 2014 (UTC)

Lots of FUD here. "which is then very slow to interpret"^{[citation needed]} "Lua still does not feature a JIT compiler"^{[citation needed]} "the instanciation of the Lua emgine takes long time"^{[citation needed]} "and it requires a significant amoint of memory"^{[citation needed]} "because Javascript engines are much faster"^{[citation needed]} "Templates are much more efficient"^{[citation needed]} "Lua would be better used if it was able to generate contents that can be stored in some allowed namespace"^{[citation needed]} "Templates are faster because they are implemented natively in code"^{[citation needed]} Anomie (talk) 13:21, 3 June 2014 (UTC)

Footers, and using Lua to enable page-specific pseudo Wikimarkup

It occurs to me that Lua could be used to place Wikimarkup-like tags on a page that are interpreted quite generally.

For example, consider the en.wiki Refdesk. The way it works is that someone starts a section with a question, then multiple people try to answer it, some starting from scratch, some responding to / disputing others. It might also be desirable, but is not done, for posters to self-rate the level of their answer i.e. one star for "I did a search and this is what I found / this is what the article says", two for "I have some idea of what this is about from college, and three for "this is a professional-grade answer from someone who knows the field". More people like to make comical references or ask little sub-questions of their own about the previous answers. Suppose posters decided they wanted to use different formatting on different kinds of questions and answers, the formatting proceeding all the way from the beginning of one person's entry to the beginning of the next, or the start of the next question. Then it would be nice to have a script that recognizes and formats from a tag to a Lua-recognized signature or next tag or section header, e.g.

<Q> why is the sky blue?
<*> See Sky#Blue
<***> See this review in Phys Letters for the factors that affect atmospheric spectroscopy and how that can be used to analyze the climate
<c>Is there a layman's introduction to that?

But for the Refdesk example, while the {{#invoke could easily be stuffed into w:Wikipedia:Reference desk/header, sending the whole page off to a Lua script to be reformatted at will, there's still that pesty }} to deal with. Refdesk readers are often some of Wikipedia's newest and youngest editors, not to mention sometimes poor speakers of English, and there is no amount of text, imprecations, warnings and cursing that could ever persuade them not to forget the }} at the bottom of a new section, nor to remove it from an old one. In order for such a system to be used you would absolutely need some kind of footer to attach itself to the output. For all the edit, page, and group notices and different ways of formatting the headline I don't know if we actually have a way to tag that final two characters onto the end of the page. Can it be done?

More generally, is this a good idea to start or just a huge performance hit waiting to happen? Wnt (talk) 20:39, 18 February 2013 (UTC)

As I understand it, you're asking if Lua could be used to post-process whole pages? CodeCat (talk) 21:13, 18 February 2013 (UTC)

That would do, yes. The main point is that I'd like some way to process the whole page without needing every user adding a new section to have to have a "}}" at the end of it. Wnt (talk) 03:10, 19 February 2013 (UTC)

No, it is not possible. -- Tim Starling (talk) 05:51, 19 February 2013 (UTC)

Hey, I figured out a way! But it makes my head hurt. :) Presently at w:User:Wnt/Templates/Sandbox, though there's no way I know of to archive a Lua script output so it won't get messed with. The way you do it is you put an <includeonly> in the header of the page you want to markup, with no terminating </includeonly>. The header has a <noinclude> section that transcludes in a template. The template then transcludes in the thing that just called it - as a parameter to be sent to the #invoke. The Lua script then does a basic string.gsub() on the input string and returns it. The result is that all the stuff after the includeonly ends up appearing on the page that contained it, with the custom tags interpreted and replaced. Wnt (talk) 07:47, 22 February 2013 (UTC)

Audience?

Hi. Who's the intended audience of this page? It doesn't seem to be serving any audience well currently, but I'm unable to help make it better until I can understand who the page is intended to serve. Developers? Wikimedia wiki readers? Wiki template editors? Board members? --MZMcBride (talk) 08:05, 19 February 2013 (UTC)

Hi, I created this "yet another Lua page within WMF's wiki space" to primarily carry info about roll outs. I can imagine audience can be diverse, but the main purpose is to give info on Scribunto's roll outs to wikis, gather general deployment feedback etc.--Kozuch (talk) 08:39, 19 February 2013 (UTC)

Okay. Can it be moved to "Lua deployments" and the soft-redirect restored, then? Or maybe a disambiguation page at Lua? --MZMcBride (talk) 10:24, 19 February 2013 (UTC)

Well, http://meta.wikimedia.org/wiki/Lua is a link which was used for both wiki and mailing lists announce messages about current Batch #1 deployment... (I dont say the page name is OK though - I chose it to be rather short) so it would be nice if this link would still easily lead to this particular page for some more time... Or if you want to create a new disabmig then at least have a bold link to something like "Lua deployment"... but let us not loose this page's watchers (I guess moving this will move watchers too?). Cheers, --Kozuch (talk) 14:28, 19 February 2013 (UTC)

Someone asked a central discussion/coordination venue for Lua matters and this has already become one, so it makes sense to keep this title (I also hope there's no other scattered discussions happening in improper places like en.wiki). --Nemo 22:19, 19 February 2013 (UTC)

Lua page on English Wikipedia

For discussions and info sharing regarding Lua on English Wikipedia, please use w:Wikipedia:Lua in Wikipedia page.--Kozuch (talk) 10:21, 19 February 2013 (UTC)

Import warning

Tim enabled import of modules from test2 to enwiki (thanks). WOSlinker did an import of two modules that I'm interested in, but there was a warning displayed during the import. It did not seem to affect anything, but someone might like to check the text mentioned ("Import failed") at w:User talk:WOSlinker#Importing Module:Convert. Johnuniq (talk) 11:53, 19 February 2013 (UTC)

The rest of the wikis

When will it be rolled out on the remaining wikis? Njardarlogar (talk) 18:12, 20 February 2013 (UTC)

Since the fist batch deployment seems to be without any problems at all I think next wikis will follow within several weeks.--Kozuch (talk) 09:45, 21 February 2013 (UTC)

Is there any way to get user input into a Lua script?

The way I understand it, a Lua script can do computations on any numbers you have actually edited and entered into a Wiki page as a revision. However, so far as I know, if you make a Lua page display a template and parameter, or a Wikilink, or whatever, that cannot be parsed into any kind of clickable link. So for example, someone tried to make a tic-tac-toe program, but there's no way to actually enter your moves, unless you actually had a premade source for each click in each state, i.e. so that you could play it as straight Wiki markup without Lua. There's no way to make a Lua script to convert Fahrenheit to Celsius that doesn't involve either you typing in the number you want to convert as an actual page revision, or else by having a different link to click pre-written for every value you might want to click on. Am I wrong? Wnt (talk) 01:35, 21 February 2013 (UTC)

You are correct. The Lua code in Scribunto is only run when the page is being parsed, the input is only what is passed as parameters or what can be transcluded (although Lua scripts can access parameters passed to the parent template), and output is wikitext that is incorporated into the page being parsed.

You can use the debugging console at the bottom of Module-namespace edit pages to evaluate Lua code in a manner more like what you're asking, but that's really only useful for debugging. Anomie (talk) 15:12, 21 February 2013 (UTC)

It would be great to let users try examples in article text, but server-side Lua probably isn't the best solution (unless it uses AJAX perhaps). πr² (t • c) 15:14, 13 March 2013 (UTC)

To do: rewrite the manual

In my humble opinion, the Mediawiki Scribunto/Lua reference manual is not very good for Wikipedians. Namely,

• "Syntactic sugar" apparently means synonym. i.e. providing a.name as syntactic sugar for a["name"] this indeed delivers "testing 1 2 3".
• var ::= prefixexp `[´ exp `]´ and such "Backus–Naur Form" pseudo language is essentially useless to anyone who actually needs to read this manual. You'd need another manual to understand the manual.
• Any example involving "print()" will fail.
• It doesn't describe the things Lua can't do, e.g. work with user input that isn't edited into a page.
• Examples are not linked to working code.
• It isn't on Wikipedia.

I suggest that the entire document should be copied in full to Wikipedia and edited mercilessly, with an eye toward documenting every single usage given in the text with a separate, short, simple module. The modules should have some consistent naming like "Tutorial for manual section 4.2.1" so they aren't confused with working modules for mainspace.

Starting this project would be easy (copy and paste!) - finishing it very hard - so it would be nice to get a sense of whether it is an idea that appeals to people so that it doesn't turn into a dead end. Wnt (talk) 07:33, 24 February 2013 (UTC)

Err, I just did that last week. Did I miss removing some of the BNF, or examples with print()? Or is there another copy somewhere besides mw:Extension:Scribunto/Lua reference manual? Personally, I don't feel it necessary to enumerate all the myriad things Lua-in-Scribunto can't do. Anyway, feel free to edit that copy, and update any links.

Re "syntactic sugar", it's specifically a synonym for something in a language that isn't actually necessary but can sometimes make code easier to read or write. When I rewrote the manual, I wikilinked it for people unfamiliar with the term. Anomie (talk) 17:02, 25 February 2013 (UTC)

Great! Sorry, it didn't dawn on me someone had already done this since I last looked - thanks, this is a big step in the right direction! Wnt (talk) 21:22, 25 February 2013 (UTC)

Why does it need to be on Wikipedia? It's about a MediaWiki extension, not specifically about Wikipedia. Feel free to make a manual that is more about your Wikipedia's policies and practices in using Lua. πr² (t • c) 22:45, 12 March 2013 (UTC)

Well, for example, as mentioned below, it seems like mw.ustring.x should be used in preference to string.x for any processing of Wikitext, so the Wikipedia manual should introduce all of the relevant concepts (pattern syntax) directly in that section. Wnt (talk) 14:43, 13 March 2013 (UTC)

I've since made a brief effort at w:Help:Lua for beginners. Wnt (talk) 17:47, 24 March 2013 (UTC)

I wonder if Wikiversity's Lua Course, Wikibooks' Lua Programming Book, and/or Wikibooks' Scribunto: An Introduction Book might be worth mentioning somewhere for anyone wanting to learn Lua, but finds the reference manual impossible to learn from. --darklama 23:30, 2 June 2014 (UTC)

Random errors

I was fooling around with Lua processing of some large Refdesk files at w:Template:RDIndex and I keep getting random sets of errors. Out of 31 days of Refdesk answers from zero to six come out with errors every time I change it. Sometimes the thing even returns "array", from a variable "output" that I do nothing with but concatenate stuff to it! Even adding a comment in the script [1], or a noinclude section on the page with the invoke, can make these errors turn up, and it's not on the same days each time. They come out in two string.find commands, even when I check tonumber() for the third parameter right before putting it in. It's very possible my program has a real bug in it, but the randomness of this makes me think that there's something off kilter in the Lua itself. Wnt (talk) 21:30, 25 February 2013 (UTC)

Huge thanks to User:Dragons flight for a major rewrite which fixed this module and improved its efficiency. A comment he left on my talk page indicates the problem may stem from using string.find rather than mw.string.find - if so, the former seems not merely limited, but out and out buggy, and it may be worth telling other new users to skip straight over it. Wnt (talk) 16:04, 26 February 2013 (UTC)

No, the string module is not buggy. It works exactly as it is documented. And part of that is that it deals with bytes rather than multi-byte characters. Without having been able to see the errors you were getting, I can't say whether that was even your problem. Anomie (talk) 16:43, 26 February 2013 (UTC)

Another way of formulating what Anomie said, is that regular Lua strings do not support unicode aka non-ASCII letters. The edit of Dragons flight did indeed introduce mw.ustring(), which was created to support unicode. Njardarlogar (talk) 16:41, 27 February 2013 (UTC)

Well, it's just that by "do not support" I was expecting some kind of error in the module when it hit an input file with a Unicode character in the wrong place - not having the thing sometimes return "array" from a set of string concatenations, and run into errors randomly when re-run with the same input. I think the layman's summary here is that the plain string functions do not work on Wikipedia. If we had our own manual I'd say, reduce their entry to that one line, and describe all the features of strings in one place under the mw.ustring functions. Wnt (talk) 16:51, 10 March 2013 (UTC)

Randomly returning an array from string functions sounds like something entirely different, that being the bug that was fixed in 51331. Anomie (talk) 13:13, 11 March 2013 (UTC)

Tracking what uses a module

How do you track what pages actually #invoke something like w:Module:math? "What links here" delivers practically an alphabetized list of Wikipedia articles. Searching for "{{#invoke:math" gets you nothing - the funny characters seem to get stripped out and/or confuse the search. Searching "math" is left as an exercise for the student. The problem is ... if you don't know what #invokes a module, how can you tell you haven't broken one of the pages if you change something? Wnt (talk) 15:11, 11 March 2013 (UTC)

"What links here" for modules works exactly like it does for templates, to the point where {{#invoke:}} calls are reported as transclusions. w:Module:math actually is used in all those pages, via various templates; you may note the great similarity between w:Special:WhatLinksHere/Module:Math and w:Special:WhatLinksHere/Template:Max, for example. Anomie (talk) 13:09, 12 March 2013 (UTC)

Right, but how do I figure out if there's another template or six - Template:Min, Template:Mid, Template:Wazzis ... It's not just a matter of finding out all the pages that eventually get to a template - it would be useful to know which templates directly #invoke it. Wnt (talk) 22:17, 12 March 2013 (UTC)

It would be nice to know which templates directly invoke some subtemplate, too. Chances are if you solve one of those problems you can solve the other too. Anomie (talk) 13:10, 13 March 2013 (UTC)

Due to the way the wikicode is parsed, it's almost impossible to track where templates or modules (or even wikilinks) are effectively present, because these entity names are conditionally generated by code in several places simultaneously. The only trackable use is not where these entity names are present in the Wiki code, but the fully rendered page.

It's extremely complex to track the effective references, except with the most basic syntax (you may think these kinds of basic invokations should be trackable where they do not depend on complex conditions, but it is complex when most templates use contextual conditions such as noinclude/includeonly conditions, unless MediaWiki is deeply modified to explore all combinations of conditional contexts, something that could cause huge problems of performance and could require lot of storage space to explore all these conditional contexts exploding exponentially: Mediawiki allows very deep levels of transclusions and references, with 40 levels a single template could generate millions of possible contexts if you count conditional contexts like #if, #switch, includeonly/noinclude, and hundreds of UI languages, and some other state variables constantly changinf like time, or wiki statistics like counters of pages in namespaces, or counters of members in categories, or user rights). So not all can be stored, much of this tracking data is only present temporarily in LRU caches and are thrown away rapidly to save storage space. verdy_p (talk) 23:47, 2 June 2014 (UTC)