Planet Perl Six

Parrot 5.4.0 "Austin Parrot" Released! by Bruce Gray

2013-05-22T18:51:15+00:00

Jimi Hendrix, deceased, drugs.
Janis Joplin, deceased, alcohol.
Mama Cass, deceased, ham sandwich.
-- Austin Powers (making a list of friends from the Summer of Love)

On behalf of the Parrot team, I'm proud to announce Parrot 5.4.0, also known
as "Austin Parrot". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages, and currently focusing on Perl 6.

Parrot 5.4.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/5.4.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 5.4.0 News:
- Core
+ Implemented the coth() and acot() math functions.
+ Fixed chomp to only trim a newline when it ends the string. [GH #958]
+ Added readlink() and Parrot_file_readlink(), with tests. [GH #967]
- Build
+ Parrot now detects the CPU model on Linux systems, as well as
detecting more CPU models on BSD, Cygwin, Solaris, Win32, and Darwin.
ARM v7 is also now recognized. [GH #962]
- Documentation
+ Threads examples now have proper POD sections and useful descriptions
with links to references.
+ Added main description for Task PMC.
+ Added descriptions to trig methods in Float PMC.
- Tests
+ Added improved test coverage targets "cover_new" and "fullcover_new".
+ Improved tests for acot(), coth(), acot() math functions.
+ Added tests for options passed to debugger.
+ Updated native PBC test files for string, number, and integer,
which resolved 11 TODOs in the test suite. [GH #959]
+ Fixed test for the auto/arch config step.
- Release process
+ Added message digests to crow.pir.
+ Added in release.json: "release.type" can be "devel" or "supported".
+ Refactored common code to sub in auto_release.pl.
- Community
+ Parrot is part of the Hackathon at YAPC::NA::2013, in Austin, TX, USA!
http://www.yapcna.org/yn2013/wiki?node=Hackathons
+ Parrot has been accepted to Google Summer of Code 2013!
+ Currently there are two high-quality proposals being worked on:
https://gist.github.com/sa1/5468408 - parrot-libgit2
https://gist.github.com/denisboyun/5472762 - App::Parrot::Create

The SHA256 message digests for the downloadable tarballs are:
4e37686911b446f5e5f2c0aa62138988ba0c411d2c5e2ba231d1a3421a85ad10 parrot-5.4.0.tar.gz
91d0e46fe3ef08e692e80756f26ee0e7311fe58e49d6c31f3f5180d4eb475696 parrot-5.4.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 18 Jun 2013.

Enjoy!

Perl 6 changes - 2013W20

2013-05-18T07:00:00+00:00

This week, the most notable change would be Rakudo sort of working on Java Virtual Machine. It still needs lots of improvements, but it can already run “Hello, world” program, what is important milestone in Rakudo for JVM.

id='incompatible_changes'>Incompatible changes>

Perl 6 specification>

You cannot create a set or bag from list of pairs or hash using a constructor.

The `bag {a => 2}` was ambiguous, because it could mean either `bag` containing `'a'` with value `2`, or `bag` containg `{a => 2}` with value `1`.

You’re expected to use `.Bag` or `.Set` instead, if you mean that.

STD.pm6>

You cannot use `set` or `bag` to mean empty set or bag.

If you really meant that, you are expected to use `set()` or `bag()` instead.

id='new_features'>New features>

Perl 6 specification>

Slice adverbs support false booleans.

You can write code like `%a :!exists` to check whatever the element doesn’t exist.

You can also write code like `%a :!delete` which does nothing, but can be used for conditionally deleting hash keys (for example `%a :delete(%a<c> :!exists)`).

Sets and bags use `===` semantics.

Previously the specification hasn’t said which semantics should be used. As anything other than `===` semantics would be slow, `===` semantics are used.

Defined `INIT`/`END` across modules.

Added `FINAL` and `COMPOSE` phases.

`FINAL` is ran at link time. `COMPOSE` is ran when a role is composed into a class.

STD.pm6>

`set<a b c>` has special error message.

This syntax wasn’t useful for anything, except for perhaps obfuscation. It meant creating empty set, and getting keys `'a'`, `'b'`, and `'c'` from it. As the empty set doesn’t have these keys, it was returning `False` 3 times.

As the behavior doesn’t make any sense, you probably wanted to use `set <a b c>` instead. The old behavior is still accessible using `set()<a b c>`, but I don’t think you would want to use this.

Perl 5 grammar allows version in `require` statement.

In Perl 5, you can write code like `require 5.010` do make runtime version check (as opposed to `use 5.010` which is compile time check). The Perl 5 grammar, included as part of `STD.pm6` now supports this syntax.

`FINAL` and `COMPOSE` phases is now recognized.

Now that these phases are in specification, the grammar was modified to recognize them.

Rakudo Perl>

JVM version of Rakudo supports simplest “Hello, world!” program.

Fixed infinite recursion in `IO::Spec::Unix.rel2abs`.

`rel2abs` made infinite recursion if neither `$path` or `$base` were absolute.

`infix:<~>` is constant folded.

`<prefix:<~>` is constant folded.

String comparison operators are constant folded.

String bitwise operators are constant folded.

`defined` is constant folded.

`Set`, `KeySet`, `Bag`, and `KeyBag` have `default` method.

When reading undefined property of those classes, `default` method contains value that you should receive. In case of sets, it’s `False`. For bags, it’s `0`.

Regexpes support more delimiters.

Previously, Rakudo didn’t accept regexpes like `m^42^` or `m€cash€`. Now, they are allowed, just like in Perl 5.

Niecza>

`Set`, `KeySet`, `Bag`, and `KeyBag` have `default` method.

Just like in Rakudo, except with Niecza.

Rakudo on JVM progress update, and some questions answered

2013-05-14T23:29:33+00:00

It’s time for another progress update on the ongoing JVM work. Last time I posted here, we’d reached the point of having a self-hosting NQP ready to merge into the master branch of the NQP repository. That has now happened, so the May release of NQP will come with support for running on the JVM (note, this does not mean the May release of Rakudo will come with this level of capability, and a JVM-based Star release with modules is further still!) . In this post, I will discuss some of the things that have happened in the last few weeks and also try to answer some of the questions left behind in the comments last time.

The Rakudo Port

With NQP pretty well ported (there are some loose ends to tie up, but it’s pretty capable), the currently ongoing step is porting Rakudo. At a high level, Rakudo breaks down into:

The core of the compiler itself: the grammar (for parsing), actions (which assign semantics to the things we parsed), world (which takes care of the declarative aspects of programs), optimizer (tries to cheat without getting caught) and a few other small pieces to support all of this. This is written in NQP.
The Perl 6 MOP (meta-object protocol) implementation, which defines what classes, roles, enums, subsets and so forth mean. This is also written in NQP.
The bootstrap, which uses the MOP to piece together various of the core Perl 6 types. It does Just Enough to let us start writing Perl 6 code to define the rest of the built-ins. Also written in NQP.
The core setting, which is where the built-in types, operators and methods live. This is written in Perl 6.
A chunk of per-VM code that does lower-level or performance-sensitive things.

The first 3 of these need…an NQP compiler. And, it turns out that we have one of those hosted on the JVM these days. But could it handle compiling the Perl 6 grammar? It turns out that, after some portability improvements to the various bits of Perl 6 compiler code, the answer was a resounding “yes”. In fact, the grammar was compiled down to JVM bytecode sufficiently accurately that I didn’t encounter a single parse failure on the entire CORE.setting (though there were lots of other kinds of failures that took a lot of work – more on that in a moment). As for the rest of the compiler code, there were bits of lingering Parrot-specific stuff throughout it, but with a little work they were abstracted away. By far the hardest was BOOTSTRAP, which actually runs a huge BEGIN block to do its setup work and then pokes the results into the EXPORT package. This is kinda neat, as it means the setup work is done once when building Rakudo and then serialized. Anyway, onto the next pieces.

Compiling the Perl 6 setting depends on the Perl 6 compiler working. Since the first thing the setting does is use the bootstrap, which in turn uses the MOP, it immediately brings all of the above three pieces together. While we talk about “compiling” the setting, there’s a little more to it than that. Thanks to various BEGIN time constructs – such as traits, constant declarations and, of course, BEGIN blocks – all of which show up in the CORE setting – we actually need to run chunks of the code during the compilation process. That’s right – we run bits of the file we’re in the middle of compiling while we’re compiling it. Of course, this will be nothing new to Perl folks – it’s just most Perl programmers probably don’t worry about how on earth you implement this. :-) Thankfully, it’s a solved problem in the NQP compiler toolchain, and the stuff that makes NQP BEGIN blocks work largely handles the Rakudo ones too.

Anyway, all of this means that even getting the setting to finish the parse/AST phase requires doing enough to run the various traits and so forth. And that in turn brings in the fifth piece: the per-VM runtime support. This includes signature binding, container handling and a few other bits. Thankfully, it no longer involves multiple dispatch, since that is written in NQP these days (apart from some caching infrastructure, which is shared with NQP’s multiple dispatch, and thus was already done). Getting through the parse/AST phase of the setting doesn’t need all of the runtime support to be implemented, but it did require a decent chunk of it. Of course, at the start everything is missing, so getting from line 1 to line 100 was O(week), from 100 to 1000 O(day) and each thousand or so from there O(hour). It’s 13,000 or so lines in all.

The parse/AST step is just the first (though biggest) phase of compiling Perl 6 code, however. Next comes the optimizer, followed by code generation. In theory, the optimizer could have been bypassed. I planned to do that, then discovered it basically worked for the set of optimizations that didn’t need signature binding to participate in the analysis, so I left it in. Code generation is part of the backend, and so is shared with NQP. So it shoulda just worked, right? Well, yes, apart from code generation is the place where nqp:: ops get resolved to lower level stuff. And Perl 6 uses a lot more of them than NQP. Note that they only have to be mapped to somewhere; the JVM is late bound enough that it won’t complain unless you actually hit the code path that tries to use something that is not yet implemented. In reality I did a bit of both: implementing those that would surely be hit soon or that were trivial, and leaving some others for later.

So, some time yesterday, I finally got to the point of having a CORE.setting.class. Yup, a JVM bytecode file containing the compiled output of near enough the entire Perl 6 core setting. So, are we done yet? Oh, no way…

Today’s task was trying to get the CORE setting to load. How hard could that be? Well, it turns out that it does a few tasks at startup, most of which hit upon something that wasn’t in place yet. Typically, it was more of the runtime support, though in some cases it was unimplemented nqp:: ops. Of course, there were a handful of bugs to hunt down in various places to.

Anyway, finally, this evening, I managed to get the CORE setting to load, at which point, at long last, I could say:

perl6 -e "say 'hello world, from the jvm'"
hello world, from the jvm

Don’t get too excited just yet. It turns out that many other simple programs will happily explode, due to hitting something missing in the runtime support. There’s still plenty of work to go yet (to give you an idea, trying to say a number explodes, and a for loop hits something not yet implemented), but this is an important milestone.

Interop

A couple of the comments in response to my last post asked about interop with Java. There’s two directions to consider here: using Java libraries from Perl 6, and using Perl 6 code from Java. Both should be possible, with some marshalling cost, which we’ll no doubt need to spend some time figuring out how to get cheap enough it’s not a problem. It may well be that invokedynamic is a big help here.

The Java stuff from Perl 6 direction can probably be made fairly convenient to use by virtue of the fact that Perl 6 has a nice, extensible MOP. The fact the object you’re making a call on lives in Java land can be just a detail; we can hide it behind the typical method call syntax, and should even be able to populate a 6model method cache with delegation methods that do the argument mapping. I’m sure there will be plenty of interesting options there. I suspect we’ll want to factor it a little like NativeCall – some lower level stuff in the runtime, and some higher level sugar.

Going the other way will be more “fun”. I mean, on the one hand the marshalling is just “in the other direction”, which we’d need to do for values coming back from Java land anyway. But trying to work out how to make it feel nice from Java land could be trickier. I don’t believe the “.” operator is very programmable, which probably leaves us with string lookups or code-generated proxy thingies. Or maybe somebody will come up with a Really Great Solution that I hadn’t thought of.

My JVM related Perl 6 dev focus for now will be getting Rakudo to work decently and start getting Perl 6 modules working on the JVM also, but interop with Java land is certainly on the roadmap of things I think should happen. As with all things, I’m delighted to be beaten to it, but will work on it if it goes undone for too long. :-)

Performance?

The first thing to say on this is that it’s too early to have a really good idea. The final pieces of the gather/take transform (which has global consequences) have yet to land, which will certainly have some negative impact and will need to happen soon. At the same time, I’ve been very much focused on making things work on the JVM at all over making them especially clever or optimal. Numerous things can be done in ways that will not only perform better in a naive sense, but that will also be much easier for the JVM’s JIT to do clever stuff with. There are many, many things we will be able to do in this area.

Since I only have a sort-of-working Perl 6 compiler, I can’t say that much about Perl 6 performance. The only result I have to share is that the CORE setting parse completes in around a third of time that it does on Parrot (noting it’s not only about parsing, but also some code generation and running of stuff). This is not especially great – of course, we need to do better than that – but it’s certainly nice that the starting point before I really dig into the performance work is already a good bit faster.

The other result I have is NQP related. nwc10++ has been doing performance testing with a Levenstein benchmark of each commit to the NQP JVM work, which is really great in so far as it gives me a rough idea if I accidentally regress (or improve ;-)) something performance wise. There, we saw a larger factor performance win, around 15 times faster than the same program running in NQP on Parrot.

The big negative news on performance is startup time. Part of this is just the nature of the JVM, but I know another enormous part of it is inefficiencies that I can do something about. I’ve plenty of ideas – but again, let’s make things work first.

From Here

Things will be a little quieter from me over the next week and a bit, due to a busy teaching schedule. But now we have a fledgling Rakudo on JVM, and from here it’s going to be making it gradually more capable, first passing the sanity tests, then moving on to the spectests and the ecosystem. There are ways to help for the adventurous. Some ideas:

Profile the code generation phase, which is one of the pieces that is slower than expected. Try to figure out why.
Have a look at how multiple dispatch stuff is currently set up, and see if the dispatch logic could possibly be shuffled off behind invokedynamic.
Try something. See how it explodes. See if you can fix it. (Yes, generic I know. :))
Have a look at a “make install” target for Rakudo on JVM.

I’ll be speaking on the JVM work at the Polish Perl Workshop the weekend after next, and hope to have something a bit more interesting than “hello world” to show off by then.

Perl 6 changes - 2013W19

2013-05-11T07:00:00+00:00

So, well, what’s new. I’m going to answer - lots of changes and fixes. Even the Niecza, where lately development had slowed down got lots of fixes.

id='incompatible_changes'>Incompatible changes>

Rakudo Perl>

Array references aren’t treated as arrays in regular expressions.

Following code used to find 1, 2, or 3.
```
my $array = [1, 2, 3];
my $number = prompt 'Give me a number!';
say "It's in array" if $number ~~ / ^ $array $ /;
```
It doesn’t do that anymore, because variables are expected to be stringified inside regexpes. If you want the old behavior, be explicit and write @$array instead of $array.

Please note that currently regular expressions are exception to this rule. This is going to be fixed in the future. Lots of code depends on regexpes to be not stringified, and first the warnings will have to be implemented. You are expected to use <$regexp> instead of $regexp.

Niecza Perl>

Changed `capitalize` to `wordcase`.

In the current specification, that method is named `wordcase`.

Removed `ucfirst`.

`ucfirst` was long deprecated, and it’s finally time to remove it. You are expected to use `tc` instead.

Stringification of `Set`, `KeySet`, `Bag`, and `KeyBag` works properly when they are undefined.

Methods `Str`, `gist` and `perl` expected they deal with defined object. This isn’t the case when dealing with the classes themselves.

Deprecated `$%` and `$@` variable checkers are working just like in STD.

The `$@(1)` was complaining about use of deprecated variable, when the real problem was use of invalid hard reference syntax.

id='new_features'>New features>

Rakudo Perl>

`IO::Spec::Win32` no longer warns when using relative pathes.

The check for initial prefix was using optional alternative (`([a | b | c])?`). Because of that, it was `Nil` if it matched nothing. Since the result of match was stringified just after it was matched, it returned a warning when `?` matches nothing. It was fixed to do `([a | b | c]?)` instead.

Added tests for `IO::Path` and `IO::Spec` to list of tests that should pass.

As `IO::Path` and `IO::Spec` were implemented, their tests are now ran when you request full test from roast test suite.

Interpolation of quotes in regexpes isn’t done twice anymore.

Previously, quotes inside RegExpes were doing interpolation two times.

`match` method is fast again.

https://github.com/rakudo/rakudo/commit/c255f1df the older commits) made it 10 times slower. The performance regression was fixed, and now `match` is as fast as it used to be.

`wordcase` method now accepts `where` which is `Mu`.

Previously, only `Any` values were accepted. While using `Mu` doesn’t make much sense, there is no reason to disallow it, so it’s allowed now.

Configuration instructions don’t say to use –gen-nqp.

It was pointless, since `--gen-parrot` automatically implied `--gen-nqp`. If you want Parrot, it’s safe to assume you want NQP for that Parrot too, right?

`our` hashes and arrays work.

If you had used `our @array`, and `our %hash`, the object wasn’t initialized properly, and it wasn’t working like array or hash. This made `our` work only with `$scalar`s previously.

Previously, trying to use `our` with arrays or hashes caused strange uninitialized variable to appear. It didn’t have `STORE` method, so you couldn’t have put anything in this variable, and had `Any` instead of empty array of hash.

It’s now possible to augment empty list.

`MONKEY_TYPING` couldn’t be used to modify lists. Now it can.

Niecza Perl>

`Pair.perl` can now deal with the cases when `Pair` is the key.

Doing `((a => 'b') => 'c').perl` used to output `"a" => "b" => "c"`, what actually meant `(a => (b => 'c'))`. `.perl` now outputs correct code in that case.

Implemented Hash.ACCEPTS.

After the #masakism workshop

2013-05-05T21:22:45+00:00

On May 1st I learned what happens if you tell a bunch of people on the Internet, not all of whom you've met before, that you're going to teach Perl (5 and 6) for free on an IRC channel for four hours.

It worked well. Actually, it worked well beyond my expectations.

Successes

Prepare. I had written a bunch of material; short-ish texts mixed with exercises. I didn't hear any feedback about the texts, but people threw themselves on the exercises. Which was lucky, because I basically didn't have time to interact much with people. Everyone did their own thing.
Map. The week before the workshop, I was taking a walk, and it occurred to me that the topics for the workshop should be laid out on a subway map. That turned out to work unexpectedly well. People followed the structure, I think. The material was pretty clear that they were free to skip/jump around if they wanted, so some did that.
Github. People suggested fixes and improvements to the material during the workshop. I handled the easy ones, and delegated what could be delegated. If anything about the workshop felt 2013-futuristic, it was the fact that participants were hacking on the workshop in real time, as it was being played out. Github was totally right for this.
- tadzik++ did an exercise and got a solution that I didn't expect. We agreed that a test was missing to force people to do the right thing, so he went ahead and added it.
- LlamaRider++ added hyperlinks between all the files.
- grtodd++ made the subway map clickable.
- choroba++ added hyperlinks to a file that I added later.

Things learned

Newbies. This was never a workshop for total beginners. Still, we got a number of those. I'm not sure how many. If there's a next time, I'll want to add a track for the people who haven't done Perl before.
Exercises. Based on the tasks that people actually solved, I got a lot of feedback on what worked and what didn't. Which exercises served the overall message of the course, and which ones didn't.
Message. After the workshop, I know much better what I wanted the whole thing to be about. I think I can go back and make that even more clear. I'm still surprised at how well it worked already the first time around... but some bits in there can certainly be improved.

Message

So... what was the real message of the workshop? What is "masakism"?

Two things:

Testing gets you far.
- My pain threshold for writing things without tests is half a screen of code. If I have more than that, there's probably something I should be writing tests for.
- That said, there's no need to be a fundamentalist about anything. I'm just saying that tests are good, and still somewhat undervalued sometimes. But it's always a question on spending time testing the stuff that matters. Of course.
- Even with tests, there's no substitute for knowing what you're doing and where you're going. Tests help a lot, but they don't write the program for you.
Keep it small and simple.
- "Inside every big interesting problem is a small interesting problem, struggling to get out."
- The realization that you can write objects that focus entirely on your core problem, is a strong one.
- People tend to bring in persistence too early, "polluting" their core classes with database code. Hindering testability. This is backwards.
- Your program should be like an onion, with the most precious stuff in the middle, and layers around that. Outer layers should point to inner layers, but not vice versa.

Once more?

So, should we do another #masakism workshop?

Yes, maybe we should. People seemed to like this first one. I'm open to finding a datetime for another one.

If you have any suggestions, get in touch.

Perl 6 changes - 2013W18

2013-05-04T07:00:00+00:00

Perl 6 gets more and more stable, less features are added, and the features that are left are worked on. Also, Rakudo Perl is being ported to JVM.

id='new_features'>New features>

Perl 6 specification>

Updated IO:: classes documentation pages.

To many minor changes to describe, just follow the link.

Added `IO::Spec`, `IO::Path.cleanup`, and `IO::Path.resolve`.

Flip flops now use smartmatch semantics instead of boolean semantics.

Now you can write code like `"BEGIN" ff "END"`, instead of using regexes for that.

STD.pm6>

Use of & in infix position is now catched after whitespace.

Previously, code like `$some-object.some-method &sub-routine` appeared to work. However, it didn’t meant what you would expect, it didn’t called the method with subroutine argument, but assumed junction and. In order to avoid surprises, this is now disallowed.

The message about no parens or colon after method name is more specific.

Now code like `$some-object.some-method 42` shows information that colon or parens are required.

Added lots of function declarations.

The STD now recognizes certain functions that were added lately.

`is-prime`

`expmod`

`mkdir`

`rmdir`

`socket`

`listen`

`accept`

`connect`

`bind`

`infix:<leg>`

`infix:<cmp>`

Added `LoL` class and `X` role.

Rakudo Perl>

Implemented IO::Spec.

This class is intended to do OS-specific manipulation of pathes. It can get the localizations of special folders (for example home directory, root directory, temponary directory), and can manipulate pathes (by joining them, spliting them, converting them to absolute or relative, and so on).
Implemented IO::Path::<os subclasses>.

Thanks to this change, the pathes objects are OS specific. You can have path marked to be Windows path, so you can parse it in OS specific way, without having to use that OS.
Repeated shift is faster.

On large arrays, shift is now noticeably faster. For example, if array @a has 50_000 elements, running following snippet of code is processed within 2.5 seconds (and not 9.0 seconds).
```
@a.shift while @a;
```
Failed regex matches now return Nil.

Previously, failed regex matches were returning the Match object that stringified to #<failed match> and was converted to False.
&diag function in Test.pm accepts Mu.

This, I think is self explanatory. &diag is a debugging function, outputing its argument. Forbidding Mu didn’t made sense, so it’s now allowed.
Exporting of subroutines now works properly.

use Module qw(sub1 sub2) should now work.
Fixed dir() to work on Windows.

This makes panda work on Windows.

To merge or not to merge?

2013-04-28T18:04:29+00:00

I have a problem: Should I keep the 3 Perl Maven sites ( Perl 5 Maven, Perl 6 Maven, and Perl Maven) separate, or should I merge them together? Or maybe I should merge P5M into PM and leave P6M separate.

For the full article visit To merge or not to merge?

Perl 6 changes - 2013W17

2013-04-27T07:00:00+00:00

Continuing, still with commit IDs.

id='incompatible_changes'>Incompatible changes>

Perl 6 specification>

`DateTime` doesn’t accept `Callable` timezones anymore. 5692ced333

Rakudo Perl>

Renamed `IO` to `IO::Handle`. `IO` itself is a role. 2dca6ed950 37c9956502

`DateTime` doesn’t accept `Callable` timezones anymore. 71ea142406

id='new_features'>New features>

Rakudo Perl>

Implemented `Cool.lines`. 0d1d1194a4

t2: Rectangle haikus

2013-04-25T20:50:50+00:00

sheep hide at bad rains

bitterly modest slumber

its star to their rains

     -- generated by one of the t2 solutions

How time flies. It's two months since I reviewed the first batch of p6cc solutions. I solemnly swear the next blog post will appear sooner than that.

## Generate rectangle haikus

Write a program that generates haikus. A haiku has three lines, where the first
and last lines have five syllables each, and the middle line has seven. For the
purpose of this exercise, each line consists only of letters and spaces.

The haikus generated have the additional requirement that each line be of equal
length. The length requirement should not be gamed in any way, for example by
padding lines with spaces.

    since this one does it
    rectangle plus a haiku
    it serves as fine text

The program should attempt to generate a new haiku each time. The haiku should
consist of English words. If the words make sense in some kind of sentence
structure, that's considered a big bonus. Humor and/or deeper meanings are even
bonuser.

If you attempt to cheat at this task, you will be defeated by people who don't.

You're free to supply your own wordlist. How you count syllables is up to you,
and part of the task. You won't be challenged on trivial differences in
syllable counts.

It's OK for the program to run for a while, but it should preferably terminate
within a reasonable span of time.

I warmly recommend people to check out the solutions to this one. People are all over the place, coming up with ways to generate a rectangular haiku.

Some patterns recur, though. Let's break things down into subtasks and discuss them separately.

Word lists. Some contestants had their wordlists inlined in the code. Others had it in a separate file. Some stored part-of-speech information together with the word, or syllable count.

Sentence structure. Solutions are all across the spectrum, all the way from "just spew words", via "generate something that could perhaps pass for a sentence", all the way to "try really hard to be grammatical". In my view, being more high-end on this bit pays off big in quality. But it also costs in code complexity and program speed.

Search. This is a biggie, because it bleads through into the whole design of the program. Some of the solutions simply brute-force it. Others generate stuff randomly but store things in a hash table, guaranteeing that eventually some triplet of lines will form a haiku of some length. Some algorithms do clever things here, like turning problems around, going from "how many syllables does this line have?" to "ok, give me a word with this many syllables". One solution partitions the integers 5 and 7 in all possible ways, and hard-codes the resulting table.

But the really interesting bit in all solutions is the syllable counting. This is where the participants really differ in approach. One went with porting the CPAN module for counting syllables. Someone else did the same, but put the module on modules.perl6.org for everyone to use. Two contestants seem to have come up with their own (flawed) syllable-counter. A few people evade the syllable-counting by just storing the values along with the word lists. (Which is fine.)

All in all, an interesting set of solutions to a fun problem. Somehow, after reviewing all of these, I have the distinct feeling that a "best" solution could be put together by combining the best bits of everyone's solutions. I might just do that myself, if no-one beats me to it.

Next up: arranging wire crossings to rearrange wires!

The #masakism workshop

2013-04-24T22:37:50+00:00

So, as part of my $dayjob, I was teaching a Perl course. Then this happened:

* masak is teaching Perl 5 today! :)
<Heather> masak teach me
<masak> Heather: later; these people are paying me. :P
[...]
<masak> seriously though; if there were enough interest here on IRC, I
        would totally do a 1-day Perl 5 IRC workshop, pro bono.
<masak> ditto a one-day Perl 6 IRC workshop.
<vaelxon> (Perl 6 IRC workshop)++
<masak> it'd be very awesome, I promise. I'd need some time to prepare.
        but then we can grab a channel and basically party/learn for a
        day together.

And that's now what's happening. It'll be a loose-knit group of us, joining up in the #masakism IRC channel for four hours on May 1.

The course will be two-pronged; Perl 5 or Perl 6, whatever people prefer.

There will be prepared study material and exercises, but basically, people who participate can go off on a tangent, just spend the four hours solving one of the exercises really well, or solving a different exercise that they make up themselves. People can focus on doing the exercises, or on discussing various ways to solve them, or on discussing general principles of programming. Think of the workshop as a hybrid of people writing/showing nice code, and people discussing programming best practices.

Why am I doing this? Because as part of teaching Perl courses for $dayjob, I realized how much I love teaching Perl. I literally spend those two course days teaching Perl in a mild state of euphoria. I teach a lot of other courses, and some of them are great to teach, but none of them are like Perl. I think it's because I feel at home in the language, my brain has somehow "shaped itself" around Perl, and the questions I get can often be answered with a happy "let's find out!" where (a) we do some quick experiment in a terminal window, and (b) I tend to know what we'll get before, so it's easy to explain.

The whipuptitude of Perl really works to one's advantage there. That, combined with knowing my way around most of the perldoc pages and most of the special variables. Oh, and I probably have a slight advantage being fairly deeply involved in Perl 6 stuff, too.

Anyway, I absolutely love teaching Perl. Five and six. So much so that I'd do it for free. So this is me doing it for free.

As I write this, there are 13 participants signed up for the course. This is already twice as big as the typical Perl course I teach in a corporate setting. It's also a wide range of people; from relative newbies to people who could probably give the workshop better than me. I expect people further up on the learning curve to step in and assist with people further down. But I hardly need to say that; this is Perl, and we're a helpful bunch.

Here's more info about the workshop, including when, where, what, and how to sign up.

Perl 6 changes - 2013W16

2013-04-20T07:00:00+00:00

It’s sixteen article in the series. The game goes so fast, and I still haven’t done pagination on my website (lazy as always). The scrollbar is still shorter (1 pixel on my PC) than on Planet Six, but most likely because of longer lines.

NQP was bootstrapped on JVM. That means it can compile itself. It still doesn’t support Perl 6 perfectly, because some code still uses pir::, but it progressed really far. Also, the new version of Rakudo Perl, 2013.04 “Albany” was released. Every improvement mentioned here, is in.

I’ve decided to experiment with commit links this time. This way, you can click (or press, or whatever, depends on your device, to get code of this change).

id='potentially_incompatible_changes'>Potentially incompatible changes>

Perl 6 specification>

Removed `IO::Dir` and `IO::File` classes. fb25fa475854

id='new_features'>New features>

Rakudo Perl>

`IO::Path` is closer to actual spec. 24532aca13

Added `Bool` method to `Capture`. 9f873d89e3

`List.ACCEPTS` now has special `Whatever` handling. f17d339185

`do...for` doesn’t propose `repeat...for` now. 41a92e808e

Documented build memory requirements (1GB of memory). 4109be8905

`require PATH|$variable|Module:file(PATH)` syntaxes are now accepted. c946d0f7f7

Removed reporting variable name for uninitialized variable warnings because it was producing too many false positives. c39b887388

Replaced more `pir::` to `nqp::`. In particular, getting env variables is now done using NQP, not Parrot directly. e9e1b5d962

NQP on JVM bootstrapped, soon will land in NQP master

2013-04-18T16:37:13+00:00

The work to get NQP running and bootstrapped on the JVM has reached an interesting milestone, and I thought I’d take a few moments from working on it to talk about what has taken place since my last post here, as well as taking a look at what will be coming next.

Getting NQP on JVM Bootstrapped

When I last posted here, I had reached the point of having an NQP cross-compiler to the JVM that covered quite a lot of NQP’s language features. Cross-compilation meant using NQP on Parrot to parse NQP source, build an AST, and then turn the AST into a kind of JVM assembly language. This in turn was then transformed into a JVM bytecode file by a small tool written in Java, which could then be executed on the JVM.

Since the last post, the cross-compiler became capable of cross-compiling NQP itself. This meant taking the NQP source files and using the cross-compiler to produce an NQP that would run on the JVM – without depending on Parrot. This also enabled support for eval-like scenarios. I reached this stage last month; the work involved tracking down a range of bugs, implementing some missing features, and doing a little more work to improve NQP’s portability. So, we had an NQP that ran on the JVM. Port done? Well, not quite.

A little vacation later, I dug into the next stage: making NQP on the JVM able to compile (and thus reproduce) itself. While I’d already implemented the deserialization used to persist meta-objects (representing classes, grammars and so forth), for this next step I had to implement the serialization side of things. Thankfully, there are a bunch of tests for this, so this was largely “just” a matter of working through making them pass. Finally, it was time to work through the last few problems, and get NQP on JVM able to build the NQP sources – and therefore able to compile new versions of itself. I decided to do this work as part of moving the JVM support into the main NQP repository.

Since we’ve only had NQP running on one backend (Parrot) up until now, certain aspects of the repository structure were not ideal. Before starting to bring in the JVM support, I first did a little bit of reorganization to segregate the VM specific components from the VM independent ones. Happily, much of NQP’s implementation falls into the latter category. Next came gradually building up the bootstrapping build process, working a file at a time, tracking down any issues that came up. This was a little tedious, especially given a couple of the problems were separate compilation leakages (where things from the running compiler would get confused with the version of the compiler that it was compiling). It was pretty clear that this was the problem from the errors I was seeing, but such problems show up long after things actually go wrong, requiring some careful analysis to hunt down. With those leaks plugged, and a few other relatively small bugs fixed, I had a working NQP on JVM…compiled by NQP on JVM.

The work from there has been to fill out the rest of the build process, adding in the second bootstrap stage and the test targets. The good news: the NQP produced by NQP on JVM passes all the tests that the original cross-compiled version did, so we’ve got no regressions there as a result of the bootstrap.

This work is currently sat in the jvm-support branch of the NQP repository. After the upcoming NQP release, it will be merged.

Supporting invokedynamic

Amongst all of this progress, we’ve also gained infrastructure to support using the invokedynamic instruction. This is a mechanism that enables those implementing non-Java languages on the JVM to teach its JIT about how their dispatch works. Most of the hard work here was done by donaldh++. I’d initially built things using BCEL in order to do code generation. While it served well up to a point, it turns out that ASM has much better support for invokedynamic, as well as being a little faster. So, donaldh got things switched over, and I soon was able to emit invokedynamic.

So far, we are not using it a great deal (just for making QAST::WVal compile to something a bit – or potentially a lot – cheaper), but in the future it will be used for things you’d typically think of as invocations (sub and method calls). I’ll write in more detail about it as things evolve.

What next?

With NQP now ported, the focus will fall on Rakudo itself. Quite a lot of preparations have already been made; for example, many pir:: ops have been replaced with nqp:: ones, multiple dispatch has been ported to NQP from C (fixing some bugs along the way), the way HLL boundaries work has been updated to cope with a fully-6model world (this also let me fix a long-standing introspection bug).

The path through getting Rakudo ported will largely follow the build order. This means starting with the module loader, then the compiler, followed by the MOP and its bootstrapping. After that comes the setting – the place where the built-ins live. There’s around 13,650 lines worth of that, so of course I expect to take it a little at a time. :-) I’ll try to remember to get a progress update here in a couple of weeks time.

The REPL trick

2013-04-18T04:54:42+00:00

A recent discussion on IRC prompted me to share a small but neat trick with you.

If there are things you want to do quite often in the Rakudo REPL (the interactive "Read-Evaluate-Print Loop"), it makes sense to create a shortcut for them. And creating shortcuts for often-used stuff is what programming languages excel at, so you do it right in Perl module:

use v6;
module REPLHelper;

sub p(Mu \x) is export {
    x.^mro.map: *.^name;
}

I have placed mine in $HOME/.perl6/repl.

And then you make sure it's loaded automatically:

$ alias p6repl="perl6 -I$HOME/.perl6/repl/ -MREPLHelper"
$ p6repl
> p Int
Int Cool Any Mu
>

Now you have a neat one-letter function which tells you the parents of an object or a type, in method resolution order. And a way to add more shortcuts when you need them.

Parrot 5.3.0 "W00tstock Parrot" Released! by Bruce Gray

2013-04-17T19:53:19+00:00

We are stardust.
Billion year old carbon.
We are golden.
Caught in the devil's bargain
And we've got to get ourselves back to the garden.
(To some semblance of a garden.)
-- "Woodstock", by Joni Mitchell

On behalf of the Parrot team, I'm proud to announce Parrot 5.3.0, also known
as "W00tstock Parrot". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages, and currently focusing on Perl 6.

Parrot 5.3.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/5.3.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 5.3.0 News:
- Build
+ Files generated by `make cover` are now correctly cleaned by `make`
and ignored by `git`.
- Tests
+ Internal testing of the Configure probe for Fink now works correctly
with the --verbose flag.
+ Tests added for .sort method of ResizableFloatArray and
ResizableIntegerArray. [GH #926], [GH #927]
+ Benchmarks added for .sort methods of various Array objects.
[GH #175]
+ Coverage analysis added for pbc_disassemble.

The SHA256 message digests for the downloadable tarballs are:
79d6f1fe20645b0afbc496cd0d7850a78b8940230e7637c5356d780f5aa1750b parrot-5.3.0.tar.gz
4cff32521c79d8a783ad57d9a13e205ea3c1b1585085e0da80138b58b77d0ed5 parrot-5.3.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 21 May 2013.

Enjoy!

Perl 6 changes - 2013W15

2013-04-14T07:00:00+00:00

I apologize I forgot about Perl 6 changes article yesterday. This ought not to happen, but it did. As for reason why it happened, well, I guess I will tell you anyway - it’s that game.

id='new_features'>New features>

Perl 6 specification>

`Parcel`s are allowed to have 0 elements.

Rakudo Perl>

`.DUMP` method can detect recursion for hashes.

Added `ForeignCode` for non-Perl 6 code objects (like C code).

Removed lots of `pir::` usage for JVM port of Rakudo.

Semicolon after keyword that takes a block, like `PRE ;` now has specific error message.

Added `symlink` and `link`.

`require` can take paths.

Niecza Perl>

Added special cases for `big_pow` when either of values is `-1`, `0`, or `1`.

Perl 6 changes - 2013W14

2013-04-06T07:00:00+00:00

You may have noticed may page went down for some time. This appears to be connected to newest domain change. In order to be more… modern, I’ve moved the repository to glitchmr.github.io, but that made entire page to go down. I apologize for inconvience, but it didn’t affected Planet Six, so I think it’s fine.

As always, I’m going to describe changes in Perl 6.

id='new_features'>New features>

Perl 6 specification>

Renamed `weekday-of-month` to `day-of-week-in-month`.

Rakudo Perl>

Fixed signature of `Pod::Block.gist`, so it works properly with `multi`s.

`for` loops are eager now.

Added `Buf.perl`.

Fixed `Buf.gist`, so it works properly with `multi`s.

Rakudo's Abstract Syntax Tree

2013-04-01T00:22:59+00:00

After or while a compiler parses a program, the compiler usually translates the source code into a tree format called Abstract Syntax Tree, or AST for short.

The optimizer works on this program representation, and then the code generation stage turns it into a format that the platform underneath it can understand. Actually I wanted to write about the optimizer, but noticed that understanding the AST is crucial to understanding the optimizer, so let's talk about the AST first.

The Rakudo Perl 6 Compiler uses an AST format called QAST. QAST nodes derive from the common superclass QAST::Node, which sets up the basic structure of all QAST classes. Each QAST node has a list of child nodes, possibly a hash map for unstructured annotations, an attribute (confusingly) named node for storing the lower-level parse tree (which is used to extract line numbers and context), and a bit of extra infrastructure.

The most important node classes are the following:

QAST::Stmts: A list of statements. Each child of the node is considered a separate statement.
QAST::Op: A single operation that usually maps to a primitive operation of the underlying platform, like adding two integers, or calling a routine.
QAST::IVal, QAST::NVal, QAST::SVal: Those hold integer, float ("numeric") and string constants respectively.
QAST::WVal: Holds a reference to a more complex object (for example a class) which is serialized separately.
QAST::Block: A list of statements that introduces a separate lexical scope.
QAST::Var: A variable
QAST::Want: A node that can evaluate to different child nodes, depending on the context it is compiled it.

To give you a bit of a feel of how those node types interact, I want to give a few examples of Perl 6 examples, and what AST they could produce. (It turns out that Perl 6 is quite a complex language under the hood, and usually produces a more complicated AST than the obvious one; I'll ignore that for now, in order to introduce you to the basics.)

Ops and Constants

The expression 23 + 42 could, in the simplest case, produce this AST:

QAST::Op.new(
    :op('add'),
    QAST::IVal.new(:value(23)),
    QAST::IVal.new(:value(42)),
);

Here an QAST::Op encodes a primitive operation, an addition of two numbers. The :op argument specifies which operation to use. The child nodes are two constants, both of type QAST::IVal, which hold the operands of the low-level operation add.

Now the low-level add operation is not polymorphic, it always adds two floating-point values, and the result is a floating-point value again. Since the arguments are integers and not floating point values, they are automatically converted to float first. That's not the desired semantics for Perl 6; actually the operator + is implemented as a subroutine of name &infix:<+>, so the real generated code is closer to

QAST::Op.new(
    :op('call'),
    :name('&infix:<+>'),    # name of the subroutine to call
    QAST::IVal.new(:value(23)),
    QAST::IVal.new(:value(42)),
);

Variables and Blocks

Using a variable is as simple as writing QAST::Var.new(:name('name-of-the-variable')), but it must be declared first. This is done with QAST::Var.new(:name('name-of-the-variable'), :decl('var'), :scope('lexical')).

But there is a slight caveat: in Perl 6 a variable is always scoped to a block. So while you can't ordinarily mention a variable prior to its declaration, there are indirect ways to achieve that (lookup by name, and eval(), to name just two).

So in Rakudo there is a convention to create QAST::Block nodes with two QAST::Stmts children. The first holds all the declarations, and the second all the actual code. That way all the declaration always come before the rest of the code.

So my $x = 42; say $x compiles to roughly this:

QAST::Block.new(
    QAST::Stmts.new(
        QAST::Var.new(:name('$x'), :decl('var'), :scope('lexical')),
    ),
    QAST::Stmts.new(
        QAST::Op.new(
            :op('p6store'),
            QAST::Var.new(:name('$x')),
            QAST::IVal.new(:value(42)),
        ),
        QAST::Op.new(
            :op('call'),
            :name('&say'),
            QAST::Var.new(:name('$x')),
        ),
    ),
);

Polymorphism and QAST::Want

Perl 6 distinguishes between native types and reference types. Native types are closer to the machine, and their type name is always lower case in Perl 6.

Integer literals are polymorphic in that they can be either a native int or a "boxed" reference type Int.

To model this in the AST, QAST::Want nodes can contain multiple child nodes. The compile-time context decides which of those is acutally used.

So the integer literal 42 actually produces not just a simple QAST::IVal node but rather this:

QAST::Want.new(
    QAST::WVal(Int.new(42)),
    'Ii',
    QAST::Ival(42),
)

(Note that Int.new(42) is just a nice notation to indicate a boxed integer object; it doesn't quite work like this in the code that translate Perl 6 source code into ASTs).

The first child of a QAST::Want node is the one used by default, if no other alternative matches. The comes a list where the elements with odd indexes are format specifications (here Ii for integers) and the elements at even-side indexes are the AST to use in that case.

An interesting format specification is 'v' for void context, which is always chosen when the return value from the current expression isn't used at all. In Perl 6 this is used to eagerly evaluate lazy lists that are used in void context, and for several optimizations.

Perl 6 changes - 2013W13

2013-03-30T07:00:00+00:00

The month is coming to end, and with that, I’m going to make yet another list of changes.

id='new_features_like_anything_is_other_than_that'>New features (like anything is other than that)>

Perl 6 specification>

Added many `IO::Path` methods: `.contents`, `.is-absolute`, `.is-relative`, `.absolute`, and `.relative`.

`.path` is now NOP that returns itself. For its previous behavior, use `.Str`.

Changed `Numeric.conjugate` to `Numeric.conj`.

Rakudo Perl>

`+(23..2300000000)` is optimized.

`--help` reports `-I` and `-M` options.

Polish Perl Workshop status update

2013-03-27T11:56:41+00:00

Exactly two months remain until Polish Perl Workshop this year. In case you didn’t know, I’m shamelessly reposting the status update from our ACT website.

We’ve got the first sponsor offers, and about 15 talk submissions so far. Awesome! However, as we are well aware that most of you will wait to submit talks until there’s less than 2 days until the deadline (or later :)) I’m hereby announcing: The talk submission deadline for Polish Perl Workshop 2013 is 14 of April. That’s about two weeks to finally make up your mind and submit something. Please do! There’ll be only one chance to talk at the First Polish Perl Workshop: there’ll be no other :)

Of course, if you’d prefer to organize more of a hands-on event, we have the entire sunday dedicated to hackathon, BoFs, classes etc. If you want to submit something more lenghty, that’s the right way to do it.

In other interesting news, we started looking at conference t-shirts (yes, they’re free for all attendees). We have something on our mind that never happened before on a Perl conference (or at least I’ve never seen or heard of something like this). We’ll keep you informed.

Parrot 5.2.0 "Stuffed Parrot" Released! by Bruce Gray

2013-03-24T13:58:55+00:00

I am not dead yet I can dance and I can sing
I am not dead yet I can do the Highland Fling
I am not dead yet No need to go to bed
No need to call the doctor Cause I'm not yet dead
-- Spamalot

On behalf of the Parrot team, I'm proud to announce Parrot 5.2.0, also known
as "Stuffed Parrot". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.

Parrot 5.2.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/5.2.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 5.2.0 News:
- Core
+ IO now only syncs buffers for the IO types where syncing makes sense.
= PIO_VF_SYNC_IO flag added
- Build
+ installable_pdump now has the correct rpath (blib corrected to lib).
- Libraries
+ Tcl/Glob.pir has been removed. (PGE/Glob.pir remains intact)
- Ecosystem
+ All Parrot tarballs are now symlinked to the 'all' directory,
regardless of their true homes ('devel' or 'stable'), to better
allow for automated downloads.
ftp://ftp.parrot.org/pub/parrot/releases/all/

The SHA256 message digests for the downloadable tarballs are:
1245d11f2b2ea44e6465aff6da5a533324d69b6eb3ddf7d84e81385ea95150ad parrot-5.2.0.tar.gz
0c538d780f9c70c510e142a8a663c30474125c9fcf9fe25d2129e68fc7baec8d parrot-5.2.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 16 Apr 2013.

Enjoy!

--
Bruce Gray (Util of PerlMonks)

Perl 6 changes - 2013W12

2013-03-23T07:00:00+00:00

I really should optimize my blog layout, so empty intro wouldn’t be ugly. But I guess I’m too lazy to do it right now, so I have to explain stuff. Every week, I list the list of changes in Perl 6 specifications, and its implementations. That’s just everything, boring, I know. Still, it doesn’t look that bad on Planet Six, so I guess I will continue to make these articles :-).

This week, the Rakudo 2013.03 was released. Every change mentioned here was implemented in it.

id='new_features'>New features>

Perl 6 specification>

Added `X::Range::InvalidArg`.

Rakudo Perl>

Removed lots of `pir::` usage.

Multi dispatch is now implemented in NQP, not C.

Released Rakudo 2013.03.

Reading from a file in Perl 6

2013-03-19T20:52:02+00:00

It can be interesting to do operations on existing data structures in Perl 6, but without Input and Output, that would have limited usability in the real world.

Therefore reading from files is a critical operation.

Perl 6 changes - 2013W11

2013-03-16T07:00:00+00:00

I would put nothing here, but the layout of blog would make this look ugly. So, well, like usually I put something here, even if it is meaningless. Just look below to see list of changes.

id='new_features'>New features>

Rakudo Perl>

Closing brace, followed by newline and infix operator was parsed incorrectly. For example, following code.
```
my %a;
{ }
%a = 1;
```
Was parsed like:
```
my %a;
{ } % a() = 1;
```
Instead of correct.
```
my %a;
{ };
%a = 1;
```
Done more pir:: to nqp:: replacements in order to help JVM port of Rakudo Perl.

Case insensitive matches with strings now work.

my $nonsense = 'asdf';
say "OK" if "ADSF" ~~ m:i/$nonsense/;

Typing else if and elif after if block shows “Please use ‘elsif’” error message.
&?ROUTINE variable is available inside regex foo { ... }.
The named params are optimized in compile time when possible.
X::Method::NotFound is raised even when dealing with "" (empty) method names.
Fixed proto of infix:<ne> in order to require Mu objects.
Stash.at_key method now works.
Postcircumfix calls can be cached now.
LEAVE is now fired on next/last in MapIter.
Exception is thrown when Range endpoint is used to create a Range.

Niecza Perl>

`Empty.gist` returns nothing.

Getting started with Rakudo * (2013.01) on Windows

2013-03-15T20:52:02+00:00

After a break of several months, I rebuild Rakudo on the server, a few days ago, and regenerated the Perl 6 Maven pages. This promptly broke the site. The fix was easy, and as expected the bug was mine.

Today I decided to try Rakudo on my Windows machine as well, but I went for the easy route and installed the pre-built msi file.

Perl 6 changes - 2013W10

2013-03-09T08:00:00+00:00

During making this update, I had lots of problems with Internet. I have managed to load commits history for every repository I check, but I haven’t checked commits. Will update this after Internet access will be more stable.

id='new_features'>New features>

Perl 6 specification>

`Rat.perl` returns decimal number when denominator is normalizable to a power of 10.

Added FALLBACK methods.

Included Failure in smartmatch passthroughs.

Loosened relationship between `Nil` and `()`.

Added `List.permutations`.

`rat64` type has unsigned denominator.

STD.pm6>

`Nil` isn’t suggested for empty list in error messages anymore.

Rakudo Perl>

Fixed expanding of variables that don’t exist when converting string to RegExp. For example, `"a" ~~ / <$blah> /` is fixed.

`Rat.perl` returns decimal number when denominator is normalizable to a power of 10.

Removed lots of `pir::` usage in order to help porting Rakudo to JVM.

`Rat.perl` provides angle brackets around fractional forms.

Added missing shellwords postcircumfixes (`<< >>` and `« »`).

Generalized “augment” check, and made error message more generic.

User defined circumfixes are parsed just like STD.pm6 does.

Added colorcircumfixes.

Rakudo now warns about integers starting with `0`.

Mistakes like `if(1 < 2) { ... }` (where mistake is that `if` is called like a function) give good errors.

`Nil` isn’t suggested for empty list in error messages anymore.

Niecza Perl>

`Rat.perl` returns decimal number when denominator is normalizable to a power of 10.

`CommonEnum.gist` returns `(Type)` instead of `Type()` for undefined.

Restored `infix:<cmp>` sorting for different types.

Failed matches return `Nil`.

Philosophical Issues with Rakudo’s .parse

2013-03-06T12:59:01+00:00

For a little bit now I’ve had the ABC module working on both Niecza and Rakudo. I got a new Linux box yesterday; this morning I was playing around with my ABC tools there, trying to see if I could get them to work. I installed them with panda, so I was using ABC with Niecza by default on my MBP and with Rakudo by default on my new machine. I tried it with a file of my own compositions. Apparently I hadn’t tried this in a while, because it had several issues. And the two systems handled them completely differently.

On Niecza, I got this message:

Unhandled exception: Did not match ABC grammar: last tune understood:
 A Hat for the Whales
d|:e~A3 BGEF|G2BG DG (3ABd|eBdB ~A3G|Bdef gfed|
e~A3 BGEF|G2BG DG (3ABd|eBdB AAGA|[1Beed e3d:|

On Rakudo, there were no error messages, but “A Hat for the Whales” was the last tune processed, even though there were a dozen more tunes in the ABC file after it.

Without thinking about it too much I corrected this batch of errors. Now on Niecza the entire file parsed, but I got the error
Unhandled exception: Illegal key signature B minor
Under Rakudo, the same ABC file just appeared to work.

What’s going on here? Well, key signature processing looks something like this (simplified):

    token key-def { <basenote> <chord_accidental>? <mode>? }
    token mode { <minor> | <major> }
    token minor { "m" ["in" ["o" ["r"]?]?]? }
    token major { "maj" ["o" ["r"]?]? }

No space allowed between basenote and mode. So when you invoke parse with that rule on “B minor”, in Niecza it fails, because the entire string was not successfully parsed. In Rakudo it succeeds, because “B” is a valid key signature, and it doesn’t care that the entire string was not parsed. In practice, this means that three of the tunes in the collection got the wrong key signature: B major instead of the correct B minor.

To my mind, Niecza’s behavior is more intuitive. If you’re trying to parse a file, parsing half of it and then silently ignoring the rest is wildly unhelpful behavior. In every use I’ve had for parse so far, I’ve wanted it to parse everything in the string I sent to the function.

But hey, I understand that some people might have different needs. Can we at least add a flag that will give you a parsing failure if the entire string is not parsed? :all, perhaps?

Perl 6 changes - 2013W09

2013-03-02T08:00:00+00:00

I would put a description, but who seriously cares?

id='new_features'>New features>

Perl 6 specification>

Inside of `try` is considered eager context, unless `try` is called in sink context, when it’s in sink context.

Rakudo Perl>

Type placeholders (like `Int`, which is null, except Int typed) now have their `.gist` method show their name within parens to suggest being undefined. For example, `(Int)`, `(Mu)`.

Changed “Cannot assign to a non-container” error message to “Cannot modify an immutable value” that hopefully is more clear.

Release announcements now use Markdown.

`Cool.substr` doesn’t coerce its first argument to `Int` anymore.

I made a pointless webpage

2013-03-01T08:00:00+00:00

So, well, I have a VPS, and decided to put a webpage on it for lulz. You can see it at mango.uk.to, and the source (sauce) is available on my GitHub profile.

Just mentioning, not that this site is important. I just wanted to run a web server just to run a web server.

By the way, this blog is also open sauce, you can see its source code on GitHub. It’s full of hacks, but that’s Jekyll for you.

Rakudo Star 2013.02 released

2013-02-24T19:21:21+00:00

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the February 2013 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the February 2013 release is available from http://rakudo.org/downloads/star/. A Windows .MSI version of Rakudo star will usually appear in the downloads area shortly after the tarball release.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2013.02.1 of the Rakudo Perl 6 compiler, version 4.10.0 of the Parrot Virtual Machine, and various modules, documentation, and other resources collected from the Perl 6 community.

Some of the new features added to this release include:

“Did you mean …” suggestions for symbol-not-found errors
Compile-time optimization of some cases of junctions in boolean context
IO::Socket.get now works again with non-ASCII characters
constant folding for routines marked as ‘is pure’
natively typed variables and better error reporting in the REPL
speed up eqv-comparison of Bufs
warnings for useless use of (some) literals, variables and constant expressions in sink context

This release also contains a range of bug fixes, improvements to error reporting
and better failure modes.

The following features have been deprecated or modified from previous
releases due to changes in the Perl 6 specification, and are being removed
or changed as follows:

.gist on a type object will return ‘(Typename)’ instead of ‘Typename()’. If you want to get the class name alone, continue to use $obj.^name
postcircumfix:<[ ]> and postcircumfix:<{ }> will become multi-subs rather than multi-methods. Both at_pos and at_key will remain methods.
Unary hyper ops currently descend into nested arrays and hashes. This will change to make them equivalent to a one-level map.
The Str.ucfirst builtin is deprecated; it will be replaced by Str.tc.
Leading whitespace in rules and under :sigspace will no longer be converted to <.ws> . For existing regexes that expect this conversion, add a <?> in front of leading whitespace to make it meta again.
The ?-quantifier on captures in regexes currently binds the capture slot to a List containing either zero or one Match objects; i.e., it is equivalent to “** 0..1″. In the future, the ?-quantifier will bind the slot directly to a captured Match or to Nil. Existing code can manage the transition by changing existing ?-quantifiers to use “** 0..1″, which will continue to return a List of matches.

There are some key features of Perl 6 that Rakudo Star does not
yet handle appropriately, although they will appear in upcoming
releases. Some of the not-quite-there features include:

advanced macros
threads and concurrency
Unicode strings at levels other than codepoints
interactive readline that understands Unicode
non-blocking I/O
much of Synopsis 9

There is an online resource at http://perl6.org/compilers/features
that lists the known implemented and missing features of Rakudo
and other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the
programmer that a given feature isn’t implemented, but there are
many that we’ve missed. Bug reports about missing and broken
features are welcomed at
href="mailto:rakudobug@perl.org">rakudobug@perl.org.

See perl6.org for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. A draft of a Perl 6 book is available as UsingPerl6-draft.pdf> in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.

Perl 6 changes - 2013W08

2013-02-23T08:00:00+00:00

Now, I’m back to the your regularly scheduled Perl 6 changes article that isn’t. Well, it is, but it’s not. Or something. Rakudo 2013.02.1 was released (well, 2013.02 was released too). The version bump was caused by the fact that NQP 2013.02 had broken native call support.

id='incompatible_changes'>Incompatible changes>

Perl 6 specification>

Failed matches result `Nil`, not `(Match)`.

id='new_features'>New features>

Perl 6 specification>

Removed `Set.powerset`. It was removed after 10 hours of existing. Consider it a printing error in my previous article, or something. Not really an incompatible change, as it practically never existed.

Type placeholders (like `Int`, which is null, except Int typed) now have their `.gist` method show their name within parens to suggest being undefined. For example, `(Int)`, `(Mu)`.

STD.pm6>

Added a warning for smartmatch of `(Match)` or `(Nil)`.

Rakudo Perl>

The sink context now can be detected. Side-effect-free operations, constants and variables warn about it.

Compiler warns about apparent subscripts in regexes being treated as regexes (for example, `@a[0]`, which is treated as any of `@a` elements followed by `0`).

It’s possible to interpolate arrays in regular expressions.

Added `int64` native type.

`/./` and `rx/./` now matches against `$_` in sink context.

`$feature is not yet implemented. Sorry.` error message doesn’t have `is` anymore.

Coercive type declarations make not yet implemented error.

Niecza Perl>

Type placeholders (like `Int`, which is null, except Int typed) now have their `.gist` method show their name within parens to suggest being undefined. For example, `(Int)`, `(Mu)`.

t1: Tell knights from knaves

2013-02-22T22:39:50+00:00

<grondilu> masak: are you sure problem #1 is computationnaly doable?  It's not NP = P or something, is it?

* grondilu thinks of it and realizes the number of possibilities is not so large

<grondilu> .oO( 2**number_of_islanders  possibilities anyway )

And so, the time-honored tradition of reviewing p6cc solutions begins. Again.

For our first task, we have the very human conundrum of figuring out who is telling the truth and who is lying. Fortunately, on the island of Smul, that's a much more tractable problem than in the real world.

The problem description follows.

## Tell knights from knaves based on what they say

On the mythical island of Smul, people suffer from a rare genetic disorder that
make them either tell the truth all the time, or lie all the time. These are
the only two types of people on the island, known as knights and knaves,
respectively.

Write a program that takes as input a number of utterances by islanders, and
outputs for each person whether that person is a knight or a knave. If there is
no possible assignment that works, the program should report that no solution
exists. In the case of multiple solutions, the program should report every
possible solution.

The islanders can make four different classes of utterances:

    X is a knight.
    X is a knave.
    X and Y are of the same type.
    X and Y are of different types.

(Here, X and Y are used as metavariables, of course, and can in fact be any
name of an islander.)

Islanders can refer to each other. The same islander can make several
utterances. If an islander mentions another islander that doesn't say anything,
your program should consider the entire input to be erroneous.

Here are a few examples:

    A: A is a knight.

Both a knight and a knave would assert the same thing. So this input has two
solutions.

    B: B is a knave.

Neither a knight or a knave would ever say this about themselves. So this input
allows no solution.

    C: C and D are of the same type.
    D: D and C are of different types.

Here, the two islanders are contradicting each other, so one of them must be a
knight and the other a knave. But this is exactly what D is saying, so D is the
knight. One solution.

Let's get one obvious thing out of the way: we can solve this with a brute-force method, supposing for each islander in a situation first that he is a knight and then that he is a knave. Each islander thus bifurcates the universe in two possible universes. Two islanders will yield four possible universes to investigate. Three islanders will yield eight universes. Ten islanders will yield 1024 universes. The universes grow exponentially with the islanders. The brute-force solution will always work, but at an exponential slowdown.

Many people choose the brute-force solution. I don't blame them; it's there for the picking. Some people get fancy with logical propositions or "affiliations" between islanders, but they all fall down the exponential pit at one point or another. The brute-forcers are all over the place in terms of style and brevity, and it's quite fun to watch.

Then there's this one contestant that gets the nice solution. It always thrills me when that happens.

Here, let me lay it out for you. Let 0 mean False/knave and 1 mean True/knight. Then let's translate the four possible utterances to equations:

"A: X is a knight." → A = X
"A: X is a knave." → A = 1 xor X
"A: X and Y are of the same type." → A = 1 xor X xor Y
"A: X and Y are of different types." → A = X xor Y

By some amazing coincidence, all the utterances can be put into more or less the same mold, and the only operator used is xor. It's actually fun to translate these formulas back, and get alternative formulations of things, sometimes giving a different perspective on things:

A = X → "A is as truthful as X"
A = 1 xor X → "A or X is truthful, but not both"

From that first formulation we see why, when A is asserting that A is a knight, that utterance devolves into a tautology A = A. It's as if an islander cannot assert his own knighthood solely on his own authority.

From the second formulation we see why, when A is asserting that A is a knave, such an utterance devolves into a contradiction A = 1 xor A. There just isn't any such number A. (We'd have to look in the murky domain of fuzzy logic, but that's outside the realm of the island of Smul.)

Anyway, translating all the utterances to this form is a big win. Now we can make a linear equation system of all the utterances, and solve the linear equation system. Do we have fast algorithms for that? Yes, we do! Gaussian elimination has an arithmetic complexity of O(n³). That's quite an improvement on exponential brute force!

We must take care to do all the additions and subtractions as xors, though. This is because our underlying algebra is truth values, outside of which we may not stray. So we're really doing Gaussian elimination on the quotient ring Z/2Z.

In the literature, we find this problem as XOR-satisfiability. Wikipedia dignifies it with two sentences.

One contestant did it this way. Yay him. You should check out his solution.

Here are the solutions, and my reviews of same. Enjoy.

Next up: rectangle haikus, possibly the most fun I've ever had with a p6cc task.

friendly interactive shell

2013-02-22T08:00:00+00:00

I’m not sure if anybody noticed, but lately, nothing was happening on my blog. So, I’m going to write an article about friendly interactive shell (fish), the best shell I’ve used ever (it’s my personal opinion, in case you haven’t noticed, see my footer blog). Actually, I think I’ve moved to it half of the year ago or something.

These days, the two popular shells exist - bash and Z shell. I’m aware of that more shells exist - just nobody uses them these days By the way, if you try to find logical fallacy, I agree, cmd.exe and Windows PowerShell are often used, but those aren’t UNIX shells. Oh, and programming language REPLs are often used too (like irb), but they aren’t UNIX shells too.

id='bash'>Bash>

The bash shell is standard shell, included in many Linux distributions. It will make you bash your head against the brick wall (pun intended).

One of examples of bash code I could have found is this. And trust me, it’s not the worst thing (thanks, Wikibooks).

# print the powers of two, from 1 to 512:
for ((i = 1; i < 1000; i *= 2)); do
  echo $i
done

Do you see double parens here? This is really strange math syntax that only supports integers. Why double parenthesis? Well, the answer is compatibility with Bourne shell (really, really old shell, usually called sh (but sometimes sh is actually bash, to make you bash)).

Also, you could have noticed keywords do and done. Bash could easily not require do keyword before for loop body. Yet, it’s required. The other strange quirks is that you aren’t allowed to put semicolon after do keyword. Why? I’ve no idea, really. The done word, while I think it’s needed, differs between blocks. For example, if blocks end with fi.

Bourne again shell has lots of strange syntax you really wouldn’t ever use. For example, bash has $'' syntax. It unescapes escape sequences inside. The strange part is '' in bash doesn’t do any processing at all, yet $'' does. Bash also has $"" syntax, but it doesn’t do escape processing - instead it uses gettext to translate string inside.

# Prints 'Hello world.', with new line in middle.
echo Hello$'\n'world.
# Prints 'Hello\nworld.', literally.
echo Hello$"\n"world.

Consistency in bash is even worse than in languages such as PHP. There are many ways to do something, but none of them are obvious. And I mean that, none. Expanding bash is adding new syntax that totally doesn’t make sense, but was a syntax error in previous versions. $ sign is used for many features that have nothing with variables.

Also, bash integrates many programs in it that really shouldn’t be inside. For example, echo.

[glitchmr@pineapple ~]$ echo --help
--help
[glitchmr@pineapple ~]$ which echo
/bin/echo
[glitchmr@pineapple ~]$ /bin/echo --help
Usage: /bin/echo [SHORT-OPTION]... [STRING]...
  or:  /bin/echo LONG-OPTION
Echo the STRING(s) to standard output.

...

NOTE: your shell may have its own version of echo, which usually supersedes
the version described here.  Please refer to your shell's documentation
for details about the options it supports.

Report echo bugs to bug-coreutils@gnu.org
GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
Report echo translation bugs to <http://translationproject.org/team/>
For complete documentation, run: info coreutils 'echo invocation'

The /bin/echo on my system is GNU echo. The bash on my system is GNU bash. They were both made by GNU. Yet, they act differently. Is there a better recipe for failure than program that actually isn’t called because your shell is “clever”.

It’s not just echo. Many commands are implemented in shell, when they should be implemented outside shell. For example, pwd. Why exactly? To confuse you when dealing with recursive structures.

[glitchmr@pineapple ~]$ mkdir recursion
[glitchmr@pineapple ~]$ cd recursion/
[glitchmr@pineapple recursion]$ ln -s . recursion
[glitchmr@pineapple recursion]$ cd recursion
[glitchmr@pineapple recursion]$ pwd
/home/glitchmr/recursion/recursion
[glitchmr@pineapple recursion]$ which pwd
/bin/pwd
[glitchmr@pineapple recursion]$ /bin/pwd
/home/glitchmr/recursion
[glitchmr@pineapple recursion]$

The shell outright lies about your location. You’re in /home/glitchmr/recursion. Why shell does that? Well, it does that to not confuse you when typing cd ... And indeed, if you type cd recursion/recursion, and cd .. after that, you will end in recursion directory, not the directory where you created the first recursion directory.

Symbolic links in UNIX are simply redirects. If you made a symbolic link to /usr/share/hell, you shouldn’t expect that cd .. made after following link follow return you anywhere that isn’t /usr/share. Yet, implementation of symbolic links in bash lies.

The lie is obvious after using anything that isn’t bash builtin. For example, ls .. will always show contents of /usr/share, even if you made symbolic link in /tmp/heaven. Yet, cd .. will return to /tmp/heaven.

By the way, in case you ask, cd is also a builtin. But that’s acceptable, because external command cannot change environment of caller (such at PATH). So it has to be a builtin. Yet, somehow, some people are still confused.

id='z_shell'>Z shell>

Z shell is a shell that is way better than bash. Well, if you configure it properly. It’s a shell that appears to have some compatibility with Bash scripters (for example $'' syntax works, but $"" doesn’t (or rather, it returns literal ’$’ and contents of "")).

Z shell has lots of features. In fact, the author is “not aware of a major interactive feature in any other freely-available shell which zsh does not also have (except smallness)”. Something that would be sane default actually isn’t default. In fact, with default settings it mostly works like bash.

For example, completion. By default, Z shell only completes file names and command names. If you will type sudo apt-get install, press space and then press Tab, it will give you file names. Useful? I don’t think so.

Of course, the option to enable better Tab completion is possible to turn on. It’s just not default. Why the shell would have default that don’t make sense. Simiarly, by default Tab completion cannot expand globs. Yet, it’s possible to enable.

When you start Z shell for first time, you see the configuration wizard (called zsh-newuser-install). The problem is, it’s the most non-user-friendly thing I’ve seen. It asks about minor details you don’t care about, without proposing any useful default. For example, the maximum number of errors you can make for the command to be still proposed in error message. I just have pressed Enter, and the response is “Please enter a number”. Do you think I care about this?

Actually, as far I know, nobody actually uses this wizard. In most cases, zsh is configured using so called oh-my-zsh. What it is actually? Well, it’s a config script with sane things configured by default. Yet, it has got 8,648 stars on GitHub and 3,326 forks as I’m writing it.

Yes, it’s just ZSH config file that has things that should be default, but they aren’t. Oh, and plugins array that contains list of plugins that mostly make aliases and completion adders that aren’t in main zsh (like CoffeeScript). For example git plugin adds aliases like gc for git commit -v.

I don’t know about you, but I don’t like programs which take 4 hours to configure. Z shell is one of these. oh-my-zsh makes the job easier, but I still don’t think it’s worth wasting the time. Perhaps if fish wouldn’t exist, it would be worth it.

id='friendly_interactive_shell'>friendly interactive shell>

Z shell, aside of totally insane configuration isn’t a bad shell (if I would want to talk about bad shells, it would be csh or cmd.exe). friendly interactive shell is as good as Z shell (with good Z shell configuration), except without having to configure it.

In fact, most of things cannot be configured. You cannot configure nonsense like maximal history length (by design). You cannot configure interpretation of numbers starting with 0 as octal (you can in Z shell - I have no idea why Z shell even allows you to type “echo 010” to get “8” when configured to do so - but it does). You cannot disable file globs.

Remember .bashrc/.zshrc configuration file? You won’t need it. The colors and themes can be easily configured using a web service (but if you prefer, you can use set command directly).

For example, if you want to configure your EDITOR to be vim, you can run the following command.

set -Ux EDITOR vim

The -U modifier means that this value is stored between session. The -x means that this variable is exported to programs ran by shell. After you have done that, your variable magically appears in every fish session.

Adding new functions is also easy. It’s enough to type funced function-name. After

Fish also offers features I haven’t seen in other shells, like syntax highlighting. Ommited quote? Syntax highlighting will help you.

In fish, everything is tab-completeable. Even if the program doesn’t have specific tab completion definitions, it’s still tab-completeable, thanks to generating tab completions from man pages. For example, gedit doesn’t have specific tab completions, yet fish managed to create them.

glitchmr@pineapple ~> gedit -
--background                                         (Run gedit in the background.)
--encoding        (Set the character encoding to be used for openi… [See Man Page])
--geometry                     (Set the X geometry window size (WIDTHxHEIGHT+X+Y).)
--help                                           (Prints the command line options.)
--list-encodings  (Display list of possible values for the encodin… [See Man Page])
--new-document            (Create a new document in an existing instance of gedit.)
--new-window       (Create a new toplevel window in an existing instance of gedit.)
--standalone                                        (Run gedit in standalone mode.)
--version                                    (Output version information and exit.)
--wait                                    (Open files and block the gedit process.)
-b                                                   (Run gedit in the background.)
-g                             (Set the X geometry window size (WIDTHxHEIGHT+X+Y).)
-s                                                  (Run gedit in standalone mode.)
-w                                        (Open files and block the gedit process.)

Of course, just like zsh (with correct configuration), it can expand more stuff. For example, kill command expands process IDs.

glitchmr@pineapple ~> kill 
1            (systemd)  127           (systemd-udevd)  479                (gconfd-2)
2           (kthreadd)  132         (systemd-journal)  508                   (gvfsd)
3        (ksoftirqd/0)  229               (hd-audio0)  512              (gvfsd-fuse)
5       (kworker/0:0H)  230               (kpsmoused)  527         (cinnamon [tty1])
7       (kworker/u:0H)  299              (irq/46-mei)  528                  (colord)
8        (migration/0)  307                (cfg80211)  534  (polkit-gnome-au [tty1])
9        (rcu_preempt)  308                    (hci0)  536          (dropbox [tty1])
10            (rcu_bh)  309            (kworker/u:1H)  539        (nm-applet [tty1])
11         (rcu_sched)  310               (scsi_eh_6)  540  (gnome-screensav [tty1])
12        (watchdog/0)  311         (rts5139-control)  558      (gsd-printer [tty1])
13        (watchdog/1)  312         (rts5139-polling)  566         (gvfs-udisks2-vo)
14       (ksoftirqd/1)  314                (ttm_swap)  568                 (udisksd)
15       (migration/1)  325          (NetworkManager)  598             (gvfsd-trash)
17      (kworker/1:0H)  329             (dbus-daemon)  728          (firefox [tty1])
18            (cpuset)  331          (systemd-logind)  744          (hexchat [tty1])
19           (khelper)  332                   (login)  746      (pxgsettings [tty1])
20         (kdevtmpfs)  335                 (polkitd)  797           (dconf-service)
21             (netns)  341                  (mysqld)  798   (gnome-terminal [tty1])
22       (bdi-default)  344          (wpa_supplicant)  805  (gnome-pty-helpe [tty1])
23           (kblockd)  348                   (nginx)  806            (fish [pts/0])
26        (khungtaskd)  349                   (nginx)  1374            (kworker/1:2)
27           (kswapd0)  351                 (php-fpm)  5790           (fish [pts/1])
28              (ksmd)  352                 (php-fpm)  10502        (gvfsd-metadata)
29        (khugepaged)  353                 (php-fpm)  13551          (fish [pts/2])
30     (fsnotify_mark)  373                (dhclient)  13635           (ssh [pts/2])
31            (crypto)  385             (fish [tty1])  13866           (kworker/1:0)
35          (kthrotld)  387                   (fishd)  18892            (gvfsd-http)
38           (deferwq)  412           (startx [tty1])  20044          (fish [pts/1])
62             (khubd)  429            (xinit [tty1])  20046        (python [pts/1])
63           (ata_sff)  430                       (X)  20405           (vim [pts/0])
64         (scsi_eh_0)  434    (gnome-session [tty1])  21370           (kworker/0:1)
65         (scsi_eh_1)  437      (dbus-launch [tty1])  21373           (kworker/u:2)
66         (scsi_eh_2)  438             (dbus-daemon)  22399          (fish [pts/3])
67         (scsi_eh_3)  440         (at-spi-bus-laun)  22472           (vim [pts/3])
68         (scsi_eh_4)  444             (dbus-daemon)  22521           (kworker/u:0)
69         (scsi_eh_5)  447         (at-spi2-registr)  22863           (kworker/0:0)
93      (kworker/0:1H)  455  (gnome-settings- [tty1])  22945           (kworker/u:1)
95      (kworker/1:1H)  464              (pulseaudio)  23238            (ps [pts/4])
103      (jbd2/sda6-8)  465            (rtkit-daemon)  23239          (grep [pts/4])
104  (ext4-dio-unwrit)  468                 (upowerd)  23240          (tail [pts/4])
110        (flush-8:0)  476            (gconf-helper)

Of course, fish has some disadvantages. One of disadvantages is the syntax. It’s not POSIX compatible by design. For example, the history substition doesn’t exist. Instead, you’re supposed (if you wanted, let’s say, sudo) to press Up, Home, and type ‘sudo’. Command substitution uses (), not `` or $(). Still, if you will accidentally use $() bashism, the shell will inform you with an error.

Also, the newest stable is really old and slow. Instead I would use version from git, which is usually more stable than stable version. In Arch Linux, it’s fish-shell-git from AUR, on other operating systems (like CentOS I use on my VPS), you have to compile it yourself. It isn’t so difficult. The GitHub repository is fish-shell/fish-shell.

As for why I’m mentioning fish shell, well, I have lately contributed a small patch to hint -C suboptions in perl - like -CSDL. Nothing special, but hey, why not make a blog post, considering I write them rarely.

Parrot 5.1.0 "Zombie Parrot" Released! by Bruce Gray

2013-02-20T18:36:17+00:00

Flat on the bunk again, he ran for his life. The Parrot stalked him
through the grey hours of morning, smoothing its fractal feathers,
shuffling itself slowly into clarity as though at the end of a
flashy film-dissolve, until at last his mind's eye had to acknowledge
a shape,
a shape,
a wink
-- From BLIT, a short story by David Langford
http://www.infinityplus.co.uk/stories/blit.htm

On behalf of the Parrot team, I'm proud to announce Parrot 5.1.0, also known
as "Zombie Parrot". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.

Parrot 5.1.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/supported/5.1.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 5.1.0 News:
- Core
+ The .sort() method was added to the FixedFloatArray PMC
+ Improved detection of system memory for machines with >2GB
+ Improved pbc_to_exe support for spacey paths
+ Fixed Parrot_io_readall_s allocating too much string space
- Build
+ Fixed generated MANIFEST files to omit $destdir
- Documentation
- Tests
+ .readall now checks that prior reads are respected.
- Community
+ Weekly IRC meetings have resumed. #parrotsketch Tuesdays at 1930 UTC

The SHA256 message digests for the downloadable tarballs are:
af26c2fcc806505ec516ebb013bdd37b218633f5fe63faaa6b843ffe55e0135e parrot-5.1.0.tar.bz2
2483963c1bec665be772cb40a71fd3d9d2621feca547932475017c81a2f7e49b parrot-5.1.0.tar.gz

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 19 Mar 2013.

Enjoy!

--
Bruce Gray (Util of PerlMonks)

NQP on JVM gets Grammars, Multiple Dispatch

2013-02-16T23:16:22+00:00

Having just reached an interesting milestone, I thought I’d blog a quick progress update on my work to port NQP to the JVM, in preparation for also doing a port of Rakudo Perl 6.

The big news is that the grammar and regex engine is pretty much ported. This was a sizable and interesting task, and while a few loose ends need to be wrapped up I think it’s fair to say that the hard bits are done. The port includes support for the basics (literals, quantifiers, character classes, positional and named captures, calls to other rules, and so forth) as well as the more advanced features (transitive Longest Token Matching for both alternations and protoregexes, and embedded code blocks and assertions). Missing are anchors, conjunctions and a small number of built-in rules; none of these are particularly demanding or require primitives that don’t already exist, however. It’s also worth pointing out that the NQP code to calculate the NFAs used in Longest Token Matching also runs just fine atop of the JVM.

Another interesting feature that I ported a little while ago is multiple dispatch. This was some effort to port, since the original implementation had been done in C. While it’s sensible to have a close-to-the-VM dispatch cache, there’s little reason for the one-off candidate sorting work (not a hot path) to be done in C, so I ported the code for this to NQP. This meant that on the JVM side, I just needed to implement a few extra primitives, and could then run the exact same candidate sorting code.

I think it’s worth noting again that I’m really doing two things in parallel here: hunting down places where NQP couples too tightly to Parrot and loosening the coupling, and also doing the porting to the JVM. The first half of this work is relevant to all future ports. In many cases, I’m also finding that the changes give us architectural improvements or just cleaner, more maintainable code. I wanted to point this out especially because I’m seeing various comments popping up suggesting that Rakudo (or even Perl 6) is on a one-way road to the JVM, forsaking all other platforms. That’s not the case. The JVM has both strengths (mature, a solid threading story, widely deployed, the only allowed deployment platform in some development shops, increasing attention to supporting non-Java languages through things like invokedynamic) as well as weaknesses (slow startup time, lack of native co-routine support, and the fact that it was originally aimed at static languages). Rakudo most certainly should run on JVM – and it most certainly should run on other platforms too. And, as I wrote in my previous post, we’ve designed things so that we are able to do so. Perl has always been a language where There’s More Than One Way To Do It. Perl also has a history of running on a very wide range of platforms. Perl 6 should continue down this track – but the new reality is that a bunch of the interesting platforms are virtual, not hardware/OS ones.

By now, the JVM porting work is fast approaching a turning point. Up until now, it’s been about getting a cross-compiler and runtime support in place and working our way through the NQP test suite. This phase is largely over. The next phase is about getting NQP itself cross-compiled – that is, cross-compiling the compiler, so that we have an NQP compiler that runs on the JVM, supporting eval and able to run independently.

Perl 6 is now half as old as Perl

2013-02-16T18:00:54+00:00

Today Perl 6 is as old as Perl was when Perl 6 was announced.

$ perl6 -e 'say Date.new(2000, 7, 18) + (Date.new(2000, 7, 18) - Date.new(1987, 12, 18))'
2013-02-16

The 1987 date is the release of Perl 1. The 2000 date is the throwing of the mugs, which I consider to be Perl 6's birthday.

It's a bit interesting to compare Perl 6 at this point with Perl back then. They are two fairly different projects, even though the people are overlapping to a great extent.

Bottom-up vs top-down

In any programming project, you can start from the small pieces and build upwards to the overreaching goals and ideas, or you can start from the ideas and build downwards to the nitty-gritty stuff.

Perl is a bottom-up project: the Perl 1 release looks puny today. It didn't do much. But it did run. It did solve people's problems. And there's an unbroken chain of commits leading from Perl 1 to today's Perl 5.

Perl 6 is definitely a top-down project. Larry mulled over those RFC's and wrote the Apocalypses. These eventually resulted in the Synopses, which guide implementation work. Implementations reach up towards the spec, and are in some sense always built bottom-up... but on the most zoomed-out scale, Perl 6 is built from the top down.

This difference was very much deliberate. For the Perl 6 project, it was felt that a specification (and a corresponding test suite) was needed. The Pugs project is currently very dormant and not actively developed, but it did result in both the Synopses and the spectest suite. Both are essential artifacts for any Perl 6 implementer.

Second system syndrome

"It's important to remember that when you start from scratch there is absolutely no reason to believe that you are going to do a better job than you did the first time." — Joel Spolsky, Things You Should Never Do, Part I

Look, here's the thing: everyone knows that The Big Rewrite isn't a good idea. That's like, Software Project Management 101. Even the people going into this project knew that. The jokes about second systems have been with us from the start, as a part of our cultural heritage.

There have been efforts to mitigate the risks and to calm people on the way. The Ponie project was an effort to put Perl 5 on Parrot, for example, to provide for a migration path or simply a communication bridge from Perl 5 to Perl 6.

Early on, it was also felt that Perl 6 wouldn't be so different from Perl 5, syntactically. Sure, the sigils would come out being invariant, and a comma would have to be added here and there for consistency. But that was about it. Perl 5 programmers would still feel right at home.

Well, guess what? The Ponie project, even with brilliant developers behind it, slowed and finally halted at the height of the Pugs era. Turns out the reasons people wanted to further themselves from the existing Perl 5 internals — that they are a big hairy mess of intertwined C macros — also made porting to Parrot exceedingly difficult. Ponie brought some permanent improvements to the Perl 5 core, but in the end it didn't reach its goal of Perl 5 targeting Parrot.

In the meantime, Perl 6 kept improving and evolving, syntactically as well as semantically. The differences piled up. Perl 6, unfettered by backward-compatibility, could take sometimes vast leaps and reach places in the state-space Perl 5 could only dream of. Also, things kept shaking around and stabilizing, features growing together into a unified whole instead of this. All good and well, but it meant that Perl 6 drifted further away from Perl 5.

So here we are, a decade later, with two distinct languages, and no way for them to interoperate. Having them actually talk to each other is still very much on the agenda. It's just that it's a big undertaking. A couple of recent projects are attempting to put Perl 5 on a platform where it can talk to Perl 6.

I think the biggest thing to realize in all this is that Perl 6, even from the very start, could never have followed the trajectory Perl did. There was one thing that existed in 2000 that didn't exist in 1987, when Perl was announced: Perl. That's why Perl 6 is a top-down project. That's why it's a second system. And that's why the obvious measure-stick of Perl 6 is Perl, with its 13-year head start.

Rubber, meet road

I'm not complaining about this, mind. We should compare Perl 6 to Perl. And to all other scripting languages that have popped up in the same niche. We should steal ideas and adapt to modern practices, like we always do, in both Perl and Perl 6.

But for a top-down project, the biggest challenge is always to reach all the way down to the ground: to actually start being useful for someone. That's perhaps our big challenge. It was back in 2008 when I became heavily involved in the Perl 6 effort, and it still is today: be useful. Be usable. Put food on someone's family.

Perl 6 is useful to me today. Not to the extent that I can write Perl 6 code all day, but to the extent that it makes me more effective and more productive now and then. Bringing that kind of usefulness to others requires lots of work writing modules, documentation, books, and tutorials, as has been discussed elsewhere. We're making some headway with this. I'm more optimistic than I was two years ago.

We also need to work on things such as performance. Perl 6 is fast enough for some things, but overall implementations are still fairly slow. Sometimes ridiculously slow. Good work is being done in this area, too.

But here's what makes me the most optimistic about the Perl 6 effort: after a few years of watching things evolve, I've noticed that while Perl 6 is being developed top-down on the outermost scale, it's actually a series of bottom-up projects that drive Perl 6 forwards.

jnthn likes to tell about how he promised to implement junctions in Rakudo back in 2008, and then realized that junctions were actually tied to method dispatch and the object system, so he had to implement those, too. Later, while pmichaud was rebuilding the parser according to our current understanding of it, jnthn was building Rakudo's meta-object protocol, a project called 6model. Each step on the way replacing the layer below based on what we had found that the layer above required. The 6model work is part of what now enables us to port nqp and Rakudo to the JVM.

That's what makes me optimistic. While Perl 6 is undeniably, unchangeably a top-down project, highly competent people are factoring that top-down knowledge into the design of components in a bottom-up way. What's been happening in these 13 years is that we've become increasingly better and more efficient at building Perl 6.

So, how're we doing?

"Chuck Norris has actually been using Perl 6 since 1987, and has been waiting for Larry to play catch-up. :)" — dukeleto on #perl6

Let's stop and compare the state of Perl 6 today with the state of Perl back in 2000.

Mugs. Back then, Jon Orwant threw coffee mugs against a wall because he felt there was nothing to energize the community and people would walk off to other things.

Are sixers doing any mug-throwing nowadays? Yup, we did, at the Perl Reunification Summit. Liz felt, and still feels, that if we don't actively bring Perl 5 and Perl 6 together, then both projects are eventually going to fizzle out.
Crisis. As part of that, the feeling Jon Orwant was expressing was that Perl was stagnating or going nowhere.

Is Perl 6 stagnating? I wouldn't say so. We're still extremely small compared to Perl 5, but my general sense is that we're still gaining speed, becoming stronger every day. Still very much pre-peak.
Usefulness. In 2000, Perl was on its thirteenth year of solving everyday problems for people.

What about Perl 6 today? Well, it's very different. Part of this is due to the top-down nature of the project. People are waiting for it to be useful to them. Part of it is perception: people are waiting, instead of jumping in and using the language in cases where it could already be useful to them. We see this in the verb tenses people are using when they ask stuff: "Will Perl 6..." Perl 6 is about the future, but if we don't do something about that perception, we'll never get there. The problem with "tomorrow" is that it never actually arrives.
Community. In 2000, Perl drew thousands of people to conferences each year.

Is Perl 6 drawing thousands? No. The Perl people are kind enough to let us piggyback on Perl conferences and workshops, and through them we have an audience of thousands. I keep feeling that there's a big interest in Perl 6, both from within the Perl community and from other communities. But there's also a sense of waiting, of not-yet-production-ready. (And I agree.)

What remains?

Which brings us to the last, perhaps most important question. What's left for Perl 6 to become a viable, useful solution to most people out there?

For years I wouldn't go near that question. Just working along, head looking down at the current sub-projects, making Perl 6 more useful to myself and hoping that was enough. But my recent visit to FOSDEM made me realize that some kind of production release is actually within reach — say, a few years away — and a focused effort to reach a releasable state by some criteria would be highly useful for us and for others.

So here are my for criteria. This is what Perl 6 needs to be ready for the world.

Features. Slightly simplified, this is what we've been working on since the Pugs days. We are doing well on features. There's not much I feel is missing nowadays from the core language. It's a pleasant language to use and to express thoughts in. There are bugs, sure, but with each month that passes, those bugs are becoming less frequent and less serious.
Speed. Working on it. I won't make any promises about the JVM port, but I have a strong feeling it will make a big difference. Parallel to this, there are various already-implemented or to-be-implemented efforts to make the compiler smarter about generating efficient code.
Concurrency. There are some good ideas in the specification about concurrency. Some of them will work fine, some probably won't pan out. What's needed now is a solid implementation of concurrency in either Rakudo or Niecza, a test-bed for the ideas in the spec, so that users can get a feel for what's there and what's missing. I'd really like for Perl 6 to have a decent concurrency story, because it's one of the things that were promised early on for Perl 6, and one of the things that Perl never got right. Also, concurrency in all its various forms seems to be growing increasingly important in the programming world of today.
CPAN. Sure, we have an ecosystem for Perl 6. But it isn't CPAN. It's three orders of magnitude as small as CPAN. And it's not realistic to port all of CPAN either — that'll simply never happen. What we really need is a high-bandwidth connection between Perl 5 and Perl 6. We need for them to run in the same runloop. Once they do that, Perl 6 programmers can have their CPAN modules, and Perl 5 programmers can work Perl 6 code into their projects. This is also something that was promised early on. It's not an easy task.

Conclusion

Perl 6 is a multi-year project. Though today's date is a little bit arbitrary, it's worthwhile to look both backwards and forwards, to see where we currently are. And though we're structurally different from the Perl project, it's still interesting to make comparisons.

We believe we're building something really nice with Perl 6. 2013 may not be the year when we're finally production-ready, but it sure feels like a year where a lot of significant things will happen (and are already happening). And, unlike 2012, I finally feel ready to speculate about the light at the end of the tunnel.

Perl 6 changes - 2013W07

2013-02-16T08:00:00+00:00

> my $perl = Date.new: '1987-12-18' # Perl 1 release^[1]
1987-12-18
> my $perl6 = Date.new: '2000-07-18' # Perl 6 announcement^[2]
2000-07-18
> Date.today - $perl6
4596
> $perl6 - $perl
4596

Normally, I wouldn’t put irrelevant snippets of code^{[citation
needed]}. But this day is different - we have Perl 6 equidieversary (thanks, moritz). It’s 4596 days since Perl 6 was announced (Wikipedia says it was annouced 2000-07-19, but actually it was earlier). In 2000-07-19, Perl was 4596 days old. But I won’t discuss about this more, as masak already made perfectly fine article about that.

Let’s return to your regularly scheduled program.

id='new_features'>New features>

Perl 6 specification>

Sets and Bags do not flatten in list context.
Metaoperators are treated as normal operators, and aren’t making list of closures. So, for example @array X* * is -> $arg { @array X* $arg }.
Inner classes and roles must be lexically scoped.
Added List.combinations and Set.combinations.

Added Set.powerset. It’s defined like that.

multi method powerset { set ∅, self.combinations }

Rakudo Perl>

`$_` is writable in `-n`/`-p` code (one-liner).

`eqv` is faster for `Buf` objects.

Configure script fails on unknown command line options.

Pattern Matching and Unpacking

2013-02-13T05:49:13+00:00

When talking about pattern matching in the context of Perl 6, people usually think about regex or grammars. Those are indeed very powerful tools for pattern matching, but not the only one.

Another powerful tool for pattern matching and for unpacking data structures uses signatures.

Signatures are "just" argument lists:

sub repeat(Str $s, Int $count) {
    #     ^^^^^^^^^^^^^^^^^^^^  the signature
    # $s and $count are the parameters
    return $s x $count
}

Nearly all modern programming languages have signatures, so you might say: nothing special, move along. But there are two features that make them more useful than signatures in other languages.

The first is multi dispatch, which allows you to write several routines with the name, but with different signatures. While extremely powerful and helpful, I don't want to dwell on them. Look at Chapter 6 of the "Using Perl 6" book for more details.

The second feature is sub-signatures. It allows you to write a signature for a sigle parameter.

Which sounds pretty boring at first, but for example it allows you to do declarative validation of data structures. Perl 6 has no built-in type for an array where each slot must be of a specific but different type. But you can still check for that in a sub-signature

sub f(@array [Int, Str]) {
    say @array.join: ', ';
}
f [42, 'str'];      # 42, str
f [42, 23];         # Nominal type check failed for parameter '';
                    # expected Str but got Int instead in sub-signature
                    # of parameter @array

Here we have a parameter called @array, and it is followed by a square brackets, which introduce a sub-signature for an array. When calling the function, the array is checked against the signature (Int, Str), and so if the array doesn't contain of exactly one Int and one Str in this order, a type error is thrown.

The same mechanism can be used not only for validation, but also for unpacking, which means extracting some parts of the data structure. This simply works by using variables in the inner signature:

sub head(*@ [$head, *@]) {
    $head;
}
sub tail(*@ [$, *@tail]) {
    @tail;
}
say head <a b c >;      # a
say tail <a b c >;      # b c

Here the outer parameter is anonymous (the @), though it's entirely possible to use variables for both the inner and the outer parameter.

The anonymous parameter can even be omitted, and you can write sub tail( [$, *@tail] ) directly.

Sub-signatures are not limited to arrays. For working on arbitrary objects, you surround them with parenthesis instead of brackets, and use named parameters inside:

multi key-type ($ (Numeric :$key, *%)) { "Number" }
multi key-type ($ (Str     :$key, *%)) { "String" }
for (42 => 'a', 'b' => 42) -> $pair {
    say key-type $pair;
}
# Output:
# Number
# String

This works because the => constructs a Pair, which has a key and a value attribute. The named parameter :$key in the sub-signature extracts the attribute key.

You can build quite impressive things with this feature, for example red-black tree balancing based on multi dispatch and signature unpacking. (More verbose explanation of the code.) Most use cases aren't this impressive, but still it is very useful to have occasionally. Like for this small evaluator.

Perl 6 changes - 2013W06

2013-02-09T08:00:00+00:00

This time without any gimmicks (like five word sentences). This will be simple description of changes I do regularly.

id='new_features_this_time_without_building_stuff_part'>New features (this time without building stuff part)>

Rakudo Perl>

Constant folded `^..^` (infix) and `^` (prefix) operators.

Niecza Perl>

Added (borrowed from Rakudo Perl) the `Match.gist` method.

Replaced “Unhandle conext character” error with “Unhandle context character” error.

Added Cool.set method. It can be used to convert argument to set.

Added Cool.bag. Just like method above, it can be used to convert argument to bag.

I have Twitter account

2013-02-06T08:00:00+00:00

Yet another ignorable self promotion. http://twitter.com/GlitchMr

(perhaps I should have different news feed for Planet Six…)

Perl 6 changes article - 2013W05

2013-02-02T08:00:00+00:00

The changes are so fun. jnthn is working on porting. nqp will be on JVM. And then, Rakudo will be. Java is very, very fast. There are some interesting problems. But nothing impossible to solve. At least, I hope so. I would continue, but well. This is not the topic. This is article about changes. Changes in Perl 6, obviously. If you want, enjoy reading.

Rakudo Star 2013.01 was released. It’s based on Rakudo 2013.01. Those changes aren’t in.

On sidenote, a small note. Five word sentences are interesting. I am not stealing ideas. Well, actually, I am now. I should stop talking now.

id='new_features_to_build_stuff'>New features to build stuff>

Rakudo, a Perl 6 distribution>

`any` and `all` are optimized. To be exact, in statements. The `if` and `while` statements. And opposites - `unless` and `until`.

Many operators are pure now. That means they are sideeffect-free.

Site libraries are loaded first.

Pure subroutines are constant folded.

REPL supports natively typed variables.

REPL has better error reporting.

A look at the preparations behind the JVM port, and a progress update

2013-02-01T23:49:17+00:00

After my last post giving a status update on the JVM porting of NQP and the compiler toolchain Rakudo builds upon, hercynium++ left a comment suggesting that I also blog about the design work behind this effort. I liked the idea, and in this post I’ll attempt to describe it a bit. I can’t possibly capture all of the interesting things in a single post, so if this doesn’t cover aspects that are particularly interesting to anybody reading, let me know and I’ll try and find time to write something on them. :-)

It started long ago…

The first commit to the repository where I’m doing the initial porting work to the JVM may have been back in November, but that isn’t where the journey really started. We’ve known for multiple years now that we would want Rakudo and NQP to target backends besides Parrot. In that time, we’ve had to build a lot of technology in order to be able to build Rakudo at all. Some things we’ve had to build more than once because the first time didn’t produce something satisfactory (where satisfactory means “actually meets our needs”, not “is the most awesome thing ever possible”). Software is, fairly often, as much about learning as it is about building. The more complex the domain you’re working in, there more this applies, and the more likely it is that you’ll have to build one to throw away. By now we’ve thrown away a parser engine, an AST, and about 3 implementations of roles. :-)

Of course, there’s the build/buy thing, where buy in open source really means “buy into”, as in use an existing library. We’ve done a bunch of that too, such as libtommath for our big integer support and dyncall for NativeCall support. But the closer something is to the “core domain” – the thing that makes your product distinctive and special – the less able you are to use something off the shelf. Parsing Perl 6 really needs to be done with a Perl 6 grammar, using Longest Token Matching. Its object system really needs something that supports meta-programming, representation polymorphism and gradual typing. Getting BEGIN/eval right and supporting compilation and having the possibility for lexical and anonymous types and packages, which can be dynamically constructed and exported, also left us with something to build (this is the work that led to bounded serialization).

Eventual portability has been a design factor in what we’ve built for quite a while. While the only 6model implementation to have become complete enough to support all of Rakudo’s object needs so far is the one running on Parrot, the initial prototypes of 6model were done on the .Net CLR. This was in no small part to make sure that there was a feasible way to implement it on such a VM. Granted, what I actually discovered was a less than awesome way to build it on the CLR (and what I’m doing on the JVM this time around fits in far better with the JVM’s world view). But it was a design consideration from the start.

When we updated PAST, the previous AST representation, to QAST (Q is just P++ :-)) then once again portability was a concern; the VM specific bits were all placed under a QAST::VM node type. This makes it easy to escape to the underlying VM where needed or where it is most expedient, but it’s both explicit and done in a way that allows specification of what to do on other backends. As part of this work we also build support for the nqp::op abstraction directly into the AST format. The nqp::ops form an opcode set independent of any particular VM. These get mapped as part of turning a QAST tree into code for the target backend (thus meaning there’s no overhead for them in the generated code). They may map directly to the VM’s opcodes, a function or method call in the VM, or do some more complex code generation.

The other important piece of the groundwork for portability is that we implemented Rakudo in a mixture of Perl 6 and NQP, and over time have got NQP to the point where it is also written in NQP (and thus can compile itself). This has been a gradual thing; the earliest NQP editions were written directly in PIR, and with time we’ve migrated those bits to NQP – usually at the same point we were doing other improvements already. For example, pmichaud++ wrote the latest edition of the regex engine, with LTM support, in NQP. PAST, written in PIR, was replaced by QAST, written in NQP. And 6model’s meta-objects were, from the start, expressed in NQP too. It’s pretty neat that NQP’s definition of things so fundamental as classes is actually written in NQP. It means that we don’t have to port classes and roles, just the primitives they are made out of.

So digging into the JVM port itself…

With all of the above mentioned things in place, it was possible to form a fairly concrete roadmap for porting NQP, then Rakudo, over to the JVM. Being comfortable that the result would enable us to get a fully functional Rakudo on the JVM and an idea of how to get there was important. It’s easy to implement a subset, but if it isn’t factored in a way that lets you do the rest afterwards then you’re in bother and it’ll be time for re-work. My hope was that, after some years of learning about things that don’t work and replacing them with things that do, this time much of the re-work could be avoided. A starting point for this was taking a good look at the JVM’s instruction set, as well as considering what JVMs are typically good at doing.

The JVM is a stack machine. This is in contrast to Parrot, which is a register machine. Thankfully, this is mostly a code generation detail rather than being especially deep. As well as the stack, a given method can have local variables (note that everything that contains code on the JVM is called a method, even subroutines, but they call them static methods because it sounds so much more OO :-)). These can hold longer-lived things, so in a sense could be used a bit like Parrot registers. In general, the code generation from QAST uses the stack where possible and falls back to locals where needed. This is because stack usage fits well with what a JVM expects to be doing, and also what its bytecode format expresses most compactly.

Locals have an important restriction: they can only be accessed directly in the scope where they are declared. There is no notion of nested methods at the JVM level. This means that locals are not suitable for implementing lexical variables. Thankfully, there is a well established solution: promote such things onto the heap, keeping them in some kind of invocation record. This is what happens with closures in C# on the CLR, for example. There are a bunch of ways to do this transform, with various trade-offs. I’ve done one that was fairly fast to implement, but also enables lookup by array indexing rather than needing a named (hash) lookup in the vast majority of cases. As well as an array index being algorithmically cheaper than a hash lookup, the JVM supports array indexing natively in its opcode set, but not hash lookups.

Talking of allocating things on the heap brings us nicely to think about objects. JVMs are very good at fast allocation and collection of objects, because they have to be; there is no stack allocation in Java of anything non-trivial. Of course, that doesn’t mean the VM can’t do escape analysis and stack allocate under the hood. That the VM is really good at object allocation and GC means we don’t need to worry too much about lexicals leading to invocation records on the heap; there’s plenty of performant uses of this approach in the wild. Furthermore, most records will be very short lived, nicely meeting the generational hypothesis (which is that most objects are either short lived or long lived, and so we can optimize separately for each through doing generational garbage collection).

While invocation records are relatively internal, of course NQP and Perl 6 involve lots of user-visible objects. From the things you think about as objects (and call “new” on) to things like scalar containers, strings, boxed integers and so forth, both NQP and Perl 6 lead to plenty of allocations. While some things are quite predictably shaped, most come from user class definitions. Ideally, we’d like it if a Perl 6 class definition like:

class Point {
    has $!surface;
    has num $!x;
    has num $!y;
}

Was to use memory similarly to if you wrote something in Java like:

class Point {
    private Object surface;
    private double x;
     private double y;
}

At the same time, we know that the JVM’s idea of type is some way off the Perl 6 notion of type, so we can’t simply turn Perl 6 classes into JVM classes. Thankfully, 6model has from the start been designed around the idea of representation polymorphism. Really, this is just a separation of concerns: we decouple the details of memory representation and access from the notion of being a type and dispatch. The former is handled by a representation, and the latter two by a meta-object. One early but important observation I made when designing 6model is that the representation will always couple closely to the underlying runtime (and thus would need to be implemented for each runtime we wanted to run on), whereas the other bits can be expressed in a higher level way, with the common cases made efficient by caching. Thus there’s no reason to re-implement classes and roles per VM, but there is a need to provide a different, VM-specific way to do P6opaque (the default representation for NQP and Perl 6 objects).

The C implementation of P6opaque on Parrot works by calculating a memory layout – essentially, figuring out a struct “on the fly”. What’s the JVM equivalent of that? Well, that’s just creating a JVM class on the fly and loading it. Is the JVM capable of doing that? Sure, it’s easily dynamic enough. Furthermore, once we’ve done that little bit of bytecode generation, it’s a perfectly ordinary JVM class. This means that the JIT compiler knows what to do with it. Does doing any of this require changes to the meta-objects for classes in NQP and Rakudo? No, because these details are all encapsulated in the representation. Things like these are good signs for a design; it tends to show that responsibilities are correctly identified and segregated.

So, how’s the cross-compiler going?

Things are going nicely. Having got much of the way there with the NQP MOP, I turned to ModuleLoader and started to get together a basic setting (the setting being the place where built-ins are defined). With those in place, work has moved on to trying to pass the NQP test suite.

The build process cross-compiles the MOP, module loader and setting. To run the test suite, each test is taken and cross-compiled against those, then the result of compiling it is run on the JVM. The fact we invoke NQP, then invoke the JVM twice in order to run each test gives quite a bit of fixed overhead per test; once we have NQP itself (that is, the compiler) cross-compiled and self-hosting on the JVM it’ll be down to a single invocation.

The NQP test suite for the NQP language itself consists of 65 test files. 3 of them are specific to Parrot, so there’s 62 that are interesting to make run. As of today, we pass 46 of those test files in full. While some of those passing tests exercise relatively simple things (literals, operators, variables, conditionals, loops, closures), others exercise more advanced features (classes, roles, basic MOP functionality, runtime mixins and so forth). Of the 16 test files that remain, 9 of them depend on regexes or grammars. Getting those to run will be the focus of the next major chunk of work: porting the regex compiler and getting the NFA, Cursor and Match classes to cross-compile (which will involve some portability improvements). The other 7 relate to non-trivial, but smaller-than-grammars features (for example, 2 are about multiple dispatch, which I’m working on porting at the moment).

It was only three weeks ago when I wrote that the JVM port did not even manage “hello world” yet, and that I had little more to show than something that could turn a range of QAST trees into JVM bytecode. Three weeks later and we’re running around 75% of the NQP test files, and timotimo++ was even able to feed an almost unmodified Levenstein distance implementation written in NQP to the cross-compiler and have it run on the JVM.

So, mad amounts of coding have taken place? Well, only sorta…I’ve taught two three-day classes for $dayjob in the last couple of weeks also. :-) Mostly, progress has been fast now because the foundations it is building upon have proved fairly solid. For the backend, this is in no small part down to having grown a test suite for the QAST to JVM phase of the work as it progressed. The fact we could happily couple this new backend to the existing NQP parser is thanks to the compiler being structured as a pipeline of stages, each one strongly isolated from the others, just passing a data structure between them. In my teaching work, I often encourage automated testing and talk a lot about the importance of enforcing carefully chosen, well-defined, information-centric boundaries between components. It’s pleasing to see these things paying off well in my Perl 6 work also. :-)

I am going to FOSDEM

2013-01-31T23:22:38+00:00

(Using exclusively five word sentences.)

Perl Mongers needed speakers quickly. "Very Late Call for Papers". "Why so late", you ask. Perl dev room was denied. Another community got the room. Perl only got a booth. The other community backed out. Perl then got the room. Therefore, talks were requested urgently. Only about one week notice. The announcement is recorded here.

I missed that blog post. But I got an email. Wendy wrote to some people. I was one of them. Talk about very short notice. Eight days before the talk! That must be a record. Nothing to be done, though. The invitation was nicely worded. I considered whether to go. Finally I decided I would.

My talk concerns Perl 6. I have given it before. It was in Bristol, England. You were likely not there. That time, jnthn helped me. Now I will talk alone. I must give it quickly. I only have 20 minutes. That is not a lot. I rather like challenges, though. Looking forward to it all.

Will you come to FOSDEM? I certainly hope you will. If you do, stop by. I will give my talk. "Where is my flying car?" A reference to the future. In the future, cars fly. Also, Perl 6 is everywhere. Especially in the flying cars. It will be totally awesome. My talk is about that. Or sorta kinda about that. It is about Perl 6. Why is it not released? What makes me keep hoping? What has been implemented already? That is what it covers.

Looking forward to the weekend.

Rakudo Star 2013.01 released

2013-01-30T18:51:44+00:00

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the January 2013 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the January 2013 release is available from the download page. A Windows .MSI version of Rakudo star will usually appear in the downloads area shortly after the tarball release.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2013.01 [0] of the Rakudo Perl 6 compiler [1], version 4.10.0 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community.

Some of the new features added to this release include:

Sink context (what some other languages call void context) is now enforced correctly. This means that for-loops are now lazy by default. It fixes the bug where a map in sink context would not execute, and also means that a Failure returned to sink context will be properly thrown.
‘require’ now works with indirect module names
Restored socket read semantics to returning the requested number of bytes
$obj.Some::Role::meth() now passes the correct $obj
try/CATCH now returns Nil when the CATCH is triggered, rather than the exception; this brings it in line with try without a CATCH
whatever-star cases of splice now implemented
can now import multis with the same name from different modules, provided all dispatchers are onlystar

This release also contains a range of bug fixes, improvements to error reporting
and better failure modes.

The following features have been deprecated or modified from previous
releases due to changes in the Perl 6 specification, and are being removed
or changed as follows:

postcircumfix:<[ ]> and postcircumfix:<{ }> will become multi-subs rather than multi-methods. Both at_pos and at_key will remain methods.
Unary hyper ops currently descend into nested arrays and hashes. This will change to make them equivalent to a one-level map.
The Str.ucfirst builtin is deprecated; it will be replaced by Str.tc.
Leading whitespace in rules and under :sigspace will no longer be
converted to <.ws> . For existing regexes that expect this conversion, add a <?> in front of leading whitespace to make it meta again.
The ?-quantifier on captures in regexes currently binds the capture slot to a List containing either zero or one Match objects; i.e., it is equivalent to “** 0..1″. In the future, the ?-quantifier will bind the slot directly to a captured Match or to Nil. Existing code can manage the transition by changing existing ?-quantifiers to use “** 0..1″, which will continue to return a List of matches.

There are some key features of Perl 6 that Rakudo Star does not
yet handle appropriately, although they will appear in upcoming
releases. Some of the not-quite-there features include:

advanced macros
threads and concurrency
Unicode strings at levels other than codepoints
interactive readline that understands Unicode
non-blocking I/O
much of Synopsis 9

There is an online resource at http://perl6.org/compilers/features
that lists the known implemented and missing features of Rakudo
and other Perl 6 implementations.

See http://perl6.org/ for links to much more information about
Perl 6, including documentation, example code, tutorials, reference
materials, specification documents, and other supporting resources.
A draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf
in the release tarball.

The development team thanks all of the contributors and sponsors
for making Rakudo Star possible. If you would like to contribute,
see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org
mailing list, or join us on IRC #perl6 on freenode.

[0] https://github.com/rakudo/rakudo/blob/nom/docs/announce/2013.01
[1] http://github.com/rakudo/rakudo
[2] http://parrot.org/

Perl 6 changes - 2013W04

2013-01-26T08:00:00+00:00

Today, the main features is .delta method in DateTime and Date. And as usually, bug fixes that make language do what you mean.

id='new_features'>New features>

Perl 6 specification>

Added `DateTime.delta` and `Date.delta` to the specification.

Rakudo Perl>

`s///` resets `$/` on failure.

`s///` returns `False` on failure.

Regular expressions allow `$()` syntax for arbitrary expressions.

Socket functions work better with non-ASCII characters.

Errors about function only taking positional arguments now specify name of function that was called.

`DateTime.truncated-to` now takes `TimeUnit` enum as argument.

Implemented `DateTime.delta` and `Date.delta`.

Names are suggested for `--> Type` (subroutine return type).

`/< hello >/` now works correctly (matching `hello`), just like `/< hello world >/` already did.

Helpful error messages

2013-01-20T08:00:00+00:00

Rakudo lately got an interesting change. Consider following buggy code (for array binary search), written in functional style (I’m using recursion). The bug is that it doesn’t work because of two different identifiers binary-search and binary_search.

Trying to run it shows compile time error message. The compile part is important. If you would remove binary-search call at end, it still would report an error, unlike Python.

===SORRY!===
Undeclared routine:
    binary_search used at lines 2, 8, 9. Did you mean '&binary-search'?

Now you know what’s wrong and you can easily fix it. Unlike let’s say, Jekyll that I use for my blog. Not only it doesn’t work on my PC for some reason, but also refused to highlight my code - it gave some sort of XML error (it simply said “REXML could not parse this XML/HTML”). I gave up and put it on Gist. And the first attempt was failed, because embed code is <script> and Jekyll self-closed the tag (XML-style, but this is HTML). This is annoying.

The Perl 6 Coding Contest (2012 edition) is now closed

2013-01-19T17:24:06+00:00

"I've found this year the tasks were harder, but I've also spend more time thinking about the problems and less worrying about bugs in the interpreter... which is definitely good."
        — a contestant

"This year's problems were a lot more approachable to me and motivated me to learn more perl6."
        — another contestant

...aaaand we're done. The five weeks are up.

I'm now all caught up processing submissions sent to me. So let's summarize, as usual:

32 people signed up as contestants this year.
8 contestants submitted solutions.
24 submissions passed a base-test run and were accepted.

These figures during p6cc2010 and p6cc2011 were (18, 5, 26) and (35, 6, 27), respectively. So all in all, it seems that we've entered a steady state both in terms of contestants and submissions. I'm grateful to get so much interesting Perl 6 code to read through and review.

Many people told me this year that they had noble plans to send in lots of solutions, but in the end didn't find the time for it. I'm assuming many contestants I haven't heard from have similar stories. That's fine; the contest is designed to encourage you to do the tasks, but not to force you to. If the contest made you take a look at the tasks, or at Perl 6, then in some sense that's a win, too.

My plan is, just like previous years, to go through the five tasks in order, publish all the reviews and an appropriately thoughtful blog post that summarizes the tasks and its solutions. With luck, we'll all learn something together.

It takes time to do these reviews. Sometimes a lot of time. That said, I hope to get through these submissions quickly. Expect something like a post each week. That's the goal.

After which I'll sum up and select a winner, also in a blog post. Exciting!

So, expect the next blog post to be about the first task: "Tell knights from knaves based on what they say." Onwards!

Perl 6 changes - 2013W03

2013-01-19T08:00:00+00:00

It’s another Perl 6 changes article (at this point I wonder why I still do that). But as long the changes are impressive (and the cake is the lie), I guess mentioning changes is worth it.

Rakudo 2013.01 was tagged in the Git repository. It wasn’t yet released, but it’s really close to the releease (and frozen).

id='new_features'>New features>

Rakudo Perl (in Rakudo 2013.01)>

`** 0` in regexpes works correctly matching nothing.

Rakudo Perl (post Rakudo 2013.01)>

Compiler hints invalid variable names. For example, typing `$variabel` when you already have `$variable` declared hints that variable.

Niecza Perl>

More meaningful error message is thrown when `printf` tries to consume more arguments than available.

A quick JVM backend update

2013-01-18T00:44:00+00:00

Things have been moving along quite rapidly on the JVM backend since my last post. Sadly, I’m too sick to hack on anything much this evening (hopefully, this turns out to be a very temporary affliction…) but I can at least just about write English, so I figured I’d provide a little update. :-)

Last time I blogged here, I was able to compile various QAST trees down to JVM bytecode and had a growing test suite for this. My hope was that, by some inductive handwaving, being able to compile a bunch of QAST nodes and operations correctly would mean that programs made up of a whole range of them would also compile correctly. In the last week or so, that has come to pass.

Having reached the point of having coverage of quite a lot of QAST, I decided to look into getting an NQP frontend plugged into my QAST to JVM backend. In the process, I found that NQP lacked the odd VM abstraction here and there in the common prelude that it includes with every QAST tree it produces. Thankfully, this was easily rectified. Even better, I got rid of a couple of old hacks that were no longer required. With those things out of the way, I found that this common prelude depended on a couple of operations that I’d not got around to implementing in the JVM backend. These were also simple to add. And…here endeth the integration story. Yup, that was it: I now had a fledgling NQP cross-compiler. An NQP compiler running on Parrot, but producing output for the JVM.

This result is rather exciting, because…

It’s using exactly the same parse and action stages as when we’re targeting Parrot. No hacks, no fork. The QAST tree we get from the NQP source code that goes in is exactly the one we get when targeting Parrot. Everything that happens differently happens is beyond that stage, in the backend. This is an extremely positive sign, architecturally.
With a couple of small additions to handle the prelude, I was immediately able to cross-compile simple NQP programs and run them on the JVM. There’s no setting or MOP yet, but the basics (variables, loops, subroutines with parameters, even closures) Just Worked.
The program I wrote to glue the JVM backend work and the existing NQP frontend together was about 30 lines of NQP code.
This whole integration process was about an afternoon’s worth of work.

Since I got that working, my focus has been on getting nqp-mo (the NQP meta-objects) to cross-compile. This is where classes and roles are implemented, and thus is a prerequisite for cross-compiling the NQP setting, which is in turn a prerequisite for being able to start cross-compiling and running the NQP test suite. The NQP MOP is about 1500 lines of NQP code, and at this point I’ve got about 1400 of them to cross-compile. So I’m almost there with it? Well, not quite. Turns out that the next thing I need to port is the bounded serialization stuff. That’s a rather hairy chunk of work.

Anyway, things are moving along nicely. The immediate roadmap is to get the bounded serialization to the point where it’s enough for the NQP MOP, then move on to getting a minimal setting cross compiling. Beyond that, it’ll be working through the test suite, porting the regex compilation and seeing what else is needed to cross-compile the rest of NQP.

Perl 6 changes - 2013W02

2013-01-12T08:00:00+00:00

I’m going to show another Perl 6 changes article. I’m so lazy, that I really don’t know what to put here, so I guess I’ll now show the list of new features.

id='new_features'>New features>

Perl 6 specification>

`-->` (return type) acts more like a normal subroutine parameter.

Added `squish` method, working like `uniq` utility in UNIX - it only removes elements if it is duplicate of previous element.

Adjusted precedence of set operators.

STD.pm6>

Adjusted precedence of set operators.

Rakudo Perl>

Implemented `require ::($module) 'something'` syntax.

`$socket.read($bytes)` now returns `$bytes` bytes, if possible. Previously it only read only 256 bytes.

`$object.My::Role::method()` works correctly and uses correct `self`.

`try` with `CATCH` returns `Nil` on exception, like it did without explicit `CATCH`.

Failures in sink context won’t throw if they were handled.

`splice` supports whatever star (`*`).

Adjusted precedence of set operators.

`Junction.Str` now gives string, instead of junction with every element converted to the string.

Niecza Perl>

Adjusted precedence of set operators.

A Bunch of Rakudo News

2013-01-10T01:59:04+00:00

Seems it’s high time for some news here. It’s not that I didn’t do any blogging about Perl 6 in December; it’s just that all of those posts were over on the Perl 6 advent calendar. Anyway, now it’s a new year, and I’m digging back into things after an enjoyable Christmas and New Year’s break in the UK with family and friends. Here’s a bunch of things that already happened but I didn’t get around to talking about here yet, and some things that will be coming up.

Better Parse Errors In 2012.12

Ever had Rakudo tell you there’s a problem on line 1, when it’s really on line 50? Or wished that even in the common case where it gets the line right, it would tell you exactly where on the line things went wrong? Or how about the time it told you “Confused” because you got a closing paren too many?

Many of my contributions to the 2012.12 Rakudo release centered around improving its reporting of parse errors. STD, the standard Perl 6 grammar, has had much better error reporting than Rakudo for a while. Therefore, I spent a bunch of time aligning our error reporting more closely with what STD does. Some of this is cosmetic: you get the colored output and the indication of the parse location. But while these cosmetic changes will be the most immediately visible thing, the changes go far deeper. Of note, a high water mark is kept so we can be a lot more accurate in reporting where things came unstuck, and we track what was expected so we can produce better errors. Just doing the cosmetic stuff without being able to give it a better location to report wouldn’t have helped so much. :-)

One other change is that we don’t bail out on the first thing that’s wrong when it’s possible to survive and continue parsing. When this is possible, up to 10 errors will be reported (since that’s typically a screen worth). Of course, some things just hose the parse and we can’t continue in any sensible way.

Hopefully, these improvements will make using Rakudo feel a lot nicer. Already on channel, I can feel the feedback we’re giving about parse errors when people use the evalbot is often a lot more pleasant and informative. Of course, there’ll be more improvements in the future too, but this is a big step forward.

Faster Auto-Threading

The junction auto-threader could sometimes be insanely slow. As in, ridiculously so. After hearing a bunch of reports about this, I decided to dig in and work out why. A rewrite later, the little benchmark I was using with it ran almost 30 times faster. Not so bad… :-) This change also made it into the 2012.12 release.

JVM Backend Preparations Underway

I’ve talked plenty about plans for NQP and Rakudo to run on things besides Parrot for a while now. Over the last year or two, we’ve laid a lot of the groundwork for this. What’s been especially pleasing is that it’s also made Rakudo a better quality Perl 6 implementation on Parrot, thanks to the many architectural improvements. Of note, in many places we’ve closed semantic gaps between what Perl 6 wants and the primitives we were building it out of; the new QAST (Q Abstract Syntax Tree) is a great example.

Anyway, with NQP now being written pretty much entirely in NQP, and many of the right abstractions in place, it felt like time to start slowly picking away at getting 6model ported to the JVM and work on turning QAST trees into Java bytecode. I quietly started on this in November, and mentioned the existence of the work on #perl6 in December. I was delighted to see Coke++ jump in and start working through the Low Hanging Fruit – a file where I’m putting tasks that should be relatively easy to pick off. I actually had to re-fill it, after the last round were depleted. ;-) By now, quite a few bits of QAST are supported and the 6model on JVM implementation is coming along nicely. Yes, this means it’s already capable of doing basic meta-programming stuff.

Note that this work isn’t at the stage where it’s of use for anything yet. You can’t even write a say(“hello world”) in NQP and have it run on the JVM yet, since all the work so far is just about turning QAST trees into JVM bytecode and building the runtime support it needs. That may seem like a curious way to work, but once you do enough compiler stuff you find yourself thinking quite naturally in trees. It meant I didn’t have to worry about creating some stripped-down NQP that could emit super-simple trees to be able to test really simple things. After all, the goal is to run NQP itself on the JVM, and then Rakudo, and only then will things be interesting to the everyday user.

To address a couple of immediate concerns that some may have…

No, this is not a case of “stop running on Parrot, start running on JVM”. It’s adding an additional backend, much like pmurias++ has been working on adding a JavaScript backend for NQP. Of course, I expect resource allocation in the future to be driven by which backends users desire most. For some, the JVM is “that evil Oracle thing” and they don’t want to touch it. For others, the JVM is “the only thing our company will deploy on”. Thus I expect this work to matter more to some people than others. That’s fine.
No, targeting multiple backends doesn’t mean performance-sucking abstractions everywhere. It’s a pretty typical thing for a compiler to do. As usual, it’s about picking the right abstractions. The debugger was implemented as an additional Rakudo frontend without a single addition to Rakudo or NQP or anything anywhere in the stack. That was possible because things were designed well. I’m sure the process of getting things running on the JVM will flag up a few places where things aren’t designed as well as they need to be, but already I’m seeing a lot of places where things are mapping over very nicely indeed.
No, this doesn’t mean that all other Rakudo development will grind to a halt. I’ve been working on the JVM backend stuff in November and December; both months saw a huge amount of Rakudo progress too. Things will go on that way.

Type System Improvements

Rakudo does a lot of things well when it comes to supporting the various kinds of types Perl 6 offers, but there are some weak areas. Here are some of the things I plan to focus on:

Getting the definedness constraint stuff working better (the :D and :U annotations). At the moment, they’re supported as a special case in multiple dispatch and the signature binder. In reality, they’re meant to be first class and work everywhere. You may not think you care about these much. Actually, you probably at least indirectly do, because once the optimizer is able to analyze them, it’ll be able to do a bunch more inlining of things than it can today. :-)
Getting coercion types in place. Again, this is turning the special-cased “as” syntax into the much more general coercion type syntax (for example, Int(Str) is the type that accepts a Str and coerces it into an Int).
Getting native types much better supported. At the moment, you can use them but…there are pitfalls. Having them available has been a huge win for us in the CORE setting, where we’ve used them in the built-ins. But they’re still a bit “handle with great care”. I want to change that.
Implementing compact arrays.
Improving our parametric type and type variable support. Many things work, but there are some fragile spots and some bugs that want some attention.

I intend to have some of these things in the January release, and a bunch more in the February one. We’ll see how I get along. :-)

Perl 6 changes - 2013W01

2013-01-05T08:00:00+00:00

I have finally updated the number before ‘W’. Well, we have the new year - 2013. And because of that I would like to say ‘Happy new year!’.

This is yet another article in “Perl 6 changes”. I lately don’t put articles other than those, but I don’t have many ideas for those, really.

By the way, I find it interesting that no new posts on Planet Six appeared between my previous Perl 6 changes article and current Perl 6 changes article.

id='incompatible_changes'>Incompatible changes>

Rakudo Perl>

“SAFE” setting is now removed. You’re supposed to use “RESTRICTED” setting now.

id='new_features'>New features>

Rakudo Perl>

Using non-associative operators when association is needed throws typed error message.

Compile time optimizations for `(|)` signature (accepting anything), It’s worth handling because it’s commonly used for `proto`s.

Fixed typos in `X::Import::Redeclaration` error message.

Removed old `perl6doc` documentation.

`DateTime.Date` now works correctly.

`.pick` now randomizes better on large ranges.

Niecza Perl>

`infix:<->(Instant, Instant)` now supported.

Bitwise operations support big integers.

Perl 6 changes - 2012W52

2012-12-29T08:00:00+00:00

This is last “Perl 6 changes” update in this year. Enjoy.

id='new_features'>New features>

Perl 6 specification>

`uniq` method/subroutine, existing for long time in implementations is now mentioned in specification (with `===` semantics).

Rakudo Perl>

Use of suffix regex modifiers (P5-ism) in `s///`, like `s////g` is now detected.

Rakudo Star 2012.12 released

2012-12-27T19:53:53+00:00

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the December 2012 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the December 2012 release is available from the download section. A Windows .MSI version of Rakudo star will usually appear in the downloads area shortly after the tarball release.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2012.12 of the Rakudo Perl 6 compiler, version 4.10.0 of the Parrot Virtual Machine, and various modules, documentation, and other resources collected from the Perl 6 community.

Some of the new features added to this release include:

Parse errors are much improved, and follow STD, the standard parser,
much more closely; they are more accurate and more information is given
Rakudo now keeps parsing after some less serious errors
Better errors for various parse failures
The junction autothreader is now an order of magnitude faster
Texas (ASCII) versions of the Set and Bag operators implemented
Nested Pairs now give correct .perl output
{ a => $_ } now correctly considered a block, not a hash as
before

This release also contains a range of performance improvements, bug fixes, improvements to error reporting and better failure modes.

The following features have been deprecated or modified from previous releases due to changes in the Perl 6 specification, and are being removed or changed as follows:

‘for’-loops will become lazy, and are only evaluated eagerly in eager or sink (void) context. This means that if a for-loop is the last statement in a routine, it will usually run after the routine has returned, so it cannot call return() anymore.
Unary hyper ops currently descend into nested arrays and hashes. This will change to make them equivalent to a one-level map.
The Str.ucfirst builtin is deprecated; it will be replaced by
Str.tc.
Leading whitespace in rules and under :sigspace will no longer be converted to <.ws> . For existing regexes that expect this conversion, add a <?> in front of leading whitespace to make it meta again.
The ?-quantifier on captures in regexes currently binds the capture slot to a List containing either zero or one Match objects; i.e., it is equivalent to “** 0..1″. In the future, the ?-quantifier will bind the slot directly to a captured Match or to Nil. Existing code can manage the transition by changing existing ?-quantifiers to use “** 0..1″, which will continue to return a List of matches.

There are some key features of Perl 6 that Rakudo Star does not
yet handle appropriately, although they will appear in upcoming
releases. Some of the not-quite-there features include:

advanced macros
threads and concurrency
Unicode strings at levels other than codepoints
interactive readline that understands Unicode
non-blocking I/O
much of Synopsis 9

There is an online resource that lists the known implemented and missing features of Rakudo and other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the programmer that a given feature isn’t implemented, but there are many that we’ve missed. Bug reports about missing and broken features are welcomed at rakudobug@perl.org.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. A draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf in the release tarball.

Day 24 – An Advent Calendar

2012-12-24T00:00:47+00:00

Recently I was unpacking some boxes of books and came across a book entitled "BASIC Computer Programs for the Home" by Charles D. Sternberg. Apparently my father had purchased this book in the early 1980s and had given it to me. In any case, my name was scrawled in the front cover in the manner an adolescent me would have done.

Mostly this book is filled with simple BASIC programs that manage simple databases of various things: recipes, household budget, address book, music collections, book collections, school grades, etc. But the program that caught my eye and made me think of the Perl 6 Advent Calendar was one for printing a calendar starting at a particular month.

Now, the version in this book is a little simple in that it asks for the starting month, year, the day of the week that the first month starts on, and how many months to print. I wanted something a little more like the Unix utility cal(1) program. Luckily, Perl 6 has date handling classes as part of the specification and both major implemenations, Rakudo and Niecza, have actual implementations of these which should make creating the calendar quite easy.

For reference, the output of the Unix cal(1) utility looks like this:

       December 2012
    Su Mo Tu We Th Fr Sa
                       1
     2  3  4  5  6  7  8
     9 10 11 12 13 14 15
    16 17 18 19 20 21 22
    23 24 25 26 27 28 29
    30 31

It also has some options to change the output in various ways, but I just want to focus on reproducing the above basic output.

I'll need a list of month names and weekday abbreviations:

    constant @months = <January February March April May June July
                        August September October November December>;
    constant @days = <Su Mo Tu We Th Fr Sa>;

And it looks like the month and year are centered above the days of the week. Generating a calendar for May shows this to be the case, so I'll need a routine that centers text:

    sub center(Str $text, Int $width) {
        my $prefix = ' ' x ($width - $text.chars) div 2;
        my $suffix = ' ' x $width - $text.chars - $prefix.chars;
        return $prefix ~ $text ~ $suffix;
    }

Now, the mainline code needs two things: a month and a year. From this it should be able to generate an appropriate calendar. But, we should have a reasonable default for these values I think. Today's month and year seem reasonable to me:

    sub MAIN(:$year = Date.today.year, :$month = Date.today.month) {

but if it's not today's month and year, then it's some arbitrary month and year we need info about. To do this we construct a new Date object from the month and year given.

        my $dt = Date.new(:year($year), :month($month), :day(1) );

Looking at the calendar generated for December, it seems like we may actually output up to 6 rows of numbers since the month can start and end on a partial week. In order to implement this, I think I'll need some "slots" for each day. Each slot will either be empty or will contain the day of the month. The number of empty slots at the beginning of the month correspond to the day of the week that the first of the month occurs on. If the first is on Sunday, there will be 0 empty slots, if the first is on a Monday there will be 1 empty slot, if the first is on a Tuesday, there will be 2 empty slots, etc. This is remarkably similar to the number we get when we interrogate a Date object for the day of the week. The only wrinkle is that it returns 7 for Sunday when we actually need a 0. That's easily remedied with a modulus operator however:

        my $dt = Date.new(:year($year), :month($month), :day(1) );
        my $ss = $dt.day-of-week % 7;
        my @slots = ''.fmt("%2s") xx $ss;

That gives us the empty slots at the beginning, but what about the ones that actually contain the days of the month? Easy enough, we'll just generate a number for each day of the month using the Date object we created earlier.

        my $days-in-month = $dt.days-in-month;
        for $ss ..^ $ss + $days-in-month {
            @slots[$_] = $dt.day.fmt("%2d");
            $dt++
        }

Now we've got an array with appropriate values in the appropriate positions, all that's left is to actually output the calendar. Using the header line for our weekdays as a metric for the width of the calendar, and the routine we created for centering text, we can output the header portion of the calendar:

        my $weekdays = @days.fmt("%2s").join: " ";
        say center(@months[$month-1] ~ " " ~ $year, $weekdays.chars);
        say $weekdays;

Then we iterate over each slot and output the appropriate values. If we've reached the end of the week or the end of the month, we output a newline:

        for @slots.kv -> $k, $v {
            print "$v ";
            print "\n" if ($k+1) %% 7 or $v == $days-in-month;
        }

Putting it all together, here is the final program:

    #!/usr/bin/env perl6

    constant @months = <January February March April May June July
                        August September October November December>;
    constant @days = <Su Mo Tu We Th Fr Sa>;


    sub center(Str $text, Int $width) {
        my $prefix = ' ' x ($width - $text.chars) div 2;
        my $suffix = ' ' x $width - $text.chars - $prefix.chars;
        return $prefix ~ $text ~ $suffix;
    }

    sub MAIN(:$year = Date.today.year, :$month = Date.today.month) {
        my $dt = Date.new(:year($year), :month($month), :day(1) );
        my $ss = $dt.day-of-week % 7;
        my @slots = ''.fmt("%2s") xx $ss;

        my $days-in-month = $dt.days-in-month;
        for $ss ..^ $ss + $days-in-month {
            @slots[$_] = $dt.day.fmt("%2d");
            $dt++
        }

        my $weekdays = @days.fmt("%2s").join: " ";
        say center(@months[$month-1] ~ " " ~ $year, $weekdays.chars);
        say $weekdays;
        for @slots.kv -> $k, $v {
            print "$v ";
            print "\n" if ($k+1) %% 7 or $v == $days-in-month;
        }
    }

Normally, cal(1) will highlight today's date on the calendar. That's a feature I left out of my calendar implementation but it could easily be added with Term::ANSIColor. Also, there's a little bit of coupling between the data being generated in the slots and the output processing (the slots are all formatted to be 2 characters wide in anticipation of the output). There are some other improvements that could be done, but for a first cut at a calendar in Perl 6, I'm happy. :-)

Day 23 – Macros

2012-12-23T17:07:05+00:00

Syntactic macros. The Lisp gods of yore provided humanity with this invention, essentially making Lisp a programmable programming language. Lisp adherents often look at the rest of the programming world with pity, seeing them fighting to invent wheels that were wrought and polished back in the sixties when giants walked the Earth and people wrote code in all-caps.

And the Lisp adherents see that the rest of us haven’t even gotten to the best part yet, the part with syntactic macros. We’re starting to get the hang of automatic memory management, continuations, and useful first-class functions. But macros are still absent from this picture.

In part, this is because in order to have proper syntactic macros, you basically have to look like Lisp. You know, with the parentheses and all. Lisp ends up having almost no syntax at all, making every program a very close representation of a syntax tree. Which really helps when you have macros starting to manipulate those same trees. Other languages, not really wanting to look like Lisp, find it difficult-to-impossible to pull off the same trick.

The Perl languages love the difficult-to-impossible. Perl programmers publish half a dozen difficult-to-impossible solutions to CPAN before breakfast. And, because Perl 6 is awesome and syntactic macros are awesome, Perl 6 has syntactic macros.

It is known, Khaleesi.

What are macros?

For reasons even I don’t fully understand, I’ve put myself in charge of implementing syntactic macros in Rakudo. Implementing macros means understanding them. Understanding them means my brain melts regularly. Unless it fries. It’s about 50-50.

I have this habit where I come into the #perl6 channel, and exclaiming “macros are just X!” for various values of X. Here are some samples:

Macros are just syntax tree manipulators.
Macros are just “little compilers”.
Macros are just a kind of templates.
Macros are just routines that do code substitution.
Macros allow you to safely hand values back and forth between the compile-time world and the runtime world.

But the definition that I finally found that I like best of all comes from scalamacros.org:

Macros are functions that are called by the compiler during compilation. Within these functions the programmer has access to compiler APIs. For example, it is possible to generate, analyze and typecheck code.

While we only cover the “generate” part of it yet in Perl 6, there’s every expectation we’ll be getting to the “analyze and typecheck” parts as well.

Some examples, please?

Coming right up.

macro checkpoint {
  my $i = ++(state $n);
  quasi { say "CHECKPOINT $i"; }
}

checkpoint;
for ^5 { checkpoint; }
checkpoint;

The quasi block is Perl 6′s way of saying “a piece of code, coming right up!”. You just put your code in the quasi block, and return it from the macro routine.

This code inserts “checkpoints” in our code, like little debugging messages. There’s only three checkpoints in the code, so the output we’ll get looks like this:

CHECKPOINT 1
CHECKPOINT 2
CHECKPOINT 2
CHECKPOINT 2
CHECKPOINT 2
CHECKPOINT 2
CHECKPOINT 3

Note that the “code insertion” happens at compile time. That’s why we get five copies of the CHECKPOINT 2 line, because it’s the same checkpoint running five times. If we had had a subroutine instead:

sub checkpoint {
  my $i = ++(state $n);
  say "CHECKPOINT $i";
}

Then the program would print 7 distinct checkpoints.

CHECKPOINT 1
CHECKPOINT 2
CHECKPOINT 3
CHECKPOINT 4
CHECKPOINT 5
CHECKPOINT 6
CHECKPOINT 7

As a more practical example, let’s say you have logging output in your program, but you want to be able to switch it off completely. The problem with an ordinary logging subroutine is that with something like:

LOG "The answer is { time-consuming-computation() }";

The time-consuming-computation() will run and take a lot of time even if LOG subsequently finds that logging was turned off. (That’s just how argument evaluation works in a non-lazy language.)

A macro fixes this:

constant LOGGING = True;

macro LOG($message) {
  if LOGGING {
    quasi { say {{{$message}}} };
  }
}

Here we see a new feature: the {{{ }}} triple-block. (Syntax is likely to change in the near future, see below.) It’s our way to mix template code in the quasi block with code coming in from other places. Doing say $message; would have been wrong, because $message is a syntax tree of the message to be logged. We need to inject that syntax tree right into the quasi, and we do that with a triple-block.

The macro conditionally generates logging code in your program. If the constant LOGGING is switched on, the appropriate logging code will replace each LOG macro invocation. If LOGGING is off, each macro invocation will be replaced by literally nothing.

Experience shows that running no code at all is very efficient.

What are syntactic macros?

A lot of things are called “macros” in this world. In programming languages, there are two big categories:

Textual macros. They substitute code on the level of the source code text. C’s macros, or Perl 5′s source filters, are examples.
Syntactic macros. They substitute code on the level of the source code syntax tree. Lisp macros are an example.

Textual macros are very powerful, but they represent the kind of power that is just as likely to shoot half your leg off as it is to get you to your destination. Using them requires great care, of the same kind needed for a janitor gig at Jurassic Park.

The problem is that textual macros don’t compose all that well. Bring in more than one of them to work on the same bit of source code, and… all bets are off. This puts severe limits on modularity. Textual macros, being what they are, leak internal details all over the place. This is the big lesson from Perl 5′s source filters, as far as I understand.

Syntactic macros compose wonderfully. The compiler is already a pipeline handing off syntax trees between various processing steps, and syntactic macros are simply more such steps. It’s as if you and the compiler were two children, with the compiler going “Hey, you want to play in my sandbox? Jump right in. Here’s a shovel. We’ve got work to do.” A macro is a shovel.

And syntactic macros allow us to be hygienic, meaning that code in the macro and code outside of the macro don’t step on each other’s toes. In practice, this is done by carefully keeping track of the macros context and the mainline’s context, and making sure wires don’t cross. This is necessary for safe and large-scale composition. Textual macros don’t give us this option at all.

Future

Both of the examples in this post work already in Rakudo. But it might also be useful to know where we’re heading with macros in the next year or so. The list is in the approximate order I expect to tackle things.

Un-hygiene. While hygienic macros are the sane and preferable default, sometimes you want to step on the toes of the mainline code. There should be an opt-out, and escape hatch. This is next up.
Introspection. In order to analyze and typecheck code, not just generate it, we need to be able to take syntax trees coming in as macro arguments, and look inside of them. There are no tools for that yet, and there’s no spec to guide us here. But I’m fairly sure people will want this. The trick is to come up with something that doesn’t tie us down to one compiler’s internal syntax-tree format. Both for the sake of compiler interoperability and future compatibility.
Deferred declarations. The sandbox analogy isn’t so frivolous, really. If you declare a class inside a quasi block, that declaration is limited (“sandboxed”) to within that quasi block. Then, when the code is injected somewhere in the mainline because of a macro invocation, it should actually run. Fortunately, as it happens, the Rakudo internals are factored in such a way that this will be fairly straightforward to implement.
Better syntax. The triple-block syntax is probably going away in favor of something better. The problem isn’t the syntax so much as the fact that it currently only works for terms. We want it to work for basically all syntactic categories. A solid proposal for this is yet to materialize, though.

With each of these steps, I expect us to find new and fruitful uses of macros. Knowing my fellow Perl 6 developers, we’ll probably find some uses that will shock and disgust us all, too.

In conclusion

Perl 6 is awesome because it puts you, the programmer, in the driver seat. Macros are simply more of that.

Implementing macros makes your brain melt. However, using them is relatively straightforward.

Threads for Rakudo Perl 6

2012-12-22T10:36:54+00:00

So it has come to this. Threads.pm is up and running, bringing the ever-wanted threaded execution to the most popular Perl 6 implementation.

You’re looking for TL;DR, aren’t you? Here’s what it’s capable of:

use Threads;
use Semaphore;

my @elements;
my $free-slots = Semaphore.new(value => 10);
my $full-slots = Semaphore.new(value => 0);

sub produce($id) {
    my $i = 0;
    loop {
        $free-slots.wait;
        @elements.push: $i;
        $i++;
        $full-slots.post;
    }
}

sub consume($id) {
    loop {
        $full-slots.wait;
        my $a = @elements.shift;
        $free-slots.post;
    }
}

for 1..5 -> $i {
    async sub { produce($i)  }
}

for 5..10 -> $i {
    async sub { consume($i) }
}

Doesn’t look that awesome. I mean, it’s just a producer-consumer problem, what’s the big deal? Let me repeat:

OMG, RAKUDO HAS WORKING THREADS.

So, once we’re done celebrating and dancing macarena all around, there’ll always be someone to ask “hold on, there’s gotta be a caveat. Something surely is missing, explain yourself!”

I’ll be delighted to say “nope, everything’s there!”, but that’d make me a liar. Yeah, there are missing pieces. First, those aren’t really native threads – just green threads. Native OS threads are already implemented in Parrot VM, but NQP (the language that Rakudo is based on) still doesn’t support them, so before some volunteer comes along to fix them, you’ll still have to build parrot --without-threads (which means: use green threads, not OS threads) for Threads.pm to work. But fear not! The API is exactly the same, so once native threads are there, both Threads.pm and the code you write with it should work without any changes.

But green threads are fine too! Except for one minor detail: whenever any of them blocks on IO, the entire Parrot comes to a halt. The plan is for Parrot threads scheduler to handle it nicely, but it’s not there yet, so if you expected nice and easy async IO, sorry, but you’re stuck on MuEvent :)

Yep, we’re not really there yet. But I claim it’s closer than ever. We have working threads implementation. You can write code with that, and it’s not a PITA. Go for it! There’s a lot of room to improve it. I didn’t try really hard for Threads.pm to follow the concurrency synopsis (in my defense, I think niecza doesn’t follow it either :)), and I think that once we unleash a wolfpack of developers which can work towards something that we’ll all love to use.

Perl 6 changes - 2012W51

2012-12-22T08:00:00+00:00

TODO: Insert the description of changes. Also, mention that Rakudo 2012.12 (Warszawa) was released and that Rakudo isn’t Rakudo Star.

id='incompatible_changes'>Incompatible changes>

Rakudo Perl (in Rakudo 2012.12)>

When disambiguating hash/block, and `$_` variable is used, Rakudo assumes block. This could break your code if you were expecting a hash.

Rakudo Perl (post Rakudo 2012.12)>

`my $a; { $a; my $a }` is an error.

`for` runs in sink (void) or list context, just like `map` (actually, `for` does exactly what `map` does). If you were depending on it, you can use `eager do` before `for` to force eager execution.

id='new_features'>New features>

Rakudo Perl (in Rakudo 2012.12)>

Added `X::Comp::Group` exception. It’s used when compilation can produce multiple issues.

Added support for warnings. Code like `my ($a, $a);` only causes warnings, not fatal errors.

Added `X::Undeclared::Symbols` exception for variables that weren’t declared, but used.

You can use `.message` on `Exception`s.

Fixed `.perl` output for a `Pair` that has key that is a `Pair`.

`X::Item` is thrown for `@a[NaN]` or `@a[Inf]`. It isn’t incompatible change because it wasn’t working before anyway.

Day 22 – Parsing an IPv4 address

2012-12-22T00:01:22+00:00

Guest post by Herbert Breunung (lichtkind).

Perl 5 brought regexes to mainstream programming and set a standard, one that is felt as relevant even in Redmond. Perl 6, of course, steps up the game by adding many new features to the regex camp, including easy-to-build grammars for your own complex parsers. But without getting too complex, you can get a lot of joy out of Perl 6′s rx (that’s how Perl 6 spells Perl 5′s qr operator, that enables you to save a Regex in a variable).

Because the Perl 6 regex syntax is less littered with exceptional cases, Larry Wall also likes to joke that he put the “regular” back into “regular expression”.

Some of the changes are:

most special variables are gone,
non-capturing groups and other grouping syntax is easier to type,
no more single/multi line modes,
x mode became default, making whitespace non-significant by default.

In summary, regexes are more regular than in Perl 5, confirming Larry’s joke. They try a bit harder to make your life easier when you need to match text. Under the hood, regexes have blossomed out into a complete sub-language within the bigger Perl 6 language. A language with its own parsing rules.

But don’t fret; not everything has changed. Some things remain the same:

/\d+/

This regex still matches one or more consecutive digits.

Similarly, if you want to capture the digits, you can do this, just like you’re used to:

/(\d+)/

You’ll find the matched digits in $0, not $1 as in Perl 5. All the special variables $0, $1, $2 are really syntactic sugar for indexing the match variable ($/[0], $/[1], $/[2]). Because indices start at 0, it makes sense for the first matched group to be $0. In Perl 5, $0 contains the name of the script or program, but this has been renamed into $*EXECUTABLE_NAME in Perl 6.

Should you be interested in getting all of the captured groups of a regex match, you can use @(), which is syntactic sugar for @($/).

The object in the $/ variable holds lots of useful information about the last match. For example, $/.from will give you the starting string position of the match.

But $0 will get us far enough for this post. We use it to extract individual features from a string.

Sometimes we want to extract a whole bunch of similar things at once. Then we can use the :g (or :global) modifier on the regex:

$_ = '1 23 456 78.9';
say .Str for m:g/(\d+)/; # 1 23 456 78 9

Note that the :g — as opposed to prior regex implementations — sits up front, right at the start of the regex. Not at the end. That way, when you read the regex from left to right, you will know from the start how the regex is doing its matching. No more end-heavy regex expressions.

Matching “all things that look like this” is so useful, that there’s even a dedicated method for that, .comb:

$str.comb(/\d+/);

If you’re familiar with .split, you can think of .comb as its cheerful cousin, matching all the things that .split discards.

Let’s tackle the matching of an IPv4 address. Coming from a Perl 5 angle, we expect to have to do something like this:

/(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/

This won’t do in Perl 6, though. First of all, the {} blocks are real blocks in a Perl 6 regex; they contain Perl 6 code. Second, because Perl 6 has lots of error handling to catch p5isms, like this, you’ll get an error saying “Unsupported use of {N,M} as general quantifier; in Perl 6 please use ** N..M (or ** N..*)”.

So let’s do that. To match between one and three digits in a Perl 6 regex, we should type:

/\d ** 1..3/

Note how the regex sublanguage re-uses parts from the main Perl 6 language. ** can be seen as a kind of exponentiation (if we squint), in that we’re taking \d “to the between-first-and-third power”. And the range notation 1..3 exists both outside and within regexes.

Using our new knowledge about the repetition quantifier, we end up with something like this:

/(\d**1..3) \. (\d**1..3) \. (\d**1..3) \. (\d**1..3)/

That’s still kinda clunky. We might end up wishing that we could use the repetition operator again, but those literal dots in between prevent us from doing that. If only we could specify repetition a given number of times and a divider.

In Perl 6 regexes, you can.

/ (\d ** 1..3) ** 4 % '.' /

The % operator here is a quantifier modifier, so it can only follow on a quantifier like * or + or **. The choice of % for this function is relatively new in Perl 6, and you may prefer to read it as “modulo”, just like in the main language. That is, “match four groups of digits, modulo literal dots in between”. Or you could think of the dots in between as the “remainder”, the separators that are left after you’ve parsed the actual elements.

Oh, and you might’ve noticed that \. changed to '.' on the way. We can use either; they mean exactly the same. In Perl 5, there isn’t a simple rule saying which symbols have a magic meaning and which ones simply signify themselves. In Perl 6, it’s easy: word characters (alphanumerics and the underscore) always signify themselves. Everything else has to be escaped or quoted to get its literal meaning.

Putting it all together, here’s how we would extract IPv4 addresses out of a string:

$_ = "Go 127.0.0.1, I said! He went to 173.194.32.32.";

say .Str for m:g/ (\d ** 1..3) ** 4 % '.' /;
# output: 127.0.0.1 173.194.32.32

Or, we could use .comb:

$_ = "Go 127.0.0.1, I said! He went to 173.194.32.32.";
my @ip4addrs = .comb(/ (\d ** 1..3) ** 4 % '.' /);

If we’re interested in individual integers, we can get those too:

$_ = "Go 127.0.0.1, I said! He went to 173.194.32.32.";
say .list>>.Str.perl for m:g/ (\d ** 1..3) ** 4 % '.' /;
# output: ("127", "0", "0", "1") ("173", "194", "32", "32")

If you want to know more, read the S05, or watch me battling with my slide deck and the English language in this presentation about regexes.

Day 21 – Collatz Variations

2012-12-21T12:25:40+00:00

The Collatz sequence is one of those interesting “simple” math problems that I’ve run into a number of times. Most recently a blog post on programming it in Racket showed up on Hacker News. As happens so often, I instantly wanted to implement it in Perl 6.

sub collatz-sequence(Int $start) { 
    $start, { when * %% 2 { $_ / 2 }; when * !%% 2 { 3 * $_ + 1 }; } ... 1;
}

sub MAIN(Int $min, Int $max) {
    say [max] ($min..$max).map({ +collatz-sequence($_) });        
}

This is a very straightforward implementation of the Racket post’s max-cycle-length-range as a stand-alone p6 script. collatz-sequence generates the sequence using the p6 sequence operator. Start with the given number. If it is divisible by two, do so: when * %% 2 { $_ / 2 }. If it is not, multiply by three and add 1: when * !%% 2 { 3 * $_ + 1 }. Repeat this until the sequence reaches 1.

MAIN(Int $min, Int $max) sets up our main function to take two integers. Many times I don’t bother with argument types in p6, but this provides a nice feedback for users:

> perl6 collatz.pl blue red
Usage:
  collatz.pl <min> <max>

The core of it just maps the numbers from $min to $max (inclusive) to the length of the sequence (+collatz-sequence) and then says the max of the resulting list ([max]).

Personally I’m a big fan of using the sequence operator for tasks like this; it directly represents the algorithm constructing the Collatz sequence in a simple and elegant fashion. On the other hand, you should be able to memoize the recursive version for a speed increase. Maybe that would give it an edge over the sequence operator version?

Well, I was wildly wrong about that.

sub collatz-length($start) {
    given $start {
        when 1       { 1 }
        when * !%% 2 { 1 + collatz-length(3 * $_ + 1) } 
        when * %% 2  { 1 + collatz-length($_ / 2) } 
    }
}

sub MAIN($min, $max) {
    say [max] ($min..$max).map({ collatz-length($_) });        
}

This recursive version, which makes no attempt whatsoever to be efficient, is actually better than twice as fast as the sequence operator version. In retrospect, this makes perfect sense: I was worried about the recursive version making a function call for every iteration, but the sequence version has to make two, one to calculate the next iteration and the other to check and see if the ending condition has been reached.

Well, once I’d gotten this far, I thought I’d better do things correctly. I wrote two framing scripts, one for timing all the available scripts, the other for testing them to make sure they work!

my @numbers = 1..200, 10000..10200;

sub MAIN(Str $perl6, *@scripts) {
    my %results;
    for @scripts -> $script {
        my $start = now;
        qqx/$perl6 $script { @numbers }/;
        my $end = now;

        %results{$script} = $end - $start;
    }

    for %results.pairs.sort(*.value) -> (:key($script), :value($time)) {
        say "$script: $time seconds";
    }
}

This script takes as an argument a string that can be used to call a Perl 6 executable and a list of scripts to run. It runs the scripts using the specified executable, and times them using p6′s now function. It then sorts the results into order and prints them. (A similar script I won’t post here tests each of them to make sure they are returning correct results.)

In the new framework, the Collatz script has changed a bit. Instead of taking a min and a max value and finding the longest Collatz sequence generated by a number in that range, it takes a series of numbers and generates and reports the length of the sequence for each of them. Here’s the sequence operator script in its full new version:

sub collatz-length(Int $start) { 
    +($start, { when * %% 2 { $_ / 2 }; when * !%% 2 { 3 * $_ + 1 }; } ... 1);
}

sub MAIN(*@numbers) {
    for @numbers -> $n {
        say "$n: " ~ collatz-length($n.Int);
    }
}

For the rest of the scripts I will skip the MAIN sub, which is exactly the same in each of them.

Framework established, I redid the recursive version starting from the new sequence operator code.

sub collatz-length(Int $n) {
    given $n {
        when 1       { 1 }
        when * %% 2  { 1 + collatz-length($_ div 2) }
        when * !%% 2 { 1 + collatz-length(3 * $_ + 1) }
    } 
}

The sharp-eyed will notice this version is different from the first recursive version above in two significant ways. This time I made the argument Int $n, which instantly turned up a bit of a bug in all implementations thus far: because I used $_ / 2, most of the numbers in the sequence were actually rationals, not integers! This shouldn’t change the results, but is probably less efficient than using Ints. Thus the second difference about, it now uses $_ div 2 to divide by 2. This version remains a great improvement over the sequence operator version, running in 4.7 seconds instead of 13.3. Changing when * !%% 2 to a simple default shaves another .3 seconds off the running time.

Once I started wondering how much time was getting eaten up by the when statements, rewriting that bit using the ternary operator was an obvious choice.

sub collatz-length(Int $start) { 
    +($start, { $_ %% 2 ?? $_ div 2 !! 3 * $_ + 1 } ... 1);
}

Timing results: Basic sequence 13.4 seconds. Sequence with div 11.5 seconds. Sequence with div and ternary 9.7 seconds.

That made me wonder what kind of performance I could get from a handcoded loop.

sub collatz-length(Int $n is copy) {
    my $length = 1;
    while $n != 1 {
        $n = $n %% 2 ?? $n div 2 !! 3 * $n + 1;
        $length++;
    }
    $length;
}

That’s by far the least elegant of these, I think, but it gets great performance: 3 seconds.

Switching back to the recursive approach, how about using the ternary operator there?

sub collatz-length(Int $n) {
    return 1 if $n == 1;
    1 + ($n %% 2 ?? collatz-length($n div 2) !! collatz-length(3 * $n + 1));
}

This one just edges out the handcoded loop, 2.9 seconds.

Can we do better than that? How about memoization? is cached is supposed to be part of Perl 6; neither implementation has it yet, but last year’s Advent calendar has a Rakudo implementation that still works. Using the last version changed to sub collatz-length(Int $n) is cached { works nicely, but takes 3.4 seconds to execute. Apparently the overhead of caching slows it down a bit. Interestingly, the non-ternary recursive version does speed up with is cached, from 4.4 seconds to 3.6 seconds.

Okay, instead of using a generic memoization, how about hand-coding one?

sub collatz-length(Int $n) {
    return 1 if $n == 1;
    state %lengths;
    return %lengths{$n} if %lengths.exists($n);
    %lengths{$n} = 1 + ($n %% 2 ?? collatz-length($n div 2) !! collatz-length(3 * $n + 1));
}

Bingo! 2.7 seconds.

I’m sure there are lots of other interesting approaches for solving this problem, and encourage people to send them in. In the meantime, here’s my summary of results so far:

Script	Rakudo	Niecza
bin/collatz-recursive-ternary-hand-cached.pl	2.5	1.7
bin/collatz-recursive-ternary.pl	3	1.7
bin/collatz-loop.pl	3.1	1.7
bin/collatz-recursive-ternary-cached.pl	3.2	N/A
bin/collatz-recursive-default-cached.pl	3.5	N/A
bin/collatz-recursive-default.pl	4.4	1.8
bin/collatz-recursive.pl	4.9	1.9
bin/collatz-sequence-ternary.pl	9.9	3.3
bin/collatz-sequence-div.pl	11.6	3.5
bin/collatz-sequence.pl	13.5	3.8

The table was generated from timing-table-generator.pl.

Day 20 – Dynamic variables and DSL-y things

2012-12-20T00:01:06+00:00

Today, let’s talk about DSLs.

Post from the past: a motivating example

Two years ago I wrote a blog post about Nim, a game played with piles of stones. I just put in ASCII diagrams of the actual Nim stone piles, telling myself that if I had time, I would put in fancy SVG diagrams, generated with Perl 6.

Naturally, I didn’t have time. My self-imposed deadline ran out, and I published the post with simple ASCII diagrams.

But time is ever-regenerative, and there for people who want it. So, let’s generate some fancy SVG diagrams with Perl 6.

Have bit array, want SVG

What do we need, exactly? Well, a subroutine that takes an array of piles as input and generates an SVG file would be a really good start.

Let’s take the last “image” in the post as an example:

3      OO O
4 OOOO
5 OOOO    O

For the moment, let’s ignore the numbers at the left margin; they’re just counting stones. We summarize the piles themselves as a kind of bitmap, which also forms the input to the function:

my @piles =
    [0, 0, 0, 0, 0, 0, 0, 0, 1],
    [1, 1, 1, 1, 0, 1, 1, 0, 1],
    [1, 1, 1, 1, 0, 0, 0, 0, 1];

nim-svg(@piles);

At this point, we need only create the nim-svg function itself, and make it render SVG from this bitmap. Since I’ve long since tired of outputting SVG by hand, I use the SVG module, which comes bundled with Rakudo Star.

use SVG;

sub nim-svg(@piles) {
    my $width = max map *.elems, @piles;
    my $height = @piles.elems;

    my @elements = gather for @piles.kv -> $row, @pile {
        for @pile.kv -> $column, $is_filled {
            if $is_filled {
                take 'circle' => [
                    :cx($column + 0.5),
                    :cy($row + 0.5),
                    :r(0.4)
                ];
            }
        }
    }
    
    say SVG.serialize('svg' => [ :$width, :$height, @elements ]);
}

I think you can follow the logic in there. The subroutine simply iterates over the bitmap, turning 1s into circles with appropriate coordinates.

That’s it?

Well, this will indeed generate an SVG image for us, with the stones correctly placed. But let’s look again at the input that helped create this image:

    [0, 0, 0, 0, 0, 0, 0, 0, 1],
    [1, 1, 1, 1, 0, 1, 1, 0, 1],
    [1, 1, 1, 1, 0, 0, 0, 0, 1];

Clearly, though we can discern the stones and gaps in there if we squint in a bit-aware programmer’s fashion, the input isn’t… visually attractive. (The zeroes even look like stones, even though they’re gaps!)

We can do better

Instead of using a bit array, let’s start from the desired SVG image and try to make the input look like that.

So, this is what I would prefer to write instead of a bitmask:

nim {
  _ _ _ _ _ _ _ _ o;
  o o o o _ o o _ o;
  o o o o _ _ _ _ o;
}

That’s better. That looks more like my original ASCII diagram, while still being syntactic Perl 6 code.

Making a DSL

Wikipedia talks about a DSL as a language “dedicated to a particular problem domain”. Well, the above way of specifying the input would be a DSL dedicated to solving the draw-SVG-images-of-Nim-positions domain. (Admittedly a fairly narrow domain. But I’m mostly out to show the potential of DSLs in Perl 6, not to change the world with this particular DSL.)

Now that we have the desired end state, how do we connect the wires and make the above work? Clearly we need to declare three subroutines: nim, _, o. (Yes, you can name a subroutine _, no sweat.)

sub nim(&block) {
    my @*piles;
    my @*current-pile;

    &block();
    finish-last-pile();
    
    nim-svg(@*piles);
}

sub _(@rest?) {
    unless @rest {
        finish-last-pile();
    }
    @*current-pile = 0, @rest;
    return @*current-pile;
}

sub o(@rest?) {
    unless @rest {
        finish-last-pile();
    }
    @*current-pile = 1, @rest;
    return @*current-pile;
}

Ok… explain, please?

A couple of things are going on here.

The two variables @*piles and @*current-pile are dynamic variables which means that they are visible not just in the current lexical scope, but also in all subroutines called before the current scope has finished. Notably, the two subroutines _ and o.
The two subroutines _ and o take an optional parameter. On each row, the rightmost _ or o acts as a silent “start of pile” marker, taking the time to do a bit of bookkeeping with the piles, storing away the last pile and starting on a new one.
Each row in the DSL-y input basically forms a chain of subroutine calls. We take this into account by both incrementally building the @*current-pile array at each step, all the while returning it as (possible) input for the next subroutine call in the chain.

And that’s it. Oh yeah, we need the bookkeeping routine finish-last-pile, too:

sub finish-last-pile() {
    if @*current-pile {
        push @*piles, [@*current-pile];
    }
    @*current-pile = ();
}

So, it works?

Now, the whole thing works. We can turn this DSL-y input:

nim {
  _ _ _ _ _ _ _ _ o;
  o o o o _ o o _ o;
  o o o o _ _ _ _ o;
}

…into this SVG output:

<svg
  xmlns="http://www.w3.org/2000/svg"
  xmlns:svg="http://www.w3.org/2000/svg"
  xmlns:xlink="http://www.w3.org/1999/xlink"
  width="9" height="3">

  <circle cx="8.5" cy="0.5" r="0.4" />
  <circle cx="0.5" cy="1.5" r="0.4" />
  <circle cx="1.5" cy="1.5" r="0.4" />
  <circle cx="2.5" cy="1.5" r="0.4" />
  <circle cx="3.5" cy="1.5" r="0.4" />
  <circle cx="5.5" cy="1.5" r="0.4" />
  <circle cx="6.5" cy="1.5" r="0.4" />
  <circle cx="8.5" cy="1.5" r="0.4" />
  <circle cx="0.5" cy="2.5" r="0.4" />
  <circle cx="1.5" cy="2.5" r="0.4" />
  <circle cx="2.5" cy="2.5" r="0.4" />
  <circle cx="3.5" cy="2.5" r="0.4" />
  <circle cx="8.5" cy="2.5" r="0.4" />
</svg>

Yay!

Summary

The principles I used in this post are fairly easy to generalize. Start from your desired DSL, and create the subroutines to make it happen. Have dynamic variables handle the communication between separate subroutines.

DSLs are nice because they allow us to shape the code we’re writing around the problem we’re solving. Using relatively little “adapter code”, we’re left to focus on describing and solving problems in a natural way, making the programming language rise to our needs instead of lowering ourselves down to its needs.

Parrot 4.11.0 "All together - Happy Birthday Lovebird" Released! by Reini Urban

2012-12-19T01:49:52+00:00

On behalf of the Parrot team, I'm proud to announce Parrot 4.11.0, also known
as "All together - Happy Birthday Lovebird".
Parrot (http://parrot.org/) is a virtual machine aimed at running all
dynamic languages.

Parrot 4.11.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/4.11.0/),
or by following the download instructions at http://parrot.org/download.
For those who would like to develop on Parrot, or help develop Parrot
itself, we recommend using
Git to retrieve the source code to get the latest and best Parrot code.

Threads are considered experimental. There are a known GC bugs when
running large threads
on darwin and sometimes linux. There are no known problems when
running large single-threaded programs.
Reading from shared variables is transparent, writing must be done via
scheduled updater subs
on the owner thread.
You can use the API without using native OS threads by configuring
--without-threads.

Parrot 4.11.0 News:
- Core
+ Parrot now supports fast and lightweight OS threads, based
on Nat Tucks's
initial GSoC work on green threads and then Stefan Seifert's extension
to true parallel OS threads as hybrid threads.
See http://wknight8111.blogspot.co.at/2010/08/gsoc-threads-chandons-results.html
and http://niner.name/Hybrid_Threads_for_the_Parrot_VM.pdf
Summary:
http://perl6advent.wordpress.com/2012/12/11/day-11-parrot-threads/
+ New parrot cmdline option --numthreads
+ Export new dynpmc_class_<ClassName> types [GH #870].
+ Changed core_ops library format and ops2c to contain the
PBC_COMPAT only,
not the parrot version triple. Removed the duplicate
PARROT_DYNOP_CORE_INIT
cpp definition in include/parrot/oplib/core_ops.h, use only
PARROT_CORE_OPLIB_INIT from config.h
+ Store PBC_MAJOR and PBC_MINOR in the config hash, stored by
auto::revision
+ Added getprotobyname method to the Socket PMC
- Documentation
+ Marked unused API parameters
+ Install man pages for all binaries
- Languages
+ nqp-rx tests updated from perl6/nqp-rx.git, and enabled to
run on make test
- Platforms
+ More compilation fixes on Windows without working mt.exe or
ranlib=rem,
enable smoking without git binary.
- Tools
+ Re-enabled pbc_dump -h --header-only
+ Improved pmc2c, added more UNUSED(arg) macros [GH #836].
Parrot is now almost
-Wunused-parameters safe.
- Tests
+ failing Pod::Simple pod_todo.t tests are now skipped
+ added nqp-rx tests
+ removed duplicate ext/nqp-rx/t/p6regex tests
+ rewrote and enabled native_pbc tests [GH #394],
renamed native_pbc testfiles

The SHA256 message digests for the downloadable tarballs are:
46de72022dfe521fb8c5d95453e1d091d4d20d8f7ca4f2ed0cd1420b682f20dc
parrot-4.11.0.tar.gz
0632ad0ca9557df391088bddbbc2608a4503adb5679dcbe62c065a73d4ce49de
parrot-4.11.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next supported release is 15 January 2013.

Enjoy!
--
Reini Urban
http://cpanel.net/ http://www.perl-compiler.org/

Day 19 – Gather and/or Coroutines

2012-12-19T00:01:43+00:00

Today I’ll write about coroutines, gather-take and why they are as much fun as one another. But since it’s all about manipulating control flow, I took the liberty to reorganize the control flow of this advent post, so coroutines will finally appear somewhere at the end of it. In the meantime I’ll introduce the backstory, the problems that coroutines solved and how it looks from the Perl 6 kitchen.

LWP::Simple is all fun and games, but sometimes you can’t afford to wait for the result to come. It would make sense to say “fetch me this webpage and drop me a note when you’re done with it”. That’s non trivial though; LWP::Simple is a black box, which we tell “get() this, get() that” and it gives us the result back. There is no possible way to intercept the internal data it sends there and around. Or is there?

If you look at Perl 5′s AnyEvent::HTTP, you’ll see that it reimplemented the entire HTTP client to have it non-blocking. Let’s see if we can do better than that.

First thing, where does LWP::Simple actually block? Behind our backs it uses the built-in IO::Socket::INET class. When it wants data from it, it calls .read() or .recv() and patiently waits until they’re done. If only we could somehow make it not rely on those two directly, hmm…

„I know!”, a gemstone-fascinated person would say, „We can monkey-patch IO::Socket::INET”. And then we have two problems. No, we’ll go the other way, and follow the glorious path of Dependency Injection.

That sounds a bit scary. I’ve heard about as many definitions of Dependency Injection as many people I know. The general idea is to not create objects inside other objects directly; it should be possible to supply them from the outside. I like to compare it to elimination of „magic constants”. No one likes those; if you think of classes as another kind of magic constants which may appear in somebody else’s code, this is pretty much what this is about. In our case it looks like this:

# LWP::Simple make_request
my IO::Socket::INET $sock .= new(:$host, :$port);

There we go. “IO::Socket::INET” is the magic constant here; if you want to use a different thing, you’re doomed. Let’s mangle it for a bit and allow the socket class to come from the outside.

We’ll add an attribute to LWP::Simple, let’s call it $!socketclass

has $.socketclass = IO::Socket::INET;

If we don’t supply any, it will just fallback to IO::Socket::INET, which is a sensible default. Then, instead of the previous .new() call, we do

my $sock = $!socketclass.new(:$host, :$port);

The actual patch (https://github.com/tadzik/perl6-lwp-simple/commit/93c182ac2) is a bit more complicated, as LWP::Simple supports calling get() not only on constructed objects but also on type objects, which have no attributes set, but we only care about the part shown above. We have an attribute $!socketclass, which defaults to IO::Socket::INET but we’re free to supply another class – dependency-inject it. Cool! So in the end it’ll look like this:

class Fakesocket is IO::Socket::INET {
    method recv($) {
        note 'We intercepted recv()';
        callsame;
    }

    method read($) {
        note 'We intercepted read()';
        callsame;
    }
}

# later
my $lwp = LWP::Simple.new(socketclass => Fakesocket);

And so our $lwp is a fine-crafted LWP::Simple which could, theorically, give the control flow back to us while it waits for read() and recv() to finish. So, how about we put theory into practice?

Here start the actual coroutines, sorry for being late :)

What do we really need in our modified recv() and read()? We need a way to say „yeah, if you could just stop executing and give time to someone else, that would be great.” Oh no, but we have no threads! Luckily, we don’t need any. Remember lazy lists?

my @a := gather { for 1..* -> $n { take $n } }

So on one hand we run an infinite for loop, and on the other we have a way to say „give back what you’ve come up with, I’ll catch up with you later”. That’s what take() does: it temporarily jumps out of the gather block, and is ready to get back to it whenever you want it. Do I hear the sound of puzzles clicking together? That’s exactly what we need! Jump out of the execution flow and wait until we’re asked to continue.

class Fakesocket is IO::Socket::INET {
    method recv($) {
        take 1;
        callsame;
    }

    method read($) {
        take 1;
        callsame;
    }
}

# later
my @a := gather {
    $lwp.get("http://jigsaw.w3.org/HTTP/300/301.html");
    take "done";
}

# give time to LWP::Simple, piece by piece
while ~@a.shift ne "done" {
    say "The coroutine is still running"
}
say "Yay, done!";

There we go! We just turned LWP::Simple into a non-blocking beast, using almost no black magic at all! Ain’t that cool.

We now know enough to create some syntactic sugar around it all. Everyone likes sugar.

module Coroutines;
my @coroutines;
enum CoroStatus <still_going done>;

sub async(&coroutine) is export {
    @coroutines.push($(gather {
        &coroutine();
        take CoroStatus::done;
    }));
}

#= must be called from inside a coroutine
sub yield is export {
    take CoroStatus::still_going;
}

#= should be called from mainline code
sub schedule is export {
    return unless +@coroutines;
    my $r = @coroutines.shift;
    if $r.shift ~~ CoroStatus::still_going {
        @coroutines.push($r);
    }
}

We maintain a list of coroutines currently running. Our async() sub just puts a block of code in the execution queue. Then every call to yield() will make it jump back to the mainline code. schedule(), on the other hand, will pick the first available coroutine to be run and will give it some time to do whatever it wants.

Now, let us wait for the beginning of the post to catch up.

Day 18 – Formulas: resistance is futile

2012-12-18T00:01:24+00:00

Today, Perl turns 25: happy birthday Perl! There’s too much to say about this language, its philosophy, its culture, … So here, I would just thank all people who make Perl a success, for such a long time.

Introduction

A formula is “an entity constructed using the symbols and formation rules of a given language“, according to Wikipedia as of this writing. These words sound really familiar for any Perl 6 users who have already played with grammars, however this is not the purpose of this article. Instead, the aim is to demonstrate how the Perl 6 language can be easily extended in order to use formulas literally in the code.

There are many domains, like Mathematics, Physics, finance, etc., that use their own specific languages. When writing programs for such a domain, it could be less error-prone and simpler to use its specific language instead of using a specific API. For example, someone who has knowledge in electronic may find the formula below:

4.7kΩ ± 5%

far more understandable than the following piece of code:

my $factory MeasureFactory.getSharedInstance();
my $resistance = $factory.createMeasure(value     => 4700,
                                        unit      => Unit::ohm,
                                        precision => 5);

The formula 4.7kΩ ± 5% will be used all along this article as an example.

Symbol `k`: return a modified value

Let’s start with the simplest symbol: k. Basically this is just a multiplier placed after a numeric value. To make the Perl 6 language support this new operator, there’s no need to know much about Perl 6 guts: operators are just funny looking sub-routines:

sub postfix:<k> ($a) is tighter(&infix:<*>) { $a * 1000 }

This just makes 4.7k return 4.7 * 1000, for example. To be a little bit picky, such kind of multiplier should not be used without a unit (ex. Ω) and not be coupled to another multiplier (ex. μ). This would have made this article a little bit more complex, so this is left as an exercise to the reader :) Regarding the tighter trait, it is already well explained in three other articles.

Symbols `%`: return a closure

The next symbol is %: it is commonly used to compute a ratio of something, that’s why 5% shouldn’t naively be transformed into 0.05. Instead, it creates a closure that computes the given percent of whatever you want:

sub postfix:<%> ($a) is tighter(&infix:<*>) { * * $a / 100 }

It’s now possible to write $f = 5%; $f(42) or 5%(42) directly, and this returns 2.1. It is worth saying this doesn’t conflict with the infix:<%> operator (modulo), that is, 5 % 42 still returns 5.

Symbol `Ω`: create a new `Measure` object

Let’s go on with the Ω symbol. One possibility is to tie the unit and the value in the same object, as in the Measure class defined below. The ACCEPTS method is explained later but the idea in this case is that two Measure objects with two different units can’t match together:

enum Unit <volt ampere ohm>;

class Measure {
    has Unit $.unit;
    has $.value;

    method ACCEPTS (Measure:D $a) {
        $!unit == $a.unit && $!value.ACCEPTS($a.value);
    }
}

Then, one operator per unit can be defined in order to hide the underlying API, that is, to allow 4.7kΩ as an equivalent of Measure.new(value => 4.7k, unit => ohm):

sub postfix:<V> (Real:D $a) is looser(&postfix:<k>) {
    Measure.new(value => $a, unit => volt)
}
sub postfix:<A> (Real:D $a) is looser(&postfix:<k>) {
    Measure.new(value => $a, unit => ampere)
}
sub postfix:<Ω> (Real:D $a) is looser(&postfix:<k>) {
     Measure.new(value => $a, unit => ohm)
}

Regarding the ACCEPTS method, it is used by ~~, the smartmatch operator, to check if the left operand can match the right operand, the one with the ACCEPTS method. In other terms, $a ~~ $b is equivalent to $b.ACCEPTS($a). Typically, this allows the intuitive comparison between two different types, like scalars and containers for example.

In this example, this method is overloaded to ensure two Measure objects can match only if they have the same unit and if their values match. That means 4kΩ ~~ 4.0kΩ is True whereas 4kΩ ~~ 4kV is False. Actually, there are many units that can mix altogether, typically currencies (¥€$) and the ones derived from the International System of Unit. But as usual, when something is a little bit more complex, it is left as an exercise to the reader ;)

Symbol `±`: create a `Range` object

There’s only one symbol left so far: ±. In the example, it is used to indicate the tolerance of the resistance. This tolerance could be either absolute (expressed in Ω) or relative (expressed in %), thus the new infix:<±> operator has several signatures and have to be declared with a multi keyword. In both cases, the value is a new Range objects with the right bounds:

multi sub infix:<±> (Measure:D $a, Measure:D $b) is looser(&postfix:<Ω>) {
    die if $a.unit != $b.unit;
    Measure.new(value => Range.new($a.value - $b.value,
                                   $a.value + $b.value),
                unit => $a.unit);
}

multi sub infix:<±> (Measure:D $a, Callable:D $b) is looser(&postfix:<Ω>) {
    Measure.new(value => Range.new($a.value - $b($a.value),
                                   $a.value + $b($a.value)),
                unit => $a.unit);
}

Actually, any Callable object could be used in the second variant, not only the closures created by the % operators.

So far, so good! It’s time to check in the Perl6 REPL interface if everything works fine:

> 4.7kΩ ± 1kΩ
Measure.new(unit => Unit::ohm, value => 3700/1..5700/1)

> 4.7kΩ ± 5%
Measure.new(unit => Unit::ohm, value => 4465/1..4935/1)

It looks good, so all the code above ought to be moved into a dedicated module in order to be re-used at will. Then, a customer could load it and write literally:

my $resistance = 4321Ω;
die "resistance is futile" if !($resistance ~~ 4.7kΩ ± 5%);

As of this writing, this works both in Niecza and Rakudo, the two most advanced implementations of Perl 6.

Symbols that aren’t operators

Symbols in a formula are not always operators, they can be symbolic constants too, like π. In many languages, constants are just read-only variables, which sounds definitely weird: a variable isn’t supposed to be … variable? In Perl 6, a constant can be a read-only variable too (hmm) or a read-only term (this sounds better). For example, to define the constant term φ:

constant φ = (1 + sqrt(5)) / 2;

Conclusion

In this article the Perl 6 language was slightly extended with several new symbols in order to embed simple formulas. Although it is possible to go further by changing the Perl 6 grammar in order to embed more specific languages, that is, languages that don’t have the same grammar rules. Indeed, there are already two such languages supported by Perl 6: regexp and quotes. The same way, Niecza use a custom language to connect its portable parts to the unportable.

Bonus: How to type these exotic symbols?

Most of the Unicode symbols can be type in Xorg — the most used interface system on Linux — thanks to the Compose key, also named Multi key. When this special key is pressed, all the following key-strokes are somewhat merged in order to compose a symbol.

There’s plenty of documentation about this support elsewhere on Internet, so only the minimal information is provided here. First, to map the Compose key to the Caps Lock key, write in a X terminal:

sh> setxkbmap -option compose:caps

Some compositions are likely already defined, for instance <caps> followed by + then - should now produce ±, but both Ω and φ are likely not defined. One solution is to write a
~/.XCompose file with the following content:

include "%L" # Don't discard the current locale setting.

<Multi_key> <o> <h> <m>      : "Ω"  U03A9
<Multi_key> <O> <underscore> : "Ω"  U03A9
<Multi_key> <underscore> <O> : "Ω"  U03A9

<Multi_key> <p> <h> <y> : "φ"  U03C6
<Multi_key> <o> <bar>   : "φ"  U03C6
<Multi_key> <bar> <o>   : "φ"  U03C6

This takes effect for each newly started applications. Feel free to leave a comment if you know how to add such a support on other
systems.

Day 17 – Perl 6 from 30,000 feet

2012-12-17T00:00:25+00:00

Many people have heard of Perl 6, especially in the greater Perl community. However, Perl 6 has a complicated ecosystem which can be a littled daunting, so as a newcomer to the Perl 6 community myself, I thought I would share what I’ve learned.

How do I install Perl 6?

It’s simple; you can just download one of the existing implementations of the language (as Perl 6 is a specification), build it, and install it! There are several implementations out there right now, in various states of completion. Rakudo is an implementation that targets Parrot, and is the implementation that I will discuss most in this post. Niecza is another implementation that targets the CLR (the .NET runtime). For more information on these implementations and on other implementations, please see Perl 6 Compilers. Perl 6 is an ever-evolving language, and any compiler that passes the official test suite can be considered a Perl 6 implementation.

You mentioned “Parrot”; what’s that?

Parrot is a virtual machine that is designed to run dynamically typed languages. Along with the virtual machine, it includes tools for generating virtual machine code from intermediate languages (named PIR and PASM), as well as a suite of tools to make writing compilers easier.

What is Rakudo written in?

Rakudo itself is written primarly in Perl 6, with some bits of C for some of the lower-level operations, like binding method arguments and adding additional opcodes to the Parrot VM. It may seem strange to implement a Perl 6 compiler in Perl 6 itself; Rakudo uses NQP for building itself.

What’s NQP?

NQP (or Not Quite Perl 6) is an implementation of Perl 6 that is focused on creating compilers for the Parrot Compiler Toolkit. It is currently focused on targetting Parrot, but in the future, it may support various compilation targets, so you will be able to use Rakudo to compile your Perl 6 programs to Parrot opcodes, a JVM class file, or perhaps Javascript so you can run it in the browser. NQP is written in NQP, and uses a pre-compiled version of NQP to compile itself.

I hope that this information was useful to you, dear reader, and that it helps to clarify the different pieces of the Perl 6 ecosystem. As I learn more about each piece, I intend to write blog posts that will hopefully help others to get started contributing to Perl 6!

-Rob

Day 16 – Operator precedence

2012-12-16T00:01:04+00:00

All the precedence men

As I was taking a walk today, I realized one of the reasons why I like Perl. Five as well as six. I often hear praise such as “Perl fits the way I think”. And I have that feeling too sometimes.

If I were the president (or prime minister, as I’m Swedish), and had a bunch of advisers, maybe some of them would be yes-men, trying to give me advice that they think I will want to hear, instead of advice that would be useful to me. Some languages are like that, presenting us with an incomplete subset of the necessary tools. The Perl languages, if they were advisers, wouldn’t be yes-men. They’d give me an accurate view of the world, even if that view would be a bit messy and hairy sometimes.

Which, I guess, is why Perl five and six are so often used in handling messy data and turning it into something useful.

To give a few specific examples:

Perl 5 takes quotes and quoting very seriously. Not just strings but lists of strings, too. (See the qw keyword.) Perl 6 does the same, but takes quoting further. See see the recent post on quoting.
jnthn shows in yesterday’s advent post that Perl 6 takes compiler phases seriously, and allows us to bundle together code that belongs together conceptually but not temporally. We need to do this because the world is gnarly and running a program happens in phases.
Grammars in Perl 6 are not just powerful, but in some sense honest, too. They don’t oversimplify the task for the programmer, because then they would also limit the expressibility. Even though grammars are complicated and intricate, they should be, because they describe a process (parsing) that is complicated and intricate.

Operators

Perl is known for its many operators. Some would describe it as an “operator-oriented” language. Where many other language will try to guess how you want your operators to behave on your values, or perhaps demand that you pre-declare all your types so that there’ll be no doubt, Perl 6 carries much of the typing information in its operators:

my $a = 5;
my $b = 6;

say $a + $b;      # 11 (numeric addition)
say $a * $b;      # 30 (numeric multiplication)

say $a ~ $b;      # "56" (string concatenation)
say $a x $b;      # "555555" (string repetition)

say $a || $b;     # 5 (boolean disjunction)
say $a && $b;     # 6 (boolean conjunction)

Other languages will want to bunch together some of these for us, using the + operator for both numeric addition and string concatenation, for example. Not so Perl. You’re meant to choose yourself, because the choice matters. In return, Perl will care a little less about the types of the operands, and just deliver the appropriate result for you.

“The appropriate result” is most often a number if you used a numeric operator, and a string if you used a string operator. But sometimes it’s more subtle than that. Note that the boolean operators above actually preserved the numbers 5 and 6 for us, even though internally it treated them both as true values. In C, if we do the same, C will unhelpfully “flatten” these results down to the value 1, its spelling of the value true. Perl knows that truthiness comes in many flavors, and retains the particular flavor for you.

Operator precedence

“All operators are equal, but some operators are more equal than others.” It is when we combine operators that we realize that the operators have different “tightness”.

say 2 * 3 + 1;      # 7, because (2 * 3) + 1
say 1 + 2 * 3;      # 7, because 1 + (2 * 3), not 9

We can always be 100% explicit and surround enough of our operations with parentheses… but when we don’t, the operators seem to order themselves in some order, which is not just simple left-to-right evaluation. This ordering between operators is what we refer to as “precedence”.

No doubt you were taught in math class in school that multiplications should be evaluated before additions in the way we see above. It’s as if factors group together closer than terms do. The fact that this difference in precedence is useful is backed up by centuries of algebra notation. Most programming languages, Perl 6 included, incorporates this into the language.

By the way, this difference in precedence is found between other pairs of operators, even outside the realm of mathematics:

      Additive (loose)    Multiplicative (tight)
      ================    ======================
number      +                       *
string      ~                       x
bool        ||                      &&

It turns out that they make as much sense for other types as they do for numbers. And group theory bears this out: these other operators can be seen as a kind of addition and multiplication, if we squint.

Operator precedence parser

Deep in the bowels of the Perl 6 parser sits a smaller parser which is very good at parsing expressions. The bigger parser which parses your Perl 6 program is a really good recursive-descent parser. It works great for creating syntax trees out of the larger program structure. It works less well on the level of expressions. Essentially, what trips up a recursive-descent parser is that it always has to create AST nodes for all the possible precedence levels, whether they’re present or not.

So this smaller parser is an operator-table parser. It knows what to do with each type of operator (prefix, infix, postfix…), and kind of weaves all the terms and operators into a syntax tree. Only the precedence levels actually used show up in the tree.

The optable parser works by comparing each new operator to the top operator on a stack of operators. So when it sees an expression like this:

$x ** 2 + 3 * $x - 5

it will first compare ** against + and decide that the former is tighter, and thus $x ** 2 should be put together into a small tree. Later, it compares + against *, and decides to turn 3 * $x into a small tree. It goes on like this, eventually ending up with this tree structure:

infix:<->
 +-- infix:<+>
      +-- infix:<**>
      |    +-- term:<$x>
      |    +-- term:<2>
      +-- infix:<*>
           +-- term:<3>
           +-- term:<$x>

Because leaf nodes are evaluated first and the root node last, this tree structure determines the order of evaluation for the expression. The order ends up being the same as if the expression had these parentheses:

(($x ** 2) + (3 * $x)) - 5

Which, again, is what we’ve learned to expect.

Associativity

Another factor also governs how these invisible parentheses are to be distributed: operator associativity. It’s the concern of how the operator combines with multiple copies of itself, or other sufficiently similar operators on the same precedence level.

Some examples serve to explain the difference:

$x = $y = $z;     # becomes $x = ($y = $z)
$x / $y / $z;     # becomes ($x / $y) / $z

In both of these cases, we may look at the way the parentheses are doled out, and say “well, of course”. Of course we must first assign to $y and only then to $x. And of course we first divide by $y and only then by $z. So operators naturally have different associativity.

The optable parser compares not just the precedence of two operators but also, when needed, their associativity. And it puts the parentheses in the right place, just as above.

User-defined operators

Now we come back to Perl not being a yes-man, and working hard to give you the appropriate tools for the job.

Perl 6 allows you to define operators. See my post from last year on the details of how. But it also allows you to specify precedence and associativity of each new operator.

As you specify a new operator, a new Perl 6 parser is automatically constructed for you behind the scenes, which contains your new operator. In this sense, the optable parser is open and extensible. And Perl 6 gives you exactly the same tools for talking about precedence and associativity as the compiler itself uses internally.

Perl treats you like a grown-up, and expects you to make good decisions based on a thorough understanding of the problem space. I like that.

Day 15 – Phasers set to stun

2012-12-15T13:28:07+00:00

When writing programs, it’s important not only to separate the concerns that need separating, but also to try and keep related things close to each other. This gives the program a sense of cohesion, and helps to avoid the inevitable problems that arise when updating one part of a program demands an update in another far-away part. One especially tricky problem can be when the things we want to do are distributed over time. This can cause us to move related things apart in order to get them to happen at the times we want.

Phasers in Perl 6 help you keep related concepts together in your code, while also indicating that certain aspects of them should happen at different points during the lifetime of the current program, invocation or loop construct. Let’s take a look at some of them.

ENTER and LEAVE

One of the things I had most fun writing in Perl 6 recently was the debugger. There are various things that need a little care. For example, the debugger needs to look out for exceptions and, when they are thrown, give the user a prompt to let them debug why the exception was thrown. However, there is also a feature where, at the prompt, you can evaluate an expression. The debugger shouldn’t re-enter itself if this expression throws, so we need to keep track of if we’re already showing the prompt. This meant setting and clearing a flag. Thing is, the prompt method is relatively lengthy; it has a given/when to identify the various different commands. I could, of course, have set the prompt flag at the start and cleared it at the end. But that would have spread out the concern of maintaining the flag. Here’s what I did instead:

method issue_prompt($ctx, $cur_file) {
    ENTER $in_prompt = True;
    LEAVE $in_prompt = False;

    # Lots of stuff here
}

This ensures the flag is set when we enter the method, cleared when we leave the method – and lets me keep the two together.

INIT and END

We’re writing a small utility and want to log what happens as we run it. Time wise, we want to:

Open the log file at the start of the program, creating it if needed and overwriting an existing one otherwise
Write log entries at various points during the program’s execution
Close the log file at the end

Those three actions are fairly spread out in time, but we’d like to collect them together. This time, the INIT and END phasers come to the rescue.

sub log($msg) {
    my $fh = INIT open("logfile", :w);
    $fh.say($msg);
    END $fh.close;
}

Here, we use INIT to perform an action at program start time. It turns out that INIT also keeps around the value produced by the expression following it, meaning it can be used as an r-value. This means we have the file handle available to us, and can write to it during the program. Then, at the END of the program, we close the file handle. All of these have block forms, should you wish to do something more involved:

sub log($msg) {
    my $fh = INIT open("logfile", :w);
    $fh.say($msg);
    END {
        $fh.say("Ran in {now - INIT now} seconds");
        $fh.close;
    }
}

Note the second use of INIT in this example, to compute and remember the program start time so we can use it in the subtraction later on.

FIRST, NEXT and LAST

These phasers work with loops. They fire the first time the loop body executes, at the end of every loop body execution, and after the last loop body execution. FIRST and LAST are especially powerful in so far as they let us move code that wants to special-case the first and last time the loop body runs inside of the loop construct itself. This makes the relationship between these bits of code and the loop especially clear, and lessens the chance somebody moves or copies the loop and forgets the related bits it has.

As an example, let’s imagine we are rendering a table of scores from a game. We want to write a header row, and also do a little ASCII art to denote the start and end of the table. Furthermore, we’d like to keep track of the best score each time around the loop, and then at the end print out the best score. Here’s how we could write it.

for %scores.kv -> $player, $score {
    FIRST say "Score\tPlayer";
    FIRST say "-----\t------";
    LAST  say "-----\t------";

    NEXT (state $best_score) max= $score;
    LAST say "BEST SCORE: $best_score";

    say "$score\t$player";
}

Notice how we keep the header/footer code together, as well as being able to keep the best score tracking code together. It’s also all inside the loop, making its relationship to the loop clear. Note how the state variable also comes in useful here. It too is a construct that lets us keep a variable scoped inside a block even if its usage spans multiple invocations of the block.

KEEP and UNDO

These are variants of LEAVE that trigger conditional on the block being successful (KEEP) or not (UNDO). A successful block completes without unhandled exceptions and returns a defined value. An unsuccessful block exits due to an exception or because it returns an undefined value. Say we were processing a bunch of files and want to build up arrays of successful files and failed files. We could write something like:

sub process($file) {
    KEEP push @success, $file;
    UNDO push @failure, $file;

    my $fh = open($file);
    # ...
}

There are probably a bunch of transaction-like constructs that can also be very neatly implemented with these two.

And there’s more!

While I’ve covered a bunch of the phasers here, there are some others. For example, there’s also BEGIN, which lets you do some computation at compile time. Hopefully, though, this set of examples gives you some inspiration in how phasers can be used effectively, as well as a better grasp of the motivation for them. Bringing related things together and setting unrelated things apart is something we need to think carefully about every day as developers, and phasers help us keep related concerns together, even if they should take place at different phases of our program’s execution.

Perl 6 changes - 2012W50

2012-12-15T08:00:00+00:00

So, this is yet another article in Perl 6 Changes series. (I know, those introductions are really boring, but would you like to see Lorem Ipsum Dolor instead)? As in previous week, Rakudo mainly improves Perl 6 parsing.

By the way, if you don’t know about Perl 6 Coding Contest, check it out. The challenges are rather easy if you had any programming language experience (if you don’t know Perl 6, check out Using Perl 6 book). If you still have questions, people in #perl6 can help you.

id='incompatible_changes'>Incompatible changes>

Rakudo Perl>

`uniq` has `===` semantics. It could break your programs if you for example were using `uniq` to remove identical arrays (previously they were converted to strings which did what you wanted, as long the strings in array didn’t contained space or you didn’t have array in array.

id='new_features'>New features>

Rakudo Perl>

“Two terms in a row” error message.

Information about what was expected, if possible.

$ perl6 -e '2 2'
===SORRY!===
Two terms in a row
at -e:1
------> 2 ⏏2
    expecting any of:
        postfix
        infix or meta-infix
        infix stopper
        statement end
        statement modifier
        statement modifier loop

for 1..10 { say so 9999 == any(1..2000) } is 30 times faster (huge junctions are faster).
Added Texas versions of the Set and Bag operators, used in Bags and Sets article in Perl 6 Advent Calendar.
Rakudo Perl now informs about obsolete use of . to concatenate strings.
Implemented sub term:<foo>. Now 2 + 2 == 5 can be true for sufficiently huge values of 2 (not like it should be).
qww now works.

Niecza Perl>

`slurp` now works without arguments.

The 2012 Perl 6 Coding Contest

2012-12-14T20:21:30+00:00

For the third year in a row, let's write some Perl 6 contest code!

The 2012 Perl 6 Coding Contest

Perl 6 has had a great year, with its two major implementations growing increasingly mature. Sometimes it's easy to forget all the progress that happens each month. Features and performance have improved all through the year. I think that will show in this year's entries.

So here's what's happening:

do five tasks, and you might win Amazon books worth €100!

Addendum: Thanks to an anonymous donor, we're proud to announce a second prize as well: Amazon books worth 100 USD! (Redeemable within a year of winning.)

The contest starts now, today on 2012-12-14. It ends five weeks later, on 2013-01-18. Registration is open for two weeks, starting now. Just send an email to cmasak@gmail.com — saying "sign me up!" or even sending your Amazon wishlist so I'll know which books to buy you if you win.

You might be curious about what you need to do to win. Here's an extract from the file rules.md:

Since "code quality" is a slightly subjective measure, let us provide a few
hints of what we'll be looking for:

* Correctness.
* Readability.
* Consistency.
* Clarity of intent.
* Algorithmic efficiency.
* Idiomatic use of Perl 6.
* Brevity.

In short, what we're looking for is top-quality code. That's how you win.

Here are the five tasks. Write Perl 6 programs to...

Tell knights from knaves based on what they say. (more)
Generate rectangle haikus. (more)
Arrange wire crossings to rearrange wires. (more)
Simulate rain in a world of cubes. (more)
Distribute weights in bags evenly. (more)

I've chosen the problems so that they're easy to explain, but allow contestants quite a bit of freedom to play around with various solutions.

This time, the whole contest is running as a GitHub repository. Easier than hosting zip files.

Now sign up — you've nothing to lose, and books to win! Also, this is your chance to finally take a look at Perl 6. 哈哈

(Looking for previous years' tasks and solutions? Here they are: 2010 tasks and 2010 solutions. 2011 tasks and 2011 solutions.)

Day 14 – Primal Needs

2012-12-14T20:00:18+00:00

Our brains are hard-wired to look for patterns, even where none exist. So, it’s no surprise that as soon as mankind started counting things, he would look for patterns in numbers. One group of numbers that have resisted the pattern matching capabilities of the human brain are the so-called “prime numbers”. These are numbers that can only be evenly divided by 1 or themselves–they have no other factors.

But you knew that already, so why am I talking about prime numbers instead of Perl 6? Because, just like our ancestors, the people that created Perl 6 and continue to shape it to be around for the next 100 years or more find prime numbers interesting. So interesting, in fact, that the language specification was modified to include a routine for determining whether or not a number is prime.

Alpha

At first, implmementations of this prime-number-finder were pure Perl 6 and took advantage of other features of the language such as ranges and junctions. An example implementation is shown below:

    sub is-prime($n) { $n %% none 2..sqrt $n }

This implementation checks to see that none of numbers from 2 to the square root of $n will evenly divide $n. If this is the case, then the number is prime.

While the above implementation works fine, it is a little slow and it does suffer a little redundancy in the numbers it checks. For instance, if you know a number isn’t evenly divisible by 2, there’s no need to check if it’s evenly divisible by 4, yet the above algorithm does so anyway.

Beta

An improvement on the algorithm is to only check if the I between 2 and the square root of the number evenly divide the number. But … but … that’s like like defining a word in terms of itself. Thanks to ubiquitous lazy evaluation in Perl 6, that’s entirely possible. Here’s an implementation:

    my @primes := 2, 3, 5, -> $p { ($p+2, $p+4 ... &is-prime)[*-1] } ... *;
    sub is-prime($n) { $n %% none @primes ...^  * > sqrt $n }

The array @primes is an infinite, lazily evaluated sequence of numbers starting with 2, 3, and 5. The next number in the sequence is generated by creating a new sequence of odd numbers that start from the last odd number and continue until we reach a prime. That prime is the next number in the sequence. But how do we know if it’s a prime? We check with our handy C subroutine that actually uses the lazy list of primes up to the square root of the number we’re testing to see if any of them are factors.

There’s a kind of mutual recursion going on here where the @primes array effectively memoizes the primes we’ve seen so far. But … then there’s the problem that @primes will continue to grow as you check bigger and bigger numbers for prime-ness. Can we do better?

Indeed we can.

Gamma: Rabin-Miller test

Well … maybe we can. It depends on your idea of “better”. The Rabin-Miller primality test is probabalistic in nature. It doesn’t require storing an ever increasing cache of prime numbers to test if they are factors of the potential prime, but there is a chance that it will tell you that a number is prime when it actually isn’t. The good news is that we can adjust the odds so that we are reasonably confident that the number is prime. Here’s an implementation (taken from http://rosettacode.org/wiki/Miller-Rabin_primality_test#Perl_6):

sub expmod(Int $a is copy, Int $b is copy, $n) {
	my $c = 1;
	repeat while $b div= 2 {
		($c *= $a) %= $n if $b % 2;
		($a *= $a) %= $n;
	}
	$c;
}
 
subset PrimeCandidate of Int where { $_ > 2 and $_ % 2 };
 
my Bool multi sub is-prime(Int $n, Int $k)            { return False; }
my Bool multi sub is-prime(2, Int $k)                 { return True; }
my Bool multi sub is-prime(PrimeCandidate $n, Int $k) {
	my Int $d = $n - 1;
	my Int $s = 0;
 
	while $d %% 2 {(PrimeCandidate $n, Int $k)
		$d div= 2;
		$s++;
	}
 
	for (2 ..^ $n).pick($k) -> $a {
		my $x = expmod($a, $d, $n);
 
		next if $x == 1 or $x == $n - 1;
 
		for 1 ..^ $s {
			$x = $x ** 2 mod $n;
			return False if $x == 1;
			last if $x == $n - 1;
		}
		return False if $x !== $n - 1;
	}
 
	return True;
}

The third multi variant of is-prime with the signature (PrimeCandidate $n, Int $k) is where all of the magic happens. This multi is only triggered when the prime candidate ($n) is an odd number because of the definition of the PrimeCandidate type.

First, we factor out the powers of 2 from $n - 1. Since $n is an odd number, $n - 1 is even and so has at least one factor of 2. What we end up with is an odd number and some power-of-2 factors of $n - 1. We then use those factors to see if a random sample of $k numbers less than $n are congruent to the square roots of unity modulo $n (expmod handles the modular exponentiation). We repeat this for all of the powers of 2 we factored out of the original number. Fermat’s little theorem says that if we find any number where the congruence does not hold, then the number can not be prime.

The probability that this method will select a composite number as prime is based on how many numbers less than $n we choose to sample. If we select $k numbers to try, the probability is 4 ** -$k. By choosing to sample more numbers, we can quickly decrease the odds of a false positive to a negligible amount.

Wrap up

But … most people don’t really have to worry about the implementation details of is-prime. Not only have is-prime and expmod been added to the Perl 6 specification, but actual implementations (ala Rabin-Miller) have been added to the Rakudo and Niecza Perl 6 compilers. So, if you want to test your new cryptographic algorithm and need some large prime numbers, or if you’re developing a new random number generator and need some candidates for the modulus, or maybe you’re developing a new hashing algorithm, Perl 6 has a built-in is-prime that can help.

Day 13 – Bags and Sets

2012-12-13T15:50:59+00:00

Over the years, I’ve written many variations on this code:

my %words;
for slurp.comb(/\w+/).map(*.lc) -> $word {
    %words{$word}++;
}

(Aside: slurp.comb(/\w+/).map(*.lc) does the standard Perl trick of reading files specified on the command line or standard in, goes through the data for words, and makes them lowercase.)

Perl 6 introduces two new Associative types for dealing with this sort of functionality. KeyBag is drop-in replacement for Hash in this sort of case:

my %words := KeyBag.new;
for slurp.comb(/\w+/).map(*.lc) -> $word {
    %words{$word}++;
}

Why would you prefer KeyBag over Hash in this case, considering that it’s a bit more code? Well, it does a better job of saying what you mean, if what you want is a positive Int-valued Hash. It actually enforces this as well:

> %words{"the"} = "green";
Unhandled exception: Cannot parse number: green

That’s Niecza’s error; Rakudo’s is less clear, but the important point is you get an error; Perl 6 detects that you’ve violated your contract and complains.

And KeyBag has a couple more tricks up its sleeve. First, four lines to initialize your KeyBag isn’t terribly verbose, but Perl 6 has no trouble getting it down to one line:

my %words := KeyBag.new(slurp.comb(/\w+/).map(*.lc));

KeyBag.new does its best to turn whatever it is given into the contents of a KeyBag. Given a List, each of the elements is added to the KeyBag, with the exact same result of our earlier block of code.

If you don’t need to modify the bag after its creation, then you can use Bag instead of KeyBag. The difference is Bag is immutable; if %words is a Bag, then %words{$word}++ is illegal. If immutability is okay for your application, then you can make the code even more compact:

my %words := bag slurp.comb(/\w+/).map(*.lc);

bag is a helper sub that just calls Bag.new on whatever you give it. (I’m not sure why there is no equivalent keybag sub.)

Bag and KeyBag have a couple more tricks up their sleeve. They have their own versions of .roll and .pick which weigh their results according to the given values:

> my $bag = bag "red" => 2, "blue" => 10;
> say $bag.roll(10);
> say $bag.pick(*).join(" ");
blue blue blue blue blue blue red blue red blue
blue red blue blue red blue blue blue blue blue blue blue

This wouldn’t be too hard to emulate using a normal Array, but this version would be:

> $bag = bag "red" => 20000000000000000001, "blue" => 100000000000000000000;
> say $bag.roll(10);
> say $bag.pick(10).join(" ");
blue blue blue blue red blue red blue blue blue
blue blue blue red blue blue blue red blue blue

They also work with all the standard Set operators, and have a few of their own as well. Here’s a simple demonstration:

sub MAIN($file1, $file2) {
    my $words1 = bag slurp($file1).comb(/\w+/).map(*.lc);
    my $words2 = set slurp($file2).comb(/\w+/).map(*.lc);
    my $unique = ($words1 (-) $words2);
    for $unique.list.sort({ -$words1{$_} })[^10] -> $word {
        say "$word: { $words1{$word} }";
    }
}

Passed two filenames, this makes a Bag from the words in the first file, a Set from the words in the second file, uses the set difference operator (-) to compute the set of words which are only in the first file, sorts those words by their frequency of appearance, and then prints out the top ten.

This is the perfect point to introduce Set. As you might guess from the above, it works much like Bag. Where Bag is a Hash from Any to positive Int, Set is a Hash from Any to Bool::True. Set is immutable, and there is also a mutable KeySet.

Between Set and Bag we have a very rich collection of operators:

Operation	Unicode	“Texas”	Result Type
is an element of	∈	(elem)	Bool
is not an element of	∉	!(elem)	Bool
contains	∋	(cont)	Bool
does not contain	∌	!(cont)	Bool
union	∪	(\|)	Set or Bag
intersection	∩	(&)	Set or Bag
set difference		(-)	Set
set symmetric difference		(^)	Set
subset	⊆	(<=)	Bool
not a subset	⊈	!(<=)	Bool
proper subset	⊂	(<)	Bool
not a proper subset	⊄	!(<)	Bool
superset	⊇	(>=)	Bool
not a superset	⊉	!(>=)	Bool
proper superset	⊃	(>)	Bool
not a proper superset	⊅	!(>)	Bool
bag multiplication	⊍	(.)	Bag
bag addition	⊎	(+)	Bag

Most of these are self-explanatory. Operators that return Set promote their arguments to Set before doing the operation. Operators that return Bag promote their arguments to Bag before doing the operation. Operators that return Set or Bag promote their arguments to Bag if at least one of them is a Bag or KeyBag, and to Set otherwise; in either case they return the type promoted to.

Please note that while the set operators have been in Niecza for some time, they were only added to Rakudo yesterday, and only in the Texas variations.

A bit of a word may be needed for the different varieties of unions and intersections of Bag. The normal union operator takes the max of the quantities in either bag. The intersection operator takes the min of the quantities in either bag. Bag addition adds the quantities from either bag. Bag multiplication multiplies the quantities from either bag. (There is some question if the last operation is actually useful for anything — if you know of a use for it, please let us know!)

> my $a = bag <a a a b b c>;
> my $b = bag <a b b b>;

> $a (|) $b;
bag("a" => 3, "b" => 3, "c" => 1)

> $a (&) $b;
bag("a" => 1, "b" => 2)

> $a (+) $b;
bag("a" => 4, "b" => 5, "c" => 1)

> $a (.) $b;
bag("a" => 3, "b" => 6)

I’ve placed my full set of examples for this article and several data files to play with on Github. All the sample files should work on the latest very latest Rakudo from Github; I think all but most-common-unique.pl and bag-union-demo.pl should work with the latest proper Rakudo releases. Meanwhile those two scripts will work on Niecza, and with any luck I’ll have the bug stopping the rest of the scripts from working there fixed in the next few hours.

A quick example of getting the 10 most common words in Hamlet which are not found in Much Ado About Nothing:

> perl6 bin/most-common-unique.pl data/Hamlet.txt data/Much_Ado_About_Nothing.txt
ham: 358
queen: 119
hamlet: 118
hor: 111
pol: 86
laer: 62
oph: 58
ros: 53
horatio: 48
clown: 47

Day 12 – Exceptions

2012-12-12T05:12:35+00:00

Sometimes things go horribly wrong, and the only thing you can do is not to go on. Then you throw an exception.

But of course the story doesn’t end there. The caller (or the caller’s caller) must somehow deal with the exception. To do that in a sensible manner, the caller needs to have as much information as possible.

In Perl 6, exceptions should inherit from the type Exception, and by convention they go into the X:: namespace.

So for example if you write a HTTP client library, and you decide that an exception should be thrown when the server returns a status code starting with 4 or 5, you could declare your exception class as

 class X::HTTP is Exception {
     has $.request-method;
     has $.url;
     has $.status;
     has $.error-string;

     method message() {
         "Error during $.request-method request"
         ~ " to $.url: $.status $.error-string";
     }
 }

And throw an exception as

 die X::HTTP.new(
     request-method  => 'GET',
     url             => 'http://example.com/no-such-file',
     status          => 404,
     error-string    => 'Not found',
 );

The error message then looks like this:

 Error during GET request to
    http://example.com/no-such-file: 404 Not found

(line wrapped for the benefit of small browser windows).

If the exception is not caught, the program aborts and prints the error message, as well as a backtrace.

There are two ways to catch exceptions. The simple Pokemon style “gotta catch ‘em all” method catches exception of any type with try:

 my $result = try do-operation-that-might-die();
 if ($!) {
     note "There was an error: $!";
     note "But I'm going to go on anyway";
 }

Or you can selectively catch some exception types and handle only them, and rethrow all other exceptions to the caller:

 my $result =  do-operation-that-might-die();
 CATCH {
     when X::HTTP {
         note "Got an HTTP error for URL $_.url()";
         # do some proper error handling
     }
     # exceptions not of type X::HTTP are rethrown
 }

Note that the CATCH block is inside the same scope as the one where the error might occur, so that by default you have access to all the interesting varibles from that scope, which makes it easy to generate better error messages.

Inside the CATCH block, the exception is available as $_, and is matched against all when blocks.

Even if you don’t need to selectively catch your exceptions, it still makes sense to declare specific classes, because that makes it very easy to write tests that checks for proper error reporting. You can check the type and the payload of the exceptions, without having to resort to checking the exact error message (which is always brittle).

But Perl 6 being Perl, it doesn’t force you to write your own exception types. If you pass a non-Exception objects to die(), it simply wraps them in an object of type X::AdHoc (which in turn inherits from Exception), and makes the argument available with the payload method:

    sub I-am-fatal() {
        die "Neat error message";
    }
    try I-am-fatal();
    say $!;             # Neat error message;
    say $!.perl;        # X::AdHoc.new(payload => "Neat error message")

To find out more about exception handling, you can read the documentation of class Exception and Backtrace.

Day 11 – Parrot threads

2012-12-11T00:08:03+00:00

Editors note: I, rurban, does know almost nothing about threads. Any errors are probably mine. I just tested them, fixed some deadlocks, added the numcpu code and merged the threads branch to master.

Parrot now supports fast and lightweight OS threads, based on Nat Tucks’s initial GSoC work together with Andrew “whiteknight” Whitworth on green threads and finally Stefan Seifert’s extension to true parallel OS threads as hybrid threads.
See http://wknight8111.blogspot.co.at/2010/08/gsoc-threads-chandons-results.html and http://niner.name/Hybrid_Threads_for_the_Parrot_VM.pdf

History

Parrot always supported “threads”, i.e. concurrency models over the last years, but we identified various problems with the particular designs and were continuously improving them. In our case without changing the API too much, as the pdd25 concurrency spec is pretty high level, describing the various models parrot should support, and also pretty low-level in describing the two PMC’s which export the threads API, the Task and the Scheduler classes.

Being born at a time when Perl 6 still looked much more similar to Perl 5 than it does nowadays, Parrot’s threading support initially was very close to Perl’s ithreads model. Previous attempts to change this into the more conventional model of data shared by default or implementing new technologies like STM “Software Transactional Memory” failed. For example Parrot has never supported running multiple threads and having garbage collection at the same time.

In the year 2005 development of faster Central Processing Units (CPUs) shifted from increased speed of a single core to adding more cores. Modern processors contain up to 12 cores with even mobile phones having up to four. To utilize a modern CPU’s power, code needs to be run in parallel. In UNIX (and thus Perl) tradition, this is accomplished using multiple processes being a good solution for many use cases. For many others like auto threading of hyper operators in Perl 6, the cost of process setup and communication would be prohibitively high except for very large data sets.

During the years of back and forth and failed attempts of adding threading support to Parrot, the Perl 6 specification evolved to a point where the largest parts of the language were covered and its features were implemented in the compilers. The lack of concurrency primitives in Parrot however prevents any progress in the area of concurrency support.

Summary

Green threads were used to simplify the implementation of a nearly lock free multithreading implementation.

Parrot supports now native Win32 threads and POSIX threads. Win32 alarm, sleep and premption is unified with POSIX, it is handled on a common timer thread.

Parrot creates at startup a thread pool of --numthreads threads, which defaults to the number of available CPU cores. Activating a new thread at runtime causes no run-time penalties, until the number of cores is utilized. When a user starts a new task, the scheduler first looks for an idle thread. If one can be found, the task is scheduled on the thread’s interpreter. If more tasks are started than the maximum number of threads, the tasks are distributed evenly among the running interpreters. This is effectively an implementation of the N:M threading model.

Green threads

Our GSOC student Nan “Chandor” Tuck worked in summer 2010 on green threads.

What I have working now is a pre-emptively scheduled green threads system for Parrot that allows programs to be written in a concurrent style. Individual green threads can do basic blocking file input without stopping other threads from running. These logical threads are accessed using the Task API that I described a couple weeks ago. This functionality makes Parrot similarly powerful at threading as the standard version of Ruby or Python: Threads can do pretty much everything except run at the same time. http://parrot.org/content/hybrid-threads-gsoc-project-results

What was missing from this green threads branch was true parallel execution in OS threads, one global_interpreter structure that is shared and protected by locks or other concurrent access rules and many local_interpreters that run simultaneously in separate OS threads.

OS threads

From Fall 2011 to Summer 2012 Stefan “nine” Seifert implemented true OS threads on top of green threads to finally allow true parallel execution of Tasks, to implement blocking IO, and to give perl6 some more advantages over perl5.

The lightweight “green” threads are used as messages in a system where reading shared variables is allowed but only the one owner thread may write to it. That’s why we call it hybrid threads.

Why is multithreading support so difficult to implement?

Low level programming languages like C provide only the bare necessities, leaving the responsibility for preventing data corruption and synchronization entirely to the user. A high-level language like Perl 6 on the other hand provides complex and compound data types, handles garbage collection and a very dynamic object system. Even seemingly simple things like a method call can become very complex. In a statically typed programming language the definition of a class is immutable. Thus, calling a method on an object contains just the steps of determining the object’s class, fetching the required method from this class and calling it. Calling the same method again can then even omit the first two steps since their results cannot change.

In a dynamic language, the object may change its class at runtime. The inheritance hierarchy of the class may be changed by adding or removing parent classes. Methods may be added to or removed from classes (or objects) at runtime and even the way how to find a method of a class may change. So a simple method call results in the following steps:

    ·  determining the class of the object,
    ·  determining the method resolution method of the class,
    ·  finding the actual method to call,
    ·  calling the method.

These steps have to be repeated for every method call, because the results may change any time. In a threaded environment, a thread running in parallel may change the underlying data and meta data in between those sequences and even between those steps. As a consequence, this meta data has to be protected from corruption introducing the need for locks in a performance critical area.

Many interpreters for dynamic languages like Python or Ruby handle this problem by using a global interpreter lock (GIL) to effectively serialize all operations. This is a proven and reliable way but leaves much of the hardware’s potential unused.

Java

In Java, the user is responsible for preventing concurrency issues. The language provides synchronization primitives like mutexes, but the interpreter (the Java Virtual Machine, JVM) does not protect the consistency of the provided data structures. The class library provides the user with high-level data structures explicitly designed for multithreaded scenarios.

Java version 1.1 used green threads to support multithreaded execution of Java programs. Green threads are threads simulated by the virtual machine (VM) but unable to use more than one CPU core for processing. Version 1.2 introduced native Operating System (OS) threading support which since has become the standard way to do multithreading in Java.

Python

The CPython implementation of the Python runtime uses a Global Interpreter Lock (GIL) to protect its internal consistency. This is a single lock taken whenever the interpreter executes Python bytecode. Because of this lock, only one thread can execute bytecode at any time so all built-in types and the object model are implicitly type safe. The drawback is that Python code cannot benefit from having multiple CPU cores available. However I/O operations and calls to external libraries are executed without holding the GIL, so in applications with multiple I/O bounded threads, there may still be a performance benefit from using multithreading.

To run Python code in parallel, multiple processes have to be used. The multiprocessing module provides support for spawning processes exposed through an API similar to the threading module. Since processes may not directly access other processes’ memory, the multiprocessing module provides several means of communication between processes: Queues, Pipes and shared memory support.

Parrot

Much of Parrot’s previous threading related code has been removed to clean up the code and improve performance. Since the existing threading support was known to be unreliable and seriously flawed, this was no trade off. The final parts were removed by the merging of the kill_threads branch on September, 21st 2011.

In 2010, Nat Tuck began working on a green_threads branch during his Google Summer of Code internship. The feature got prototyped using pure PIR and then implemented in Parrot’s core. He got it to work in simple cases and started to work on OS thread support but the internship ended before the code was ready to be merged into the master branch. The code lay dormant until the work on hybrid threads in the threads branch started in 2011.

In Parrot, green threads are called Tasks. Each task is assigned a fixed amount of execution time. After this time is up a timer callback sets a flag which is checked at execution of every branch operation. Since the interpreter’s state is well defined at this point, its internal consistency is guaranteed. The same holds for the GC. Since task preemption is only done while executing user-level code, the GC can do its work undisturbed and without the need for measures like locking. Since user-level code is allowed to disable the scheduler, it can be guaranteed to run undisturbed through critical sections.

The scheduler is implemented as a PMC type. This allows the user to subclass this PMC thus allowing fine-grained control over the scheduling policy. Features, a user could add this way would be for example giving different priorities to tasks or implementing the possibility to suspend and resume a task.

Shared data

Cross-thread writes to shared variables may endanger the internal consistency of the interpreter. Traditionally, the solution to this problem is the use of locks of varying granularity. Fine-grained locking allows code to run in parallel but taking and releasing locks costs performance. It not only increases the instruction count and memory accesses but it also forces the CPU cores to coordinate and thus communicate. Even a seemingly simple operation like an atomic increment can take two orders of magnitude longer than a normal increment. While the gain through being able to utilize multiple CPU cores may offset this cost, it is still impacting the common case of having only a single thread running.

Too coarse locking on the other hand would reduce scalability and the performance gains through parallel execution by having threads wait for extended periods for locks to become available. In the extreme case of having a global interpreter lock it would effectively serialize all computations costing much of the benefits of using threads in the first place.

The other problem with locking is the possibility of introducing deadlocks. For example, two functions F1 and F2 both use two resources A and B protected by locks. If F1 first locks A and then tries to lock B while F2 has already locked B and is now trying to lock A, the program would come to a halt. Both functions would be left waiting for the other to unlock the resource which will never happen. With fine-grained locking, the possibilities for such bugs grow quickly. At the same time, it is easy to miss a case where a lock would be appropriate leading to difficult to diagnose corruption bugs.

The solution for these problems implemented in hybrid threads is to sidestep them altogether by disallowing write access to shared variables. The programmer (or in most cases the compiler) is obliged to declare a list of all shared variables before a newly created task is started. The interpreter would then create proxy objects for these variables which the task can use to access the data. These proxies contain references to the original objects. They use these references to forward all reading vtable functions to the originals. Write access on the other hand would lead to a runtime error.

In other words, all data is owned by the thread creating it and only the owner may write to it. Other threads have only read access.

For threads to be able to communicate with their creators and other threads, they still need to write to shared variables. This is where green threads come into play. Since green threads are light weight, it is feasible for a thread to create a task just for updating a variable. This task is scheduled on the interpreter owning this variable. To reduce latency, the task is flagged to run immediately. The data-owning interpreter will preempt the currently running task and process the new write task. Put another way, the data-owning interpreter is told what to write to its variables, so other threads don’t have to.

Proxies

Proxies are the arbiters between threads. They are the only means for a thread to access another thread’s data and are implemented by the Proxy PMC type.

Proxy has default implementations for all functions, writing functions raise a cant_do_write_method runtime exception. If a method returns a PMC from the target’s interp, another proxy object has to be created and wrapped around it so it can be safely returned to the caller.

Sub

The Sub PMC represents executable subroutines. A Sub does not only contain the code to execute but also the context in which to execute the code such as visible globals and namespaces. If a proxy to such a Sub were created and invoke called on it, the code would access this context directly since it belongs to the same interp as the proxied Sub itself. Thus, an operation like get_global fetches a global from an unproxied namespace and an unproxied global is be put into the target register. Since this is happening while running invoke on the original Sub, Proxy cannot intercept the call and create a Proxy for the result.

This is the reason why Parrot_thread_create_proxy does not create a Proxy for a Sub but uses Parrot_thread_create_local_sub to create a copy on the thread’s interp with proxies for all PMC attributes.

Writing to shared variables

To write to shared variables, a thread creates a task and schedules it on the data owning interpreter. An example task looks like this:

    .sub write_to_variable
         .param pmc variable
         variable = 1
    .end

This is a subroutine with just one parameter. The variable passed as this parameter is the one the task should write to. In this case the constant value 1 would be written to the variable. In PIR, an assignment to a PMC gets translated to a method call. In this case, the set_integer_native is called changing the variable’s value. Since PMCs are passed by reference, it is the original variable which gets written to.

Code to create the task looks like:

    1    write_task = new ['Task']
    2    setattribute write_task, 'code', write_to_variable
    3    setattribute write_task, 'data', shared_variable
    4    interp.'schedule_proxied'(write_task, shared_variable)

Line 1 creates a new task object. The example subroutine is used for the task’s code attribute. shared_variable is used for data. At this point, shared_variable is actually the proxy object created for the shared integer PMC. The interpreter object contains a schedule_proxied method which is used to schedule the write_task on the thread owning the original variable.

schedule_proxied uses Parrot_thread_create_local_task which in this case detects that the data given as parameter for the task’s code is actually a proxy already and unwraps the proxied object. Parrot_cx_schedule_immediate is then used to make the data owning interpreter execute the task as soon as possible.

To protect a critical section, preemption can be disabled so the critical section runs uninterrupted:

    1 .sub swap_variables
    2     .param pmc a, b
    3     .local temp
    4     disable_preemption
    5     temp = a
    6     a = b
    7     b = temp
    8     enable_preemption
    9 .end

wait

Using tasks to write to shared variables makes such actions inherently asynchronous. This is not always what is needed by the implemented algorithm. For example, when the shared variable is a lock, processing should continue as soon as it’s acquired. The wait operation is used to wait for a task’s completion. The waiting task is added to the waited for task’s waiters list and preempted immediately. When a task finishes, all the tasks in the waiters list are scheduled again for execution. Since for each task a local copy is created on the target thread, the running task not only checks its own waiters list but also its partner’s.

If a task on the main thread was waiting for a task on another thread to finish and no other tasks are in the scheduler’s queue on the main thread, the main thread exits if no alarms are pending. To prevent this unintended exit, all tasks are added to the scheduler’s foreign_tasks list when they are scheduled on other threads. To end the program with other threads still running, an explicit exit operation has to be used.

Benchmarks

Preliminary benchmarks have shown Parrot’s performance to be within an order of magnitude of that of an optimized implementation in Perl 5.

Since Parrot does not yet offer the user any synchronization primitives, locks had to be implemented using a shared variable which is written to only by the main thread. Replacing this primitive method with a native semaphore implementation would probably reduce runtime to a small fraction.

Runtime comparison for matrix multiplication

                singlethreaded  computation      multithreaded   computation
    1. run          28.522 s       19.530 s        17.543 s          8.478 s
    2. run          28.427 s       19.463 s        17.320 s          8.283 s
    3. run          28.200 s       19.235 s        17.489 s          8.473 s
    average         28.383 s       19.409 s        17.451 s          8.411 s

This test implements matrix multiplication using four threads. For simplicity the second matrix only has one column. The program is written in the Winxed programming language. Winxed is a low-level language with Javascript like syntax and the possibility to include sections of PIR code verbatim making it possible to try experimental opcodes while writing more readable and concise code than with PIR alone. The complete source code is available in examples/threads/matrix_part.winxed

The program consists of the parts initialization, computation and verification. Computation is parallelized using four tasks each calculating one fourth of the result vector. Runtime is compared to a simple singlethreaded implementation. Run times were measured using the time command and are recorded in the above table.

As can be seen, the multithreaded implementation gives an average speedup of 2.31 for the computation and 1.61 in total.

Runtime comparison for Mandelbrot set calculation

                 singlethreaded  1 thread    2 threads   4 threads    8 threads
    1. run           89.931 s    89.978 s    45.813 s     24.028 s     17.445 s
    2. run           89.707 s    89.871 s    45.906 s     24.048 s     17.695 s
    3. run           90.318 s    89.839 s    45.951 s     24.049 s     17.573 s
    average          89.985 s    89.896 s    45.890 s     24.042 s     17.571 s
    speedup           1.000        1.001       1.959       3.739        5.116

The complete source code is available in examples/pir/mandel.pir

Calculating an image of the Mandelbrot set is a common benchmark for multithreading implementations since calculations of points are independent of each other and are thus easily parallelizable. A simple implementation of the escape time algorithm written in Winxed has been used to determine scalability properties of the threading implementation. The image is split into lines which are calculated alternatedly by a configured number of tasks. Run times were measured using the time command on an Intel Core i7 3770K processor with 16 GiB RAM running openSUSE 12.1 and are recorded in the Mandelbrot table. As can be seen, the implementation scales nearly linearly up to four threads reflecting the CPU’s four physical cores. Using eight threads, the speedup is only 1.368 compared to four threads but this seems to be more a limitation of the hardware than the implementation.

Questions

On IRC and on the mailing list some detailed questions have been asked.

See here:
http://lists.parrot.org/pipermail/parrot-dev/2012-December/007295.html

Day 9 – Longest Token Matching

2012-12-09T00:01:19+00:00

Perl 6 regular expressions prefer to match the longest alternative when possible.

say "food and drink" ~~ / foo | food /;   # food

This is in contrast to Perl 5, which would prefer the first alternative above, and produce the match “foo”.

You can still get the first-alternative behavior if you want; it’s tucked away in the slightly longer alternation operator ||:

say "food and drink" ~~ / foo || food /;  # foo

…And that’s it! That’s Longest Token Matching. ☺ Short post.

“Huh, wait!” I hear you exclaim, in a desperate attempt to make the daily Perl 6 Advent goodness last a bit longer. “Why is Longest Token Matching such a big deal? Who would ever be so obsessed with long tokens?”

I’m glad you asked. As it turns out, Longest Token Matching (or LTM for short) plays very well with our intuition about how things should be parsed. If you’re creating a language, you want people to be able to declare a variable forest_density without the mention of this variable clashing with the syntax of for loops. LTM will see to that.

I like “strange consistencies” — when distal parts of a language design turn out to have commonalities that make the language feel more uniform. There is that kind of consistency here, between classes and grammars. Perl 6 basically exploits that consistency to the max. Let me briefly map out what I mean.

We’re all used to writing classes at this point. From a birds-eye view, they look like this:

class {
    method
    method
    method
}

Grammars have a suspiciously similar structure:

grammar {
    rule
    rule
    rule
}

(The keywords are actually regex, token and rule, but when we talk about them as a group, we just call them “rules”.)

We’re also used to being able to derive classes into subclasses (class B is A), and add or override methods in a way which produces a nice mix of old and new behavior. Perl 6 provides multi methods which even allow you to add new methods of the same name, and the old ones won’t be overridden, they’ll just all try to match alongside the new methods. The dispatch is handled by a (usually autogenerated) proto method that dispatches to all eligible candidates.

What does all this have to do with grammars and rules? Well, it turns out that first off, you can derive new grammars from old ones. It works the same as deriving classes. (In fact, under the hood it’s exactly the same mechanism. Grammars are classes with a different metaclass object.) New rules will override old rules just like you’d expect with methods.

S05 has a cute example with parsing of letters, and deriving the grammar to parse formal letters:

     grammar Letter {
         rule text     {    }
         rule greet { [Hi|Hey|Yo] $=(\S+?) , $$}
         rule body     { +? }   # note: backtracks forwards via +?
         rule close { Later dude, $=(.+) }
     }

     grammar FormalLetter is Letter {
         rule greet { Dear $=(\S+?) , $$}
         rule close { Yours sincerely, $=(.+) }
     }

The derived FormalLetter overrides greet and close, but not body.

But what about all the goodness with multi methods? Could we define some kind of “proto rule” that would allow us to have several rules in a grammar with the same name but different bodies? For example, we might want to parse a language with a rule term, but there are many different terms: strings, numbers… and maybe the numbers can be decimal or binary or octal or hexadecimal…

Perl 6 grammars can contain a proto rule, and then you can define and redefine a rule with the same name as many times as you want. And now we’re back full circle with the / foo | food / alternation from the start of the article. All those rules you write with the same name compile down to one big alternation. Not only that — rules which call other rules, some of them possibly proto rules, all of that will be “flattened” out into one big LTM alternation. In practice that means that all the possible things a term can be are tried out all at once, on equal footing. Neither alternative wins because you happened to define it before the others. An alternative wins because it is the longest.

The strange consistency resides in the fact that in the call-a-method side of things, the most specific method wins, and “most specific” has to with signature narrowness. The better the types in the signature describe the arguments coming in, the more specific the method.

In the parse-with-a-rule side of things, the most specific rule wins, but here “most specific” has to do with parse success. The better the rule can describe what comes next in the text, the more specific the rule.

And that’s strangely consistent, because on the surface methods and rules look like quite different beasts.

We really believe we have something going with this whole principle of deriving a grammar and getting a new language. LTM is right at the center of that because it allows new rules and old to intermix in a fair and predictable way. It’s a kind of meritocracy: rules win not based on whether they’re young or old, but based on whether they are able to parse the text well.

In fact, the Perl 6 compiler itself works this way. It parses your program using a Perl 6 grammar, and that grammar is derivable… whenever you declare a new operator in your program, a new grammar is derived for you. The parsing of your operator is added as a new rule in the new grammar, and the new grammar is given the task of parsing the rest of your program. Your new operator will win against similar but shorter ones, and lose against similar but longer ones.

Day 8 – Panda package manager

2012-12-08T12:49:03+00:00

Perl 6 is not just the language. While without modules it can do more than Perl 5, modules can make life easier. About two years ago neutro was discussed on this blog. I’m not going to talk about it, as it’s deprecated today.

Today, the standard way of installing modules is the panda utility. If you’re using Rakudo Star, you should have it already installed (try the panda command in console to check it). After running it and waiting a few seconds, you should see help for the panda utility.

$ panda
Usage:
  panda [--notests] [--nodeps] install [ ...] -- Install the specified modules
  panda [--installed] [--verbose] list -- List all available modules
  panda update -- Update the module database
  panda info [ ...] -- Display information about specified modules
  panda search  -- Search the name/description

As you can see, it doesn’t have many options (it’s actually similar to RubyGems or cpanminus in its simplicity). You can see the current list of modules at Perl 6 Modules page. Let’s say you would want to parse an INI file. First, you can find module for it using the search command.

$ panda search INI
JSON::Tiny               *          A minimal JSON (de)serializer
Config::INI              *          .ini file parser and writer module for
                                    Perl 6
MiniDBI                  *          a subset of Perl 5 DBI ported to Perl 6
                                    to use while experts build the Real Deal
Class::Utils             0.1.0      Small utilities to help with defining
                                    classes

Config::INI is module you want. Other modules were found because my query wasn’t specific enough and found “ini” in other words (minimal, MiniDBI and defining). Config::INI isn’t part of Rakudo Star, so you have to install it.

Panda installs modules globally when you writing access to the installation directory, locally otherwise. Because of that you can use panda even when Perl 6 is installed globally without installing modules like local::lib, like you have to in Perl 5.

$ panda install Config::INI
==> Fetching Config::INI
==> Building Config::INI
Compiling lib/Config/INI.pm
Compiling lib/Config/INI/Writer.pm
==> Testing Config::INI
t/00-load.t .... ok
t/01-parser.t .. ok
t/02-writer.t .. ok
All tests successful.
Files=3, Tests=55, 3 wallclock secs ( 0.04 usr 0.00 sys + 2.38 cusr 0.14 csys = 2.56 CPU)
Result: PASS
==> Installing Config::INI
==> Succesfully installed Config::INI

After the module has been installed, you can update it as easily – by installing it. Currently panda cannot automatically upgrade modules, but after a module has been updated (you can watch repositories on GitHub to know when it happens – every module is available on GitHub), you can easily upgrade it by reinstalling the module.

When a module was installed, you can check if it works by trying to use it. This is a sample script that can be used to convert INI file into a Perl 6 data structure.

#!/usr/bin/env perl6
use Config::INI;
multi sub MAIN($file) {
    say '# your INI file as seen by Perl 6';
    say Config::INI::parse_file($file).perl;
}

Perl 6 changes - 2012W49

2012-12-08T08:00:00+00:00

To begin with, I’ve made a post for Perl 6 Advent Calendar. If you were wondering why it was late, I will tell you - it’s actually early. Somebody else has claimed article for eighth day and forgot to upload it, so somebody had to put replacement article. I’ve moved my article from 12th day to 8th day, just so Perl 6 Advent Calendar could be continued. Moritz, to protect against that, put the just-in-case article about exceptions (it’s exceptional), if that would happen again.

Enough talking about Perl 6 Advent Calendar, as obviously this blog post isn’t about it. Every week, I describe changes in Perl 6 and its implementations. This is yet another article about changes.

Jonathan Worthington lately was working on improving syntax errors messages in Rakudo Perl. Now they are better and more accurate, just like they’re already in STD.pm6 or Niecza.

id='new_features'>New features>

Rakudo Perl>

Syntax errors now provide context using ⏏ symbol, just like STD.pm6 does.

Rakudo is now more accurate about error locations and doesn’t inform about errors on line 1 (like missing semicolon) that actually are on line 777.

Syntax errors now have colors.

Niecza Perl>

A `.join` method isn’t O(N⁴) anymore.

A `~` infix operator is faster.

Numeric ranges are faster.

Day 7 – MIME::Base64 – On encoded strings

2012-12-07T06:26:11+00:00

parrot MIME::Base64 FixedIntegerArray: index out of bounds!

Ronaldxs created the following parrot ticket #813 4 months ago:

“Was playing with p6 MIME::Base64 and utf8 sampler page when I came across this. It seems that the parrot MIME Base64 library can’t handle some UTF-8 characters as demonstrated below.”

.sub go :main
    load_bytecode 'MIME/Base64.pbc'

    .local pmc enc_sub
    enc_sub = get_global [ "MIME"; "Base64" ], 'encode_base64'

    .local string result_encode
    result_encode = enc_sub(utf8:"\x{203e}")

    say result_encode
.end

FixedIntegerArray: index out of bounds!
current instr.: 'parrot;MIME;Base64;encode_base64' pc 163 (runtime/parrot/library/MIME/Base64.pir:147)
called from Sub 'go' pc 11 (die_utf8_base64.pir:8)

This was interesting, because parrot strings store the encoding information in the string. The user does not need to store the string encoding information somewhere else as in perl5, nor have to do educated guesses about the encoding. parrot supports ascii, latin1, binary, utf-8, ucs-2, utf-16 and ucs-4 string encodings natively.
So we thought we the hell cannot parrot handle simple utf-8 encoded strings?

As it turned out, the parrot implementation of MIME::Base64, which can be shared to all languages which use parrot as VM, stored the character codepoints for each character as array of integers. On multibyte encodings such as UTF-8 this leads to different data held in memory than a normal multibyte string which is encoded as the byte buffer and the additional encoding information.

Internal string representations

For example an overview of different internal string representations for the utf-8 string “\x{203e}”:

perl5 strings:

len=3, utf-8 flag, "\342\200\276" buf=[e2 80 be]

parrot strings:

len=1, bufused=3, encoding=utf-8, buf=[e2 80 be]

The Unicode tables:

U+203E	‾	e2 80 be	OVERLINE

gdb perl5

Let’s check it out:

$ gdb --args perl -e'print "\x{203e}"'
(gdb) start
(gdb) b Perl_pp_print
(gdb) c
(gdb) n 

.. until if (!do_print(*MARK, fp))

(gdb) p **MARK
$1 = {sv_any = 0x404280, sv_refcnt = 1, sv_flags = 671106052, sv_u = {
      svu_pv = 0x426dd0 "‾", svu_iv = 4353488, svu_uv = 4353488, 
      svu_rv = 0x426dd0, svu_array = 0x426dd0, svu_hash = 0x426dd0, 
      svu_gp = 0x426dd0, svu_fp = 0x426dd0}, ...}

(gdb) p Perl_sv_dump(*MARK)
ALLOCATED at -e:1 for stringify (parent 0x0); serial 301
SV = PV(0x404280) at 0x4239a8
  REFCNT = 1
  FLAGS = (POK,READONLY,pPOK,UTF8)
  PV = 0x426dd0 "\342\200\276" [UTF8 "\x{203e}"]
  CUR = 3
  LEN = 16
$2 = void

(gdb) x/3x 0x426dd0
0x426dd0:	0xe2	0x80	0xbe

We see that perl5 does store the utf-8 flag, but not the length of the string, the utf8 length (=1), only the length of the buffer (=3).
Any other multi-byte encoded string, such as UCS-2 is stored differently. We suppose as utf-8.

We are already in the debugger, so let’s try the different cmdline argument.

(gdb) run -e'use Encode; print encode("UCS-2", "\x{203e}")'
  The program being debugged has been started already.
  Start it from the beginning? (y or n) y
Breakpoint 2, Perl_pp_print () at pp_hot.c:712
712	    dVAR; dSP; dMARK; dORIGMARK;

(gdb) p **MARK

$3 = {sv_any = 0x404b30, sv_refcnt = 1, sv_flags = 541700, sv_u = {
    svu_pv = 0x563a50 " >", svu_iv = 5651024, svu_uv = 5651024, 
    svu_rv = 0x563a50, svu_array = 0x563a50, svu_hash = 0x563a50, svu_gp = 0x563a50, 
    svu_fp = 0x563a50}, ...}

(gdb) p Perl_sv_dump(*MARK)
ALLOCATED at -e:1 by return (parent 0x0); serial 9579
SV = PV(0x404b30) at 0x556fb8
  REFCNT = 1
  FLAGS = (TEMP,POK,pPOK)
  PV = 0x563a50 " >"
  CUR = 2
  LEN = 16
$4 = void

(gdb) x/2x 0x563a50
0x563a50:	0x20	0x3e

But we don’t see the UTF8 flag in encode(“UCS-2″, “\x{203e}”), just the simple ascii string ” >”, which is the UCS-2 representation of [20 3e].
Because ” >” is perfectly representable as non-utf8 ASCII string.
UCS-2 is much much nicer than UTF-8, is has a fixed size, it is readable, Windows uses it, but it cannot represent all Unicode characters.

Encode::Unicode contains this nice cheatsheet:

Quick Reference
                Decodes from ord(N)           Encodes chr(N) to...
       octet/char BOM S.P d800-dfff  ord > 0xffff     \x{1abcd} ==
  ---------------+-----------------+------------------------------
  UCS-2BE       2   N   N  is bogus                  Not Available
  UCS-2LE       2   N   N     bogus                  Not Available
  UTF-16      2/4   Y   Y  is   S.P           S.P            BE/LE
  UTF-16BE    2/4   N   Y       S.P           S.P    0xd82a,0xdfcd
  UTF-16LE    2/4   N   Y       S.P           S.P    0x2ad8,0xcddf
  UTF-32        4   Y   -  is bogus         As is            BE/LE
  UTF-32BE      4   N   -     bogus         As is       0x0001abcd
  UTF-32LE      4   N   -     bogus         As is       0xcdab0100
  UTF-8       1-4   -   -     bogus   >= 4 octets   \xf0\x9a\af\8d
  ---------------+-----------------+------------------------------

gdb parrot

Back to parrot:

If you debug parrot with gdb you get a gdb pretty-printer thanks to Nolan Lum, which displays the string and encoding information automatically.
In perl5 you have to call Perl_sv_dump with or without the my_perl as first argument, if threaded or not. With a threaded perl, e.g. on Windows you’d need to call p Perl_sv_dump(my_perl, *MARK).
In parrot you just ask for the value and the formatting is done with a gdb pretty-printer plugin.
The string length is called strlen (of the encoded string), the buffer size is called bufused.

Even in a backtrace the string arguments are displayed abbrevated like this:

#3  0x00007ffff7c29fc4 in utf8_iter_get_and_advance (interp=0x412050, str="utf8:� [1/2]", 
    i=0x7fffffffdd00) at src/string/encoding/utf8.c:551
#4  0x00007ffff7a440f6 in Parrot_str_escape_truncate (interp=0x412050, src="utf8:� [1/2]",
    limit=20) at src/string/api.c:2492
#5  0x00007ffff7b02fb3 in trace_op_dump (interp=0x412050, code_start=0x63a1c0, pc=0x63b688)
    at src/runcore/trace.c:450

[1/2] means strlen=1 bufused=2
Each non-ascii or non latin-1 encoded string is printed with the encoding prefix.
Internally the encoding is of course a index or pointer in the table of supported encodings.

You can set a breakpoint to utf8_iter_get_and_advance and watch the strings.

(gdb) r t/library/mime_base64u.t
Breakpoint 1, utf8_iter_get_and_advance (interp=0x412050, str="utf8:\\x{00c7} [8/8]", 
                i=0x7fffffffcd40) at src/string/encoding/utf8.c:544
(gdb) p str
$1 = "utf8:\\x{00c7} [8/8]"
(gdb) p str->bufused 
$3 = 8
(gdb) p str->strlen
$4 = 8
(gdb) p str->strstart
$5 = 0x5102d7 "\\x{00c7}"

This is escaped. Let’s advance to a more interesting utf8 string in this test, i.e. until str=”utf8:Ā [1/2]“
You get the members of a struct with tab-completion, i.e. press <TAB> after p str->

(gdb) p str->
_buflen    _bufstart  bufused    encoding   flags      hashval    strlen     strstart
(gdb) p str->strlen
$9 = 8

(gdb) dis 1
(gdb) b utf8_iter_get_and_advance if str->strlen == 1
(gdb) c
Breakpoint 2, utf8_iter_get_and_advance (interp=0x412050, str="utf8:Ā [1/2]", 
                i=0x7fffffffcd10) at src/string/encoding/utf8.c:544
544	    ASSERT_ARGS(utf8_iter_get_and_advance)

(gdb) p str->strlen
$10 = 1
(gdb) p str->strstart
$11 = 0x7ffff7faeb58 "Ā"
(gdb) x/2x str->strstart
0x7ffff7faeb58:	0xc4	0x80
(gdb) p str->encoding
$12 = (const struct _str_vtable *) 0x7ffff7d882e0
(gdb) p *str->encoding

$13 = {num = 3, name = 0x7ffff7ce333f "utf8", name_str = "utf8", bytes_per_unit = 1,
  max_bytes_per_codepoint = 4, to_encoding = 0x7ffff7c292b0 <utf8_to_encoding>, chr =
  0x7ffff7c275c0 <unicode_chr>, equal = 0x7ffff7c252e0 <encoding_equal>, compare =
  0x7ffff7c254e0 <encoding_compare>, index = 0x7ffff7c25690 <encoding_index>, rindex
  = 0x7ffff7c257a0 <encoding_rindex>, hash = 0x7ffff7c25a20 <encoding_hash>, scan =
  0x7ffff7c29380 <utf8_scan>, partial_scan = 0x7ffff7c29460 <utf8_partial_scan>, ord
  = 0x7ffff7c297e0 <utf8_ord>, substr = 0x7ffff7c25de0 <encoding_substr>, is_cclass =
  0x7ffff7c26000 <encoding_is_cclass>, find_cclass =
  0x7ffff7c260e0 <encoding_find_cclass>, find_not_cclass =
  0x7ffff7c26220 <encoding_find_not_cclass>, get_graphemes =
  0x7ffff7c263d0 <encoding_get_graphemes>, compose =
  0x7ffff7c27680 <unicode_compose>, decompose = 0x7ffff7c26450 <encoding_decompose>,
  upcase = 0x7ffff7c27b20 <unicode_upcase>, downcase =
  0x7ffff7c27be0 <unicode_downcase>, titlecase = 0x7ffff7c27ca0 <unicode_titlecase>,
  upcase_first = 0x7ffff7c27d60 <unicode_upcase_first>, downcase_first =
  0x7ffff7c27dc0 <unicode_downcase_first>, titlecase_first =
  0x7ffff7c27e20 <unicode_titlecase_first>, iter_get =
  0x7ffff7c29c40 <utf8_iter_get>, iter_skip = 0x7ffff7c29d60 <utf8_iter_skip>,
  iter_get_and_advance = 0x7ffff7c29eb0 <utf8_iter_get_and_advance>,
  iter_set_and_advance = 0x7ffff7c29fd0 <utf8_iter_set_and_advance>}

encode_base64(str)

$ perl -MMIME::Base64 -lE'$x="20e3";$s="\x{20e3}";
  printf "0x%s\t%s=> %s",$x,$s,encode_base64($s)'
Wide character in subroutine entry at -e line 1.

Oops, I’m clearly a unicode perl5 newbie. Does my term not understand utf-8?

$ echo $TERM
xterm

No, it should. encode_base64 does not understand unicode.
perldoc MIME::Base64
“The base64 encoding is only defined for single-byte characters. Use the Encode module to select the byte encoding you want.”

Oh my! But it is just perl5. It just works on byte buffers, not on strings.
perl5 strings can be utf8 and non-utf8. Why on earth an utf8 encoded string is disallowed and only byte buffers of unknown encodings are allowed goes beyond my understanding, but what can you do. Nothing. base64 is a binary only protocol, based on byte buffers. So we decode it manually to byte buffers. The Encode API for decoding is called encode.

$ perl -MMIME::Base64 -MEncode -lE'$x="20e3";$s="\x{20e3}";
  printf "0x%s\t%s=> %s",$x,$s,encode_base64(encode('utf8',$s))'
Wide character in printf at -e line 1.
0x20e3	=> 4oOj

This is now the term warning I know. We need -C

$ perldoc perluniintro

$ perl -C -MMIME::Base64 -MEncode -lE'$x="20e3";$s="\x{20e3}";
  printf "0x%s\t%s=> %s",$x,$s,encode_base64(encode('utf8',$s))'
0x20e3	=> 4oOj

Over to rakudo/perl6 and parrot:

$ cat >m.pir << EOP
.sub main :main
    load_bytecode 'MIME/Base64.pbc'
    $P1 = get_global [ "MIME"; "Base64" ], 'encode_base64'
    $S1 = utf8:"\x{203e}"
    $S2 = $P1(s1)
    say $S1
    say $S2
.end
EOP

$ parrot m.pir
FixedIntegerArray: index out of bounds!
current instr.: 'parrot;MIME;Base64;encode_base64'
                pc 163 (runtime/parrot/library/MIME/Base64.pir:147)

The perl6 test, using the parrot library, from https://github.com/ronaldxs/perl6-Enc-MIME-Base64/

$ git clone git://github.com/ronaldxs/perl6-Enc-MIME-Base64.git
Cloning into 'perl6-Enc-MIME-Base64'...

$ PERL6LIB=perl6-Enc-MIME-Base64/lib perl6 <<EOP
use Enc::MIME::Base64;
say encode_base64_str("\x203e");
EOP

> use Enc::MIME::Base64;
Nil
> say encode_base64_str("\x203e");
FixedIntegerArray: index out of bounds!
...

The pure perl6 workaround:

$ PERL6LIB=perl6-Enc-MIME-Base64/lib perl6 <<EOP
use PP::Enc::MIME::Base64;
say encode_base64_str("\x203e");
EOP

> use PP::Enc::MIME::Base64;
Nil
> say encode_base64_str("\x203e");
4oC+

Wait. perl6 creates a different enoding than perl5?
What about coreutils base64 command.

$ echo -n "‾" > m.raw
$ od -x m.raw
0000000 80e2 00be
0000003
$ ls -al m.raw
-rw-r--r-- 1 rurban rurban 3 Dec  6 10:23 m.raw
$ base64 m.raw
4oC+

[80e2 00be] is the little-endian version of [e2 80 be], 3 bytes, flipped.
Ok, at least base64 agrees with perl6, and I must have made some encoding mistake with perl5.

Back to debugging our parrot problem:

parrot unlike perl6 has no debugger yet. So we have to use gdb, and we need to know in which function the error occured. We use the parrot -t trace flag, which is like the perl5 debugging -Dt flag, but it is always enabled, even in optimized builds.

$ parrot --help
...
    -t --trace [flags] 
    --help-debug
...
$ parrot --help-debug
...
--trace -t [Flags] ...
    0001    opcodes
    0002    find_method
    0004    function calls

$ parrot -t7 m.pir
...
009f band I9, I2, 63         I9=0 I2=0 
00a3 set I10, P0[I5]         I10=0 P0=FixedIntegerArray=PMC(0xff7638) I5=[2063]
016c get_results PC2 (1), P2 PC2=FixedIntegerArray=PMC(0xedd178) P2=PMCNULL
016f finalize P2             P2=Exception=PMC(0x16ed498)
0171 pop_eh
lots of error handling
...
0248 callmethodcc P0, "print" P0=FileHandle=PMC(0xedcca0) 
FixedIntegerArray: index out of bounds!

We finally see the problem, which matches the run-time error.

00a3 set I10, P0[I5]         I10=0 P0=FixedIntegerArray=PMC(0xff7638) I5=[2063]

We want to set I10 to the I5=2063′th element in the FixedIntegerArray P0, and the array is not big enough.

After several hours of analyzing I came to the conclusion that the parrot library MIME::Base64 was wrong by using ord of every character in the string. It should use a bytebuffer instead.
Which was fixed with commit 3a48e6. ord can return integers > 255, but base64 can only handle chars < 255.

The fixed parrot library was now correct:

$ parrot m.pir
‾
4oC+

But then the tests started failing. I spent several weeks trying to understand why the parrot testsuite was wrong with the mime_base64 tests, the testdata came from perl5. I came up with different implementation hacks which would match the testsuite, but finally had to bite the bullet, changing the tests to match the implementation.

And I had to special case the tests for big-endian, as base64 is endian agnostic. You cannot decode a base64 encoded powerpc file on an intel machine, when you use multi-byte characters. And utf-8 is even more multi-byte than ucs-2. I had to accept the fact the big-endian will return a different encoding. Before the results were the same. The tests were written to return the same encoding on little and big-endian.

Summary

The first reason why I wrote this blog post was to show how to debug into crazy problems like this, when you are not sure if the core implementation, the library, the spec or the tests are wrong. It turned out, that the library and the tests were wrong.
You saw how easily you could use gdb to debug into such problems, as soon as you find out a proper breakpoint.

The internal string representations looked like this:

MIME::Base64 internally:

len=1, encoding=utf-8, buf=[3e20]

and inside the parrot imcc compiler the SREG

len=8, buf="utf-8:\"\x{203e}\""

parrot is a register based runtime, and a SREG is the string representation of the register value. Unfortunately a SREG cannot hold the encoding info yet, so we prefix the encoding in the string, and unquote it back. This is not the reason why parrot is still slower than the perl5 VM. I benchmarked it. parrot still uses too much sprintf’s internally and the encoding quote/unquoting counts only for a 4th of the time of the sprintf gyrations.
And parrot function calls are awfully slow and de-optimized.

The second reason is to explain the new decode_base64() API, which only parrot – and therefore all parrot based languages like rakudo – now have got.

decode_base64(str, ?:encoding)

“Decode a base64 string by calling the decode_base64() function.
This function takes as first argument the string to decode, as optional second argument the encoding string for the decoded data.
It returns the decoded data.

Any character not part of the 65-character base64 subset is silently ignored.
Characters occurring after a ‘=’ padding character are never decoded.”

So decode_base64 got now a second optional encoding argument. The src string for encode_base64 can be any encoding and is automatically decoded to a bytebuffer. You can easily encode an image or unicode string without any trouble, and for the decoder you can define the wanted encoding beforehand. The result can be the encoding binary or utf-8 or any encoding you prefer, no need for additional decoding of the result. The default encoding of the decoded string is either ascii, latin-1 or utf-8. parrot will upgrade the encoding automatically.

You can compare the new examples of pir against the perl5 version:

parrot:

.sub main :main
    load_bytecode 'MIME/Base64.pbc'

    .local pmc enc_sub
    enc_sub = get_global [ "MIME"; "Base64" ], 'encode_base64'

    .local string result_encode
    # GH 814
    result_encode = enc_sub(utf8:"\x{a2}")
    say   "encode:   utf8:\"\\x{a2}\""
    say   "expected: wqI="
    print "result:   "
    say result_encode

    # GH 813
    result_encode = enc_sub(utf8:"\x{203e}")
    say   "encode:   utf8:\"\\x{203e}\""
    say   "expected: 4oC+"
    print "result:   "
    say result_encode

.end

perl5:

use MIME::Base64 qw(encode_base64 decode_base64);
use Encode qw(encode);

my $encoded = encode_base64(encode("UTF-8", "\x{a2}"));
print  "encode:   utf-8:\"\\x{a2}\"  - ", encode("UTF-8", "\x{a2}"), "\n";
print  "expected: wqI=\n";
print  "result:   $encoded\n";
print  "decode:   ",decode_base64("wqI="),"\n\n"; # 302 242

my $encoded = encode_base64(encode("UTF-8", "\x{203e}"));
print  "encode:   utf-8:\"\\x{203e}\"  -> ",encode("UTF-8", "\x{203e}"),"\n";
print  "expected: 4oC+\n";
print  "result:   $encoded\n"; # 342 200 276
print  "decode:   ",decode_base64("4oC+"),"\n";

for ([qq(a2)],[qq(c2a2)],[qw(203e)],[qw(3e 20)],[qw(1000)],[qw(00c7)],[qw(00ff 0000)]){
    $s = pack "H*",@{$_};
    printf "0x%s\t=> %s", join("",@{$_}), encode_base64($s);
}

perl6:

use Enc::MIME::Base64;
say encode_base64_str("\xa2");
say encode_base64_str("\x203e");

Day 6 – Lexical Imports

2012-12-06T00:00:44+00:00

Perl 6 is built on lexical scopes. Variables, subroutines, constants and even types are looked up lexically first, and subroutines are only looked up in lexical scopes.

So it is only fitting that importing symbols from modules is also done into lexical scopes. I often write code such as

    use v6;

    # the main functionality of the script
    sub deduplicate(Str $s) {
        my %seen;
        $s.comb.grep({!%seen{ .lc }++}).join;
    }

    # normal call
    multi MAIN($phrase) {
        say deduplicate($phrase)
    }

    # if you call the script with --test, it runs its unit tests
    multi MAIN(Bool :$test!) {
        # imports &plan, &is etc. only into the lexical scope
        use Test;
        plan 2;
        is deduplicate('just some words'),
            'just omewrd', 'basic deduplication';
        is deduplicate('Abcabd'),
            'Abcd', 'case insensitivity';
    }

This script removes all but the first occurrence of each character given on the command line:

    $ perl6 deduplicate 'Duplicate character removal'
    Duplicate hrmov

But if you call it with the --test option, it runs its own unit tests:

    $ perl6 deduplicate --test
    1..2
    ok 1 - basic deduplication
    ok 2 - case insensitivity

Since the testing functions are only necessary in a part of the program — in a lexical scope, to be more precise –, the use statement is inside that scope, and limits the visibility of the imported symbols to this scope. So if you try to use the is function outside the routine in which Test is used, you get a compile-time error.

Why, you might ask? From the programmer's perspective, it reduces risk of (possibly unintended and unnoticed) name clashes the same way that lexical variables are safer than global variables.

From the point of view of language design, the combination of lexical importing, runtime-immutable lexical scopes and lexical-only lookup of subroutines allows resolving subroutine names at compile time, which again allows neat stuff like detecting calls to undeclared functions, compile-time type checking of arguments, and other nice optimizations.

But subroutines are only the tip of the iceberg. Perl 6 has a very flexible syntax, which you can modify with custom operators and macros. Those too can be exported, and imported into lexical scopes. Which means that language modifications are also lexically by default. So you can safely load any language-modifying extension, without running into danger that a library you use can't cope with it — the library doesn't even see the language modification.

So ultimately, lexical importing is another facet of encapsulation.

Day 5 – A Perl 6 Debugger

2012-12-05T00:01:22+00:00

There’s much more to the developer experience of a language than its design, features and implementations. While the language and its implementations are perhaps the thing developers will spend most time with, the overall experience will also involve interaction with the community, reading documentation, using modules and employing various development tools. Thus, it’s important that Perl 6 make progress on these fronts too. Over the past year, we’ve taken some good steps forward in these areas; there’s now doc.perl6.org, the module ecosystem has grown, and the module installation tooling has improved. Another big step forward with regards to tooling – and the topic of this post – is that an interactive Perl 6 debugger is now available.

Running With The Debugger

The debugger has been included with the last few Rakudo * releases. If you have one of those, you’re all set. Just run perl6-debug instead of perl6. It takes the same set of options, so if your normal invocation involves, for example, using the -I flag to set the include path for modules, it’ll Just Work Like Usual. Of course, what happens next is entirely different. The debugger will show you each module it is loading, followed by placing you at the first interesting statement of the program, highlighted in yellow.

Note how it takes care to put you on the first line that actually does something, skipping the my statement above it (it’s getting increasingly smart about this).

The Basics

Hitting enter allows you to single-step through the program. At any point, you can look at variables, call methods on variables, or even evaluate expressions.

If you want to move statement by statement, but never descend into a function call or method call, type an s, followed by enter. To step out of the current sub (that is, run until it returns then break in its caller), use so to step out. To run the program until it hits an exception, just use r. Even at the point you get an exception, you can still access variables to try and dissect what went wrong.

One final variant, rt, will run until an exception is thrown, but handled. You’ll break at the point of the throw. This means you’re not disadvantaged in the debugger if you took care to handle exceptions well in your program; you can still break when they are thrown and use the debugger to help understand why. :-)

Breakpoints

Sometimes, you know exactly where the juicy stuff happens in your program that you wish to debug. If only you could just run until you got there. Turns out you can – that’s what breakpoints are for. We can add one, use r to run, and it will stop where we placed the breakpoint.

Note that you don’t have to type out the full name of the file you want to put the breakpoint in; any unambiguous substring of the name of a file that is loaded will be sufficient.

I won’t cover them here, but there are also tracepoints, which instead of breaking will log the value of an expression each time a certain place in the program is hit. Later, you can display the log. It’s like adding print statements, but without the print statement going in your code, removing the risk of them accidentally making it into a commit (‘cus we’ve all done that one, right? :-))

Regex and Grammar Debugging

When the debugger detects you are in a regex or grammar, it offers a little extra help. As well as allowing you to single-step your way through the regex, atom by atom, it also displays the match text, indicating what has been matched so far.

Here, you can see that the pattern already successfully matched SELECT, and is now looking for a literal * or will try to call the field list rule. If in a regex, which may backtrack, the match position jumps backwards when backtracking happens, so you can understand the backtracking behavior of the pattern.

Yes, Perl 5 Regexes Too!

Rakudo has some support for the :P5 adverb on regexes, which allows use of the Perl 5 regex syntax. Here the debugger is used in REPL mode (where you enter an expression, then can immediately debug it) to explore the difference between alternations in Perl 5 and Perl 6 (in Perl 5 they go left to right, in Perl 6 they have longest token matching semantics, such that it tries the thing that will match most characters first).

The debugger in REPL mode is great for exploring and understanding how things will execute (and as such can serve as a learning or teaching aid). Another use is for debugging modules without having to write a test script; just write a use statement in the debugger or you can even supply the module using the -M command line flag and the debugger will load it!

What About Funky Stuff, Like Macros?

That is, macros, BEGIN time, eval, and those other things where your Perl 6 program does the time warp again, doing a bit of runtime at compile time or compiling some more stuff at runtime. The debugger is built for it. If a macro is applied, the debugger will place you in it. Notice below how we’re still in the process of loading the second file, and did not get to the third yet – we really are debugging at BEGIN time!

Any lines of code that are stripped out by the macro are simply never hit at runtime. And what about statements in quasi blocks? The debugger will take you there, so you not only know what macro was applied, but can dig into exactly what it does too.

Just as bits of runtime happening at compile time work out fine, any code that gets eval‘d at runtime is also compiled with debug hooks, meaning that you can step straight into it and debug the evaluated code.

Written in Perl 6 and NQP!

You might think that writing a debugger must involve all kinds of low-level hackery. In fact, that’s not the case. The debug hooks mechanism is written in NQP, and the command line user interface is written in Perl 6. This is significant from a couple of angles. The first is the fact that we can write something like this without breaking the encapsulation of the compiler, but instead just by subclassing the Grammar, Actions and Compiler objects and twiddling with the AST. In fact, the debugger was built without any changes being required to Rakudo as it already existed. This provides important feedback on our compiler architecture – this time, very positive feedback. Things are extensible in the ways they were designed to be. The second is that writing so much of it in Perl 6 is a healthy bit of dogfooding – using the product in order to build further products. My hope is that, since most of what people would want to change is actually written in the Perl 6 part, it will feel quite hackable by the community at large.

And What Of Future Plans?

Various features are still to come: conditional breakpoints, dumping tracepoint output to a file, showing the path taken through a grammar to get to the current point, and various bits of configurability. The command line interface is nice, but of course having some extra options would be even nicer. I’m interested in a web-based interface, but also in integration with tools like Padre. There’s some work afoot on a common protocol for these things, which could make such integration possible without having to re-invent too many wheels. In the meantime, having an interactive debugger which is aware of and works well with a wide range of Perl 6 language features is a solid step forward. Happy debugging, and feature ideas (or patches ;-)) are welcome; here’s the GitHub repo!

Day 4 – Having Fun with Rakudo and Project Euler

2012-12-04T00:01:40+00:00

Rakudo, the leading Perl6 implementation, is not perfect, and performance is a particularly sore subject. However, the pioneer does not ask ‘Is it fast?’, but rather ‘Is it fast enough?’, or perhaps even ‘How can I help to make it faster?’.

To convince you that Rakudo can indeed be fast enough, we’ll take a shot at a bunch of Project Euler problems. Many of those involve brute-force numerics, and that’s something Rakudo isn’t particularly good at right now. However, that’s not necessarily a show stopper: The less performant the language, the more ingenious the programmer needs to be, and that’s where the fun comes in.

All code has been tested with Rakudo 2012.11.

We’ll start with something simple:

Problem 2

By considering the terms in the Fibonacci sequence whose values do not exceed four million, find the sum of the even-valued terms.

The solution is beautifully straight-forward:

    say [+] grep * %% 2, (1, 2, *+* ...^ * > 4_000_000);

Runtime: 0.4s

Note how using operators can lead to code that’s both compact and readable (opinions may vary, of course). We used

whatever stars * to create lambda functions
the sequence operator (in its variant that excludes the right endpoint) ...^ to build up the Fibonacci sequence
the divisible-by operator %% to grep the even terms
reduction by plus [+] to sum them.

However, no one forces you to go crazy with operators – there’s nothing wrong with vanilla imperative code:

Problem 3

What is the largest prime factor of the number 600,851,475,143?

An imperative solution looks like this:

    sub largest-prime-factor($n is copy) {
        for 2, 3, *+2 ... * {
            while $n %% $_ {
                $n div= $_;
                return $_ if $_ > $n;
            }
        }
    }

    say largest-prime-factor(600_851_475_143);

Runtime: 2.6s

Note the is copy trait, which is necessary as Perl6 binds arguments read-only by default, and that integer division div is used instead of numeric division /.

Nothing fancy going on here, so we’ll move along to

Problem 53

How many, not necessarily distinct, values of ⁿC_r, for 1 ≤ n ≤ 100, are greater than one-million?

We’ll use the feed operator ==> to factor the algorithm into separate steps:

    [1], -> @p { [0, @p Z+ @p, 0] } ... * # generate Pascal's triangle
    ==> (*[0..100])()                     # get rows up to n = 100
    ==> map *.list                        # flatten rows into a single list
    ==> grep * > 1_000_000                # filter elements exceeding 1e6
    ==> elems()                           # count elements
    ==> say;                              # output result

Runtime: 5.2s

Note the use of the Z meta-operator to zip the lists 0, @p and @p, 0 with +.

The one-liner generating Pascal’s triangle has been stolen from Rosetta Code, another great resource for anyone interested in Perl6 snippets and exercises.

Let’s do something clever now:

Problem 9

There exists exactly one Pythagorean triplet for which a + b + c = 1000. Find the product abc.

Using brute force will work (solution courtesy of Polettix), but it won’t be fast (~11s on my machine). Therefore, we’ll use a bit of algebra to make the problem more managable:

Let (a, b, c) be a Pythagorean triplet
 a < b < c
 a² + b² = c²
For N = a + b + c it follows
 b = N·(N - 2a) / 2·(N - a)
 c = N·(N - 2a) / 2·(N - a) + a²/(N - a)
which automatically meets b < c.

The condition a < b gives the constraint
 a < (1 - 1/√2)·N

We arrive at

    sub triplets(\N) {
        for 1..Int((1 - sqrt(0.5)) * N) -> \a {
            my \u = N * (N - 2 * a);
            my \v = 2 * (N - a);

            # check if b = u/v is an integer
            # if so, we've found a triplet
            if u %% v {
                my \b = u div v;
                my \c = N - a - b;
                take $(a, b, c);
            }
        }
    }

    say [*] .list for gather triplets(1000);

Runtime: 0.5s

Note the declaration of sigilless variables \N, \a, …, how $(…) is used to return the triplet as a single item and .list – a shorthand for $_.list – to restore listy-ness.

The sub &triplets acts as a generator and uses &take to yield the results. The corresponding &gather is used to delimit the (dynamic) scope of the generator, and it could as well be put into &triplets, which would end up returning a lazy list.

We can also rewrite the algorithm into dataflow-driven style using feed operators:

    constant N = 1000;

    1..Int((1 - sqrt(0.5)) * N)
    ==> map -> \a { [ a, N * (N - 2 * a), 2 * (N - a) ] }
    ==> grep -> [ \a, \u, \v ] { u %% v }
    ==> map -> [ \a, \u, \v ] {
        my \b = u div v;
        my \c = N - a - b;
        a * b * c
    }
    ==> say;

Runtime: 0.5s

Note how we use destructuring signature binding -> […] to unpack the arrays that get passed around.

There’s no practical benefit to use this particular style right now: In fact, it can easily hurt performance, and we’ll see an example for that later.

It is a great way to write down purely functional algorithms, though, which in principle would allow a sufficiently advanced optimizer to go wild (think of auto-vectorization and -threading). However, Rakudo has not yet reached that level of sophistication.

But what to do if we’re not smart enough to find a clever solution?

Problem 47

Find the first four consecutive integers to have four distinct prime factors. What is the first of these numbers?

This is a problem where I failed to come up with anything better than brute force:

    constant $N = 4;

    my $i = 0;
    for 2..* {
        $i = factors($_) == $N ?? $i + 1 !! 0;
        if $i == $N {
            say $_ - $N + 1;
            last;
        }
    }

Here, &factors returns the number of prime factors. A naive implementations looks like this:

    sub factors($n is copy) {
        my $i = 0;
        for 2, 3, *+2 ...^ * > $n {
            if $n %% $_ {
                ++$i;
                repeat while $n %% $_ {
                    $n div= $_
                }
            }
        }
        return $i;
    }

Runtime: unknown (33s for N=3)

Note the use of repeat while … {…}, the new way to spell do {…} while(…);.

We can improve this by adding a bit of caching:

    BEGIN my %cache = 1 => 0;

    multi factors($n where %cache) { %cache{$n} }
    multi factors($n) {
        for 2, 3, *+2 ...^ * > sqrt($n) {
            if $n %% $_ {
                my $r = $n;
                $r div= $_ while $r %% $_;
                return %cache{$n} = 1 + factors($r);
            }
        }
        return %cache{$n} = 1;
    }

Runtime: unknown (3.5s for N=3)

Note the use of BEGIN to initialize the cache first, regardless of the placement of the statement within the source file, and multi to enable multiple dispatch for &factors. The where clause allows dynamic dispatch based on argument value.

Even with caching, we’re still unable to answer the original question in a reasonable amount of time. So what do we do now? We cheat and use Zavolaj – Rakudo’s version of NativeCall – to implement the factorization in C.

It turns out that’s still not good enough, so we refactor the remaining Perl code and add some native type annotations:

    use NativeCall;

    sub factors(int $n) returns int is native('./prob047-gerdr') { * }

    my int $N = 4;

    my int $n = 2;
    my int $i = 0;

    while $i != $N {
        $i = factors($n) == $N ?? $i + 1 !! 0;
        $n = $n + 1;
    }

    say $n - $N;

Runtime: 1m2s (0.8s for N=3)

For comparison, when implementing the algorithm completely in C, the runtime drops to under 0.1s, so Rakudo won’t win any speed contests just yet.

As an encore, three ways to do one thing:

Problem 29

How many distinct terms are in the sequence generated by a^b for 2 ≤ a ≤ 100 and 2 ≤ b ≤ 100?

A beautiful but slow solution to the problem can be used to verify that the other solutions work correctly:

    say +(2..100 X=> 2..100).classify({ .key ** .value });

Runtime: 11s

Note the use of X=> to construct the cartesian product with the pair constructor => to prevent flattening.

Because Rakudo supports big integer semantics, there’s no loss of precision when computing large numbers like 100¹⁰⁰.

However, we do not actually care about the power’s value, but can use base and exponent to uniquely identify the power. We need to take care as bases can themselves be powers of already seen values:

    constant A = 100;
    constant B = 100;

    my (%powers, %count);

    # find bases which are powers of a preceeding root base
    # store decomposition into base and exponent relative to root
    for 2..Int(sqrt A) -> \a {
        next if a ~~ %powers;
        %powers{a, a**2, a**3 ...^ * > A} = a X=> 1..*;
    }

    # count duplicates
    for %powers.values -> \p {
        for 2..B -> \e {
            # raise to power \e
            # classify by root and relative exponent
            ++%count{p.key => p.value * e}
        }
    }

    # add +%count as one of the duplicates needs to be kept
    say (A - 1) * (B - 1) + %count - [+] %count.values;

Runtime: 0.9s

Note that the sequence operator ...^ infers geometric sequences if at least three elements are provided and that list assignment %powers{…} = … works with an infinite right-hand side.

Again, we can do the same thing in a dataflow-driven, purely-functional fashion:

    sub cross(@a, @b) { @a X @b }
    sub dups(@a) { @a - @a.uniq }

    constant A = 100;
    constant B = 100;

    2..Int(sqrt A)
    ==> map -> \a { (a, a**2, a**3 ...^ * > A) Z=> (a X 1..*).tree }
    ==> reverse()
    ==> hash()
    ==> values()
    ==> cross(2..B)
    ==> map -> \n, [\r, \e] { (r) => e * n }
    ==> dups()
    ==> ((A - 1) * (B - 1) - *)()
    ==> say();

Runtime: 1.5s

Note how we use &tree to prevent flattening. We could have gone with X=> instead of X as before, but it would make destructuring via -> \n, [\r, \e] more complicated.

As expected, this solution doesn’t perform as well as the imperative one. I’ll leave it as an exercise to the reader to figure out how it works exactly ;)

That’s it

Feel free to add your own solutions to the Perl6 examples repository under euler/.

If you’re interested in bioinformatics, you should take a look at Rosalind as well, which also has its own (currently only sparsely populated) examples directory rosalind/.

Last but not least, some solutions for the Computer Language Benchmarks Game – also known as the Debian language shootout – can be found under shootout/.

You can contribute by sending pull requests, or better yet, join #perl6 on the Freenode IRC network and ask for a commit bit.

Have the appropriate amount of fun!

Day 3 – Whatever the layout manager is

2012-12-03T00:01:36+00:00

Introduction

This article aims to demonstrate how Whatever — one of the many interesting Perl 6 curiosities — could be useful to easily implement and use complex things like a layout manager. In a couple of words, a layout manager is the part of a graphical interface in charge of the spatial arrangement of objects like windows or widgets. For the sake of simplicity, the layout manager implemented in this article will comply with the following three rules:

there are only two kinds of widgets: terminal or container, the latter can contain other widgets of either kind;
a widget cannot be overlapped, except for containers which fully contain their sub-widgets; and
only the height can be adjusted, this size can be either static, dynamic, or intentionally left unspecified.

Usage

From the user's point-of-view, this layout manager aims to be as easy to use as possible. For example, it shouldn't be hard to specify such typical interface below, inspired from a text-based program. In this example, the interface and body widgets are containers, all the others are terminals:

interface (X lines)
+----> +------------------------------------------+
|      | menu bar (1 line)                        |  body (remaining space)
|      +------------------------------------------+ <----+
|      | subpart 1 (1/3 of the remaining space)   |      |
|      |                                          |      |
|      |                                          |      |
|      +------------------------------------------+      |
|      | subpart 2 (remaining space)              |      |
|      |                                          |      |
|      |                                          |      |
|      |                                          |      |
|      |                                          |      |
|      |                                          |      |
|      +------------------------------------------+ <----+
|      | status bar (1 line)                      |
+----> +------------------------------------------+

The user don't know what the remaining space is in advance because such an interface is arbitrary resizable. As a consequence it should be specified as a non-predefined value; this is where * — the Whatever object — comes in handy. This object is interesting for two reasons:

from the user's point-of-view, the definition of non-static sizes is as simple as: * / 3 for the subpart 1 (dynamic) and just * for the subpart 2 (unspecified); and
from the developer's point-of-view, Perl 6 transforms automatically things like $size = * / 3 into a closure: x → x / 3. Then, it could be called like a regular function: $size($x).

That way, the previous GUI can be transliterated into the following lines of code:

my $interface =
    Widget.new(name => 'interface', size => $x, sub-widgets => (
        Widget.new(name => 'menu bar', size => 1),
        Widget.new(name => 'main part', size => *, sub-widgets => (
            Widget.new(name => 'subpart 1', size => * / 3),
            Widget.new(name => 'subpart 2', size => *))),
        Widget.new(name => 'status bar', size => 1)));

Implementation

The drawing of terminal widgets is straightforward since most of the work is done by containers. Those are in charge to compute the remaining space as well as to uniformly distribute widgets that have an unspecified size:

class Widget {
    has $.name;
    has $.size is rw;
    has Widget @.sub-widgets;

    method compute-layout($remaining-space? is copy, $unspecified-size? is copy) {
        $remaining-space //= $!size;

        if @!sub-widgets == 0 {  # Terminal
            my $computed-size = do given $!size {
                when Real     { $_                  };
                when Callable { .($remaining-space) };
                when Whatever { $unspecified-size   };
            }

            self.draw($computed-size);
        }
        else {  # Container
            my @static-sizes   =  grep Real,     @!sub-widgets».size;
            my @dynamic-sizes  =  grep Callable, @!sub-widgets».size;
            my $nb-unspecified = +grep Whatever, @!sub-widgets».size;

            $remaining-space -= [+] @static-sizes;

            $unspecified-size = ([-] $remaining-space, @dynamic-sizes».($remaining-space))
                                 / $nb-unspecified;

            .compute-layout($remaining-space, $unspecified-size) for @!sub-widgets;
        }
    }

    method draw(Real $size is copy) {
        "+{'-' x 25}+".say;
        "$!name ($size lines)".fmt("| %-23s |").say;
        "|{' ' x 25}|".say while --$size > 0;
    }
}

Here, any Callable object can be used to specify a dynamic size, as far as it takes the computed remaining space as argument. That means it is possible to specify more sophisticated dynamic size by passing a code Block. For example, { max(5, $^x / 3) } ensures the widget has a proportional size that can't decrease below 5.

Conclusion

It's time to check if this trivial layout manager works correctly both in Rakudo and Niecza, the two most advanced implementations of Perl 6. The following test is rather simple, it creates and draws an interface, then resize it and draws it again:

my $interface =
    Widget.new(name => 'interface', size => 11, sub-widgets => (
        Widget.new(name => 'menu bar', size => 1),
        Widget.new(name => 'main part', size => *, sub-widgets => (
            Widget.new(name => 'subpart 1', size => * / 3),
            Widget.new(name => 'subpart 2', size => *))),
        Widget.new(name => 'status bar', size => 1)));

$interface.compute-layout;  # Draw
$interface.size += 3;       # Resize
$interface.compute-layout;  # Redraw

The results before and after resizing are respectively displayed below. They are close enough from the initial mockup, n'est-ce pas?

+-------------------------+            +-------------------------+
| menu bar (1 lines)      |            | menu bar (1 lines)      |
+-------------------------+            +-------------------------+
| subpart 1 (3 lines)     |            | subpart 1 (4 lines)     |
|                         |            |                         |
|                         |            |                         |
+-------------------------+            |                         |
| subpart 2 (6 lines)     |            +-------------------------+
|                         |            | subpart 2 (8 lines)     |
|                         |            |                         |
|                         |            |                         |
|                         |            |                         |
|                         |            |                         |
+-------------------------+            |                         |
| status bar (1 lines)    |            |                         |
                                       |                         |
                                       +-------------------------+
                                       | status bar (1 lines)    |

Finally, the implementation of such a flexible program is really simple in Perl 6: everything is already there, in the core language. Obviously, this trivial layout manager isn't ready for prime-time since a lot of things are missing: sanity checks, multiple dimensions, … but those are left as exercises to you, the reader ;) For any questions or comments, feel free to meet Perl 6 fellows on IRC (#perl6 on freenode).

Bonus

As seen previously, $!size can be Whatever, but it can't be whatever you want. For example, a negative Real or a string are not correct values. Once again Perl 6 provides a simple yet powerful feature: constrained types. In a couple of words this permits to define new types from a set of constraints:

subset PosReal of Real where * >= 0;
subset Size where {   .does(PosReal)
                   or .does(Callable) and .signature ~~ :(PosReal --> PosReal)
                   or .does(Whatever) };

has Size $.size is rw;

Day 2 – Anonymous functions for great good

2012-12-02T00:01:21+00:00

Perl 6 has great support for functions. It packs function signatures full with awesome, and lets you have your cake and eat it a couple of times over with all the ways you can specify a function. You can specify parameter types, optional parameters, named parameters, and even those cool where clauses. If I didn’t know better, I’d suspect Perl 6 was compensating for some predecessor’s rather rudimentary handling of parameters. (*cough* @_ *cough*)

Among all these other things, Perl 6 also allows you to define functions without naming them.

sub { say "lol, I'm so anonymous!" }

How is this useful? If you can’t name the function, you can’t call it, right? Wrong.

You can store the function in a variable. Or return it from another function. Or pass it to another function. In fact, when you don’t name your function, the focus becomes much more what code you’re going to run later. Like an executable “to do”-list.

Of course, Perl 5 has anonymous functions, too. With exactly the same syntax, even. In fact, all the big languages do anonymous functions, according to this list of languages on Wikipedia. Except, it seems, the historically significant languages C and Pascal. And the more modern but lumbering Java. “Planned for Java 8″. Haha, Java, catch up! Even C++ has them now.

How important are anonymous functions? Very. In the 1930s, Alan Turing showed how all computer processes could be simulated using just a pre-programmed machine that looks like a tape recorder, reading and writing values on a really long tape. (The Turing Machine.) Meanwhile, across the Atlantic, Alonzo Church showed how all computer processes could be simulated using just anonymous functions, no tape recorder required. (Lambda calculus.) It’s all quite elegant.

Later languages like Lisp and Scheme lean heavily on anonymous functions as a key component in the language. And lately a scrappy language called JavaScript, which also leans heavily on anonymous functions, has taken over the world while we were all busy surfing the web.

But let’s talk possibilities here. What can anonymous functions do for us? And how would it look in Perl 6?

Well, take sorting as a famous example. You could imagine Perl 6 having a sort_lexicographically function and a sort_numerically function. But it doesn’t. It has a sort function. When you want it to sort in a certain way, you just pass an anonymous function to it.

my @sorted_words = @words.sort({ ~$_ });
my @sorted_numbers = @numbers.sort({ +$_ });

(Technically, those are blocks, not functions. But the difference isn’t significant if you’re not planning to return anywhere inside.)

And of course it goes further than just those two sort orders. You could sort by shoe size, or maximum ground speed, or decreasing likelihood of spontaneous combustion. All because you can pass in any logic as an argument. Object-oriented people are very proud of this pattern, and call it “dependency injection”.

Come to think of it, map and grep and reduce all depend on this kind of function-passing. We sometimes refer to passing functions to functions as “higher order programming”, as if it was only something people with special privileges should be doing. But in fact it’s a very useful and broadly applicable technique.

The above examples all run the anonymous functions as part of their own execution. But there’s no need to restrict ourselves to this. We can create functions, return them, and then run them later:

sub make_surprise_for($name) {
    return sub { say "Sur-priiise, $name!" };
}

my $reveal_surprise = make_surprise_for("Finn");    # nothing happens, yet
# ...wait for it...
# ...wait...
# ...waaaaaaait...
$reveal_surprise();        # "Sur-priiise, Finn!"

The function in $reveal_surprise remembers the value of $name even though the original function passing it in has exited long ago. That’s pretty nice. This effect is referred to as the anonymous function closing over the variable $name. But there’s no need to get technical — the long and short of it is “it’s awesome”.

And in fact, it feels quite natural if we just look at anonymous functions alongside other staple storage mechanisms such as arrays and hashes. All of these can be stored in variables, passed as arguments or returned from functions. An anonymous array allows you to store a sequence of things for later. An anonymous hash allows you to store mappings/translations of things for later. An anonymous function allows you to store calculations or behavior for later.

Later this month, I’ll go through how to exploit dynamic scoping in Perl 6 to create nice DSL-y interfaces. We’ll see how anonymous functions come into play there as well.

Perl 6 changes - 2012W48

2012-12-01T08:00:00+00:00

Rakudo Star 2012.11 was released with new impressive improvements (check my previous posts for more information). If you’re reading this, please check out the Perl 6 Advent Calendar - new posts are added daily.

id='incompatible_changes'>Incompatible changes>

Rakudo Star (post Rakudo Star 2012.11)>

Removed `~/.perl6/lib` from `@*INC`.

id='new_features'>New features>

Perl 6 specification>

`Str.indent` ignores empty lines.

Rakudo Perl (in Rakudo Star 2012.11)>

Unknown metacharacters in regexp throw `X::Syntax::Regex::UnrecognizedMetachar`.

Null regexp throws `X::Syntax::Regex::NullRegex`.

Unrecognized backslash sequence throws `X::Backslash::UnrecognizedSequence`.

Non-variable dollar throws `X::Backslash::NonVariableDollar`.

Rakudo Perl (post Rakudo Star 2012.11)>

`--doc` option now works.

`Str.indent` ignores empty lines.

Added `Signature.ACCEPTS`.

Niecza Perl>

`make install` now works.

Day 1 – State of Perl 6 in 2012

2012-12-01T00:00:36+00:00

Welcome to another edition of your annual Perl 6 advent calendar.

As is tradition on the first of December, you can read a short overview over what has changed in the past year, and where we are standing now.

The list of major changes to the specification is pretty short. The IO subsystem has undergone a rewrite, and now much better reflects the realities in implementations, and actually has a measure of common sense applied. S32::Exceptions has gone through lots of changes (mostly extensions), and now there is a decent core of exception classes in Perl 6.

Both Rakudo and Niecza, the two major Perl 6 compilers, have matured a great deal. Contrary to last year, chances are pretty good that if your program works on one of the compilers, it also works on the other. Niecza also temporarily overtook Rakudo on the count of passing tests.

Niecza had a revamp of the roles implementation, has gained constant folding, awesome Unicode support in regexes, list comprehensions and a no strict; mode. To name just a few of the major changes.

Rakudo now supports heredocs, all phasers (special blocks like BEGIN, END, FIRST, …), longest-token matching in regexes, typed exceptions, much nicer backtraces and operator adverbs. And it now has a debugger, which is shipped with the Rakudo Star distribution.

The module ecosystem has grown a lot, and there is much more documentation for Perl 6 than a year ago.

So, after all these changes, where are we now?

Reports from production uses of Perl 6 are slowly starting to trickle in, and these days if your Perl 6 code has bugs, the chances are much higher that your code is to blame than the compilers. Perl 6 has never been this much fun to use. It surely has been a good and productive year for Perl 6, and we’re sure that this last month will continue the tradition. Have fun!

-2 6

2012-11-30T08:00:00+00:00

You know, -2 is highly connected to Perl 6. It’s valid syntax for -(2).

Rakudo Star 2012.11 released

2012-11-28T19:12:18+00:00

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the November 2012 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the November 2012 release is available for download. A Windows .MSI version of Rakudo star will usually appear in the downloads area shortly after the tarball release.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2012.11of the Rakudo Perl 6 compiler, version 4.6.0 of the Parrot Virtual Machine, and various modules, documentation, and other resources collected from the Perl 6 community.

Some of the new features added to this release include:

heredocs
quote adverbs (like q:w//)
implemented precedence related traits (equiv, looser, tighter, assoc)
Perl 6 grammar NFAs are pre-computed, saving some work on each invocation; this shaved around 10% off the time needed to run the spectests
regexes and quotes have better support for user-selected delimiters
FIRST/NEXT/LAST can now be used in all types of loop (previously limited to for)
several fixes related to module precompilation. This should make working with larger code bases much less painful.

This release also contains a range of performance improvements, bug fixes, improvements to error reporting and better failure modes.

The following features have been deprecated or modified from previous releases due to changes in the Perl 6 specification, and are being removed or changed as follows:

at present, a reference to an &foo that does not exist evalutes to Nil. This will become a CHECK-time failure, in line with STD.
Unary hyper ops currently descend into nested arrays and hashes. This will change to make them equivalent to a one-level map.

~/.perl6/lib will go away from the default include path (@*INC).
Instead %*CUSTOM_LIB now holds paths to four library locations:

perl    Rakudo installs its core modules here
vendor  OS-level package managers should install their modules here
site    for local module installations (e.g. with panda or ufo)
home    like site, but always under the user's home directory.
        fallback if site isn't writable.

The Str.ucfirst builtin is deprecated; it will be replaced by Str.tc.
Leading whitespace in rules and under :sigspace will no longer be converted to <.ws> . For existing regexes that expect this conversion, add a <?> in front of leading whitespace to make it meta again.
The ?-quantifier on captures in regexes currently binds the capture slot to a List containing either zero or one Match objects; i.e., it is equivalent to “** 0..1″. In the future, the ?-quantifier will bind the slot directly to a captured Match or to Nil. Existing code can manage the transition by changing existing ?-quantifiers to use “** 0..1″, which will continue to return a List of matches.

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Some of the not-quite-there features include:

advanced macros
threads and concurrency
Unicode strings at levels other than codepoints
interactive readline that understands Unicode
non-blocking I/O
much of Synopsis 9

There is an online resource at http://perl6.org/compilers/features that lists the known implemented and missing features of Rakudo and other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the programmer that a given feature isn’t implemented, but there are many that we’ve missed. Bug reports about missing and broken features are welcomed at .

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. A draft of a Perl 6 book is available as docs/UsingPerl6-draft.pdf in the release tarball.

Announce: Niecza Perl 6 v24 by Stefan O'Rear

2012-11-26T23:18:27+00:00

Announce: Niecza Perl 6 v24

This is release number 24 of Niecza Perl 6, marking 24 months since the first
public release. How time flies.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you still need a
binary for bootstrapping.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Bug fixes]

.classify and .categorize now return hashes. (Solomon Foster)

qx// is now sandboxed. (Solomon Foster)

[Features]

Added special functions gamma, expm1, log1p, erf, lgamma. (Solomon Foster,
using library code by John D. Cook)

Export $*CUSTOM-LIB for panda's use. (Solomon Foster)

Added Parcel.Array and List.Array. (Solomon Foster)

Postfix operaters which mimic methods can now be used. (Larry Wall)

Added functionality for a "precompiled system modules" folder, eliminating
the need for each user to separately compile the setting in a multi-user
install.

Added line editing to the REPL.

Added an install target for *nixes.

[Internal]

Niecza has been significantly refactored to use a single instance of the
kernel during compilation, opening the door to direct object sharing
between the compiler and the runtime and significantly simplifying the
bootstrap procedure.

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing.

[1] https://github.com/downloads/sorear/niecza/niecza-24.zip
[2] https://github.com/sorear/niecza

Perl 6 changes - 2012W47

2012-11-24T08:00:00+00:00

So, well, now that my code compiles (well, C code, not Perl 6 code :-)) I’ve time to describe Perl 6 changes, instead of simply doing what XKCD says - I shouldn’t waste time.

Rakudo Perl 2012.11 is now released. The change mentioned in this article is already in.

id='new_features'>New features>

Perl 6 specification>

`NativeCall` module is now part of specification.

Calls to `trait_mod` routines are evaluated in sink context.

STD.pm6>

`-->` (return type) acts more like a normal subroutine parameter.

Rakudo Perl>

`:($a, $b) := \(1, 2)` updates `$a` and `$b` instead of doing nothing.

No such pipe, or this pipe has been deleted

2012-11-23T08:42:40+00:00

This data comes from pipes.yahoo.com but the Pipe does not exist or has been deleted.

The Rakudo Codebase: Visualized! (Partially)

2012-11-22T08:59:05+00:00

Yesterday I stumbled upon this old perl6-compiler mailing list message which inspired me to actually try to split up the compilation of CORE.setting into smaller, saner pieces. I’ve started work on this already (so far having just modified the Makefile).

The one and only response to that post suggested using the stub syntax to resolve missing class issues. Sadly, that does not work, because perl6 needs to see the stubbed class in full sometime later in the file.

So, the best option I’ve seen is to include the files containing needed classes and roles into files that need them (through either use or require). This requires knowing what to include (as including everything everywhere ruins the point), and I’ve started to try to grasp what’s going on in the rakudo codebase, specifically src/core, in terms of dependencies.

Here’s what I have so far. It’s right now just one, single script. This single script graphs the inheritance chain of all the classes and roles in src/core (what classes and roles “is” and “does”). You need the modules Term::ANSIColor, Term::ProgressBar, and IO::Capture::Simple (at this time pulled in, but apparently unused, by Term::ProgressBar). You also need ack and graphviz (which is not used in the script, but needed to process the resulting core.dot file).

Wanna see the graph for nom commit bf472b0, the latest as of this writing? Here you go (click to embiggen, clearly):

The arrows point towards what a particular box is inheriting, and the one or two white boxes mean that the class or role doesn’t exist within the src/core files. Yellow denotes classes, red denotes roles. And each box contains the name of the class/role, and the file it was found in and the line number of its definition.

This, however, doesn’t even begin to deal with all the dependencies abound in the src/core files. See all those abandoned colored blocks? Those aren’t inherited by anything in src/core, but they most certainly are utilized (e.g. the Pod::Config block, though not inherited, could easily be utilized as a variable type in the Pod code). I have a lot more work to do before I have a clear picture on how to separate CORE.setting into smaller pieces. (I might even need to utilize STD.pm6 or similar to do all the parsing I’ll eventually be doing!)

Just to finish off, Here are a couple more graphs of the same data. The first graph above was generated with GraphViz’s “dot”. This one (generated by their “twopi” program) I think looks hilarious:

And this picture was generated by GraphViz’s “fdp”. I think this one is the easiest to follow.

Happy viewing!

Parrot 4.10.0 "Red-eared Parakeet" Released by Reini Urban

2012-11-21T17:27:35+00:00

On behalf of the Parrot team, I'm proud to announce Parrot 4.10.0, also known
as "Red-eared Parakeet". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.

Parrot 4.10.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/4.10.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 4.10.0 News:
- Core
+ Add PMCNULL check to result_info op
+ Fixed record separator bug on io handles
- Build
+ Add -fpermissive so that nqp builds under gcc 4.7.2
+ Fixed warnings for old ICU 4.4
- Platforms
+ More VMS fixes, but not yet done
- Branches
+ The threads branch is ready to be merged, but nqp needs some changes,
so it will be merged with 4.11.0.
- Examples
+ Parrot mini-web server fixes

The SHA256 message digests for the downloadable tarballs are:
82f38e6e596ab101c52313ad3e3560d2c8a72985362b54d9898619ab17a6f17d
parrot-4.10.0.tar.gz
7d6a592c386d25bcee429a269cc571ae81c5c080b15d14a884401a27fe3f5d4b
parrot-4.10.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 18 December 2012.

The release is a day late. I apologize for the lateness.
--
Reini Urban
http://cpanel.net/ http://www.perl-compiler.org/

Planet Perl Six

Parrot 5.4.0 "Austin Parrot" Released! by Bruce Gray

Perl 6 changes - 2013W20

Perl 6 specification> You cannot create a set or bag from list of pairs or hash using a constructor. The bag {a => 2} was ambiguous, because it could mean either bag containing 'a' with value 2, or bag containg {a => 2} with value 1. You’re expected to use .Bag or .Set instead, if you mean that.

STD.pm6> You cannot use set or bag to mean empty set or bag. If you really meant that, you are expected to use set() or bag() instead. id='new_features'>New features>

Niecza> Set, KeySet, Bag, and KeyBag have default method. Just like in Rakudo, except with Niecza.

Rakudo on JVM progress update, and some questions answered

The Rakudo Port

Interop

Performance?

From Here

Perl 6 changes - 2013W19

Niecza Perl> Pair.perl can now deal with the cases when Pair is the key. Doing ((a => 'b') => 'c').perl used to output "a" => "b" => "c", what actually meant (a => (b => 'c')). .perl now outputs correct code in that case. Implemented Hash.ACCEPTS.

After the #masakism workshop

Successes

Things learned

Message

Once more?

Perl 6 changes - 2013W18

To merge or not to merge?

Perl 6 changes - 2013W17

Perl 6 specification> DateTime doesn’t accept Callable timezones anymore. 5692ced333

Rakudo Perl> Renamed IO to IO::Handle. IO itself is a role. 2dca6ed950 37c9956502 DateTime doesn’t accept Callable timezones anymore. 71ea142406 id='new_features'>New features>

Rakudo Perl> Implemented Cool.lines. 0d1d1194a4

t2: Rectangle haikus

The #masakism workshop

Perl 6 changes - 2013W16

Perl 6 specification> Removed IO::Dir and IO::File classes. fb25fa475854 id='new_features'>New features>

NQP on JVM bootstrapped, soon will land in NQP master

Getting NQP on JVM Bootstrapped

Supporting invokedynamic

What next?

The REPL trick

Parrot 5.3.0 "W00tstock Parrot" Released! by Bruce Gray

Perl 6 changes - 2013W15

Perl 6 specification> Parcels are allowed to have 0 elements.

Niecza Perl> Added special cases for big_pow when either of values is -1, 0, or 1.

Perl 6 changes - 2013W14

Perl 6 specification> Renamed weekday-of-month to day-of-week-in-month.

Rakudo Perl> Fixed signature of Pod::Block.gist, so it works properly with multis. for loops are eager now. Added Buf.perl. Fixed Buf.gist, so it works properly with multis.

Rakudo's Abstract Syntax Tree

Ops and Constants

Variables and Blocks

Polymorphism and QAST::Want

Perl 6 changes - 2013W13

Perl 6 specification> Added many IO::Path methods: .contents, .is-absolute, .is-relative, .absolute, and .relative. .path is now NOP that returns itself. For its previous behavior, use .Str. Changed Numeric.conjugate to Numeric.conj.

Rakudo Perl> +(23..2300000000) is optimized. --help reports -I and -M options.

Polish Perl Workshop status update

Parrot 5.2.0 "Stuffed Parrot" Released! by Bruce Gray

Perl 6 changes - 2013W12

Perl 6 specification> Added X::Range::InvalidArg.

Rakudo Perl> Removed lots of pir:: usage. Multi dispatch is now implemented in NQP, not C. Released Rakudo 2013.03.

Reading from a file in Perl 6

Perl 6 changes - 2013W11

Niecza Perl> Empty.gist returns nothing.

Getting started with Rakudo * (2013.01) on Windows

Perl 6 changes - 2013W10

Perl 6 specification> Rat.perl returns decimal number when denominator is normalizable to a power of 10. Added FALLBACK methods. Included Failure in smartmatch passthroughs. Loosened relationship between Nil and (). Added List.permutations. rat64 type has unsigned denominator.

STD.pm6> Nil isn’t suggested for empty list in error messages anymore.

Niecza Perl> Rat.perl returns decimal number when denominator is normalizable to a power of 10. CommonEnum.gist returns (Type) instead of Type() for undefined. Restored infix:<cmp> sorting for different types. Failed matches return Nil.

Philosophical Issues with Rakudo’s .parse

Perl 6 changes - 2013W09

Perl 6 specification> Inside of try is considered eager context, unless try is called in sink context, when it’s in sink context.

I made a pointless webpage

Rakudo Star 2013.02 released

Perl 6 changes - 2013W08

Perl 6 specification> Failed matches result Nil, not (Match). id='new_features'>New features>

STD.pm6> Added a warning for smartmatch of (Match) or (Nil).

Niecza Perl> Type placeholders (like Int, which is null, except Int typed) now have their .gist method show their name within parens to suggest being undefined. For example, (Int), (Mu).

t1: Tell knights from knaves

friendly interactive shell

Parrot 5.1.0 "Zombie Parrot" Released! by Bruce Gray

NQP on JVM gets Grammars, Multiple Dispatch

Perl 6 is now half as old as Perl

Bottom-up vs top-down

Second system syndrome

Rubber, meet road

So, how're we doing?

What remains?

Conclusion

STD.pm6>

You cannot use `set` or `bag` to mean empty set or bag.

If you really meant that, you are expected to use `set()` or `bag()` instead.

id='new_features'>New features>

Niecza>

`Set`, `KeySet`, `Bag`, and `KeyBag` have `default` method.

Just like in Rakudo, except with Niecza.

Niecza Perl>

`Pair.perl` can now deal with the cases when `Pair` is the key.

Doing `((a => 'b') => 'c').perl` used to output `"a" => "b" => "c"`, what actually meant `(a => (b => 'c'))`. `.perl` now outputs correct code in that case.

Implemented Hash.ACCEPTS.

Perl 6 specification>

`DateTime` doesn’t accept `Callable` timezones anymore. 5692ced333

Rakudo Perl>

Renamed `IO` to `IO::Handle`. `IO` itself is a role. 2dca6ed950 37c9956502

`DateTime` doesn’t accept `Callable` timezones anymore. 71ea142406

id='new_features'>New features>

Rakudo Perl>

Implemented `Cool.lines`. 0d1d1194a4

Perl 6 specification>

Removed `IO::Dir` and `IO::File` classes. fb25fa475854

id='new_features'>New features>

Perl 6 specification>

`Parcel`s are allowed to have 0 elements.

Niecza Perl>

Added special cases for `big_pow` when either of values is `-1`, `0`, or `1`.

Perl 6 specification>

Renamed `weekday-of-month` to `day-of-week-in-month`.

Rakudo Perl>

Fixed signature of `Pod::Block.gist`, so it works properly with `multi`s.

`for` loops are eager now.

Added `Buf.perl`.

Fixed `Buf.gist`, so it works properly with `multi`s.

Perl 6 specification>

Added many `IO::Path` methods: `.contents`, `.is-absolute`, `.is-relative`, `.absolute`, and `.relative`.

`.path` is now NOP that returns itself. For its previous behavior, use `.Str`.

Changed `Numeric.conjugate` to `Numeric.conj`.

Rakudo Perl>

`+(23..2300000000)` is optimized.

`--help` reports `-I` and `-M` options.

Perl 6 specification>

Added `X::Range::InvalidArg`.

Rakudo Perl>

Removed lots of `pir::` usage.

Multi dispatch is now implemented in NQP, not C.

Released Rakudo 2013.03.

Niecza Perl>

`Empty.gist` returns nothing.

Perl 6 specification>

`Rat.perl` returns decimal number when denominator is normalizable to a power of 10.

Added FALLBACK methods.

Included Failure in smartmatch passthroughs.

Loosened relationship between `Nil` and `()`.

Added `List.permutations`.

`rat64` type has unsigned denominator.

STD.pm6>

`Nil` isn’t suggested for empty list in error messages anymore.

Niecza Perl>

`Rat.perl` returns decimal number when denominator is normalizable to a power of 10.

`CommonEnum.gist` returns `(Type)` instead of `Type()` for undefined.

Restored `infix:<cmp>` sorting for different types.

Failed matches return `Nil`.

Perl 6 specification>

Inside of `try` is considered eager context, unless `try` is called in sink context, when it’s in sink context.

Perl 6 specification>

Failed matches result `Nil`, not `(Match)`.

id='new_features'>New features>

STD.pm6>

Added a warning for smartmatch of `(Match)` or `(Nil)`.

Niecza Perl>

Type placeholders (like `Int`, which is null, except Int typed) now have their `.gist` method show their name within parens to suggest being undefined. For example, `(Int)`, `(Mu)`.

Rakudo Perl>

`$_` is writable in `-n`/`-p` code (one-liner).

`eqv` is faster for `Buf` objects.

Configure script fails on unknown command line options.

Rakudo Perl>

Constant folded `^..^` (infix) and `^` (prefix) operators.

Perl 6 specification>

Added `DateTime.delta` and `Date.delta` to the specification.

Rakudo Perl (in Rakudo 2013.01)>

`** 0` in regexpes works correctly matching nothing.

Rakudo Perl (post Rakudo 2013.01)>

Compiler hints invalid variable names. For example, typing `$variabel` when you already have `$variable` declared hints that variable.

Niecza Perl>

More meaningful error message is thrown when `printf` tries to consume more arguments than available.

Perl 6 specification>

`-->` (return type) acts more like a normal subroutine parameter.

Added `squish` method, working like `uniq` utility in UNIX - it only removes elements if it is duplicate of previous element.

Adjusted precedence of set operators.

STD.pm6>

Adjusted precedence of set operators.

Niecza Perl>

Adjusted precedence of set operators.

Rakudo Perl>

“SAFE” setting is now removed. You’re supposed to use “RESTRICTED” setting now.

id='new_features'>New features>

Niecza Perl>

`infix:<->(Instant, Instant)` now supported.

Bitwise operations support big integers.

Perl 6 specification>

`uniq` method/subroutine, existing for long time in implementations is now mentioned in specification (with `===` semantics).

Rakudo Perl>

Use of suffix regex modifiers (P5-ism) in `s///`, like `s////g` is now detected.

Rakudo Perl (in Rakudo 2012.12)>

When disambiguating hash/block, and `$_` variable is used, Rakudo assumes block. This could break your code if you were expecting a hash.