Damian Conway - interview

From LXF Wiki

Behind Perl 6

Perl is one of the most widely used programming languages in existence. This is in part thanks to the productive relationship between its founder, Larry Wall, and his chief co-conspirator, Damian Conway.

You can't fail to miss Damian Conway in a crowd of developers. He's the guy with the grey trousers, the black T-shirt and a gaggle of Perl hackers hanging off his coat tails. He's the Yin to Larry Wall's Yang, the devil's advocate. Larry and Damian run the forge from whence the mighty Sword of Perl was wrought, and from where it is being reworked into Perl 6. The future of geek-kind depends upon it. Graham Morrison was set with the task of meeting Damian and finding out exactly what's happening with Perl 6.

Linux Format: What occupies your time at the moment?

Damian Conway: About 50% is occupied with just earning income now. So, I am spending my time preparing classes, preparing presentations, and then travelling around the world and delivering them. Another 20% of the time perhaps is in finding work: finding contracts and people who want me to come in to do the work, and just during the sheer logistical arrangements that are required to make that happen.

The rest of my time is focused almost exclusively on the development of Perl 6. At this point that really consists of working with Larry on the final touches to make sure that they all make sense and work well together, and helping him to solve the last remaining pieces of the puzzle.

Interestingly enough, this endgame phase of Perl 6 is a little bit harder than the overall design phase, because now that we have so much in place and we're so happy with most of it, it's the last few things which need to be as good as everything else is that give us fewer options about how to design them. So it's finding the right solution to those things that I find is the challenge and interesting now.

LXF: Is there an advantage to working in that way?

DC: It's a challenge but yes, I think there is an advantage to it. And that advantage is that we are not beholden to anyone's particular agenda. If we were working for a company, that company would have goals for Perl 6 and we as employees of that company would have to ensure that Perl 6 met those goals before it did anything else. We're working for the community, so we can focus on the community's goals for Perl 6, which are incredibly more diverse than any organisation, no matter how large, could possibly have.

People have said, "We've been designing Perl 6 for six years. Why isn't it out already?" Well, I think that has actually been an advantage ­ that has given us the time to work out exactly what was going to be the best thing for the next 20 years. It hasn't been good from a PR point of view, but in terms of the language that we will be delivering (and in fact we have already delivered), it is I'm sure as good as we could possibly have made it. When you think of the calibre of people that have been involved in it, that means it's incredibly good.

One of the talks that I give more often than anything else is, "Here's what's in Perl 6." The excitement that I see in people's faces once they've even just seen half an hour of some of the things that we're bringing to the language; the, frankly, lust that I see ­ "Why isn't it here now? Why can't I go home and use it now?"... I think that one of the great things to have happened in the last two years since we last spoke [Interview, LXF62] was that we've had the whole Pugs project with Audrey Tang spearheading that, so you now can go and do that [use Perl 6 now].

When I show something, the last slide I'll show is the URL for Pugs code. People go away and download it, and are playing with actual Perl 6 code within half an hour or so. That has generated a lot of excitement, a lot of energy. And a lot of frustration too, because people love the fact that they can do it, but it's not really designed to be production-ready and high speed in the way that Perl 5 is.

But by and large I'm extremely pleased with where we are and I'm extremely pleased that we've taken the time to get it right. A lot of that comes back to the fact that we weren't on any corporation's timetable.

LXF: Has the development process over the last six years changed or realigned with things like the rise of PHP or Ruby?

DC: I'm not entirely sure that either of those two that you mentioned have influenced it in that particular timeframe. Ruby certainly has influenced Perl 6 quite lot; I'd never take anything away from what Matz [Yukihiro Matsumoto, Ruby's author] has done with Ruby. A lot of the stuff that he took out of Perl and rejigged in Ruby has been fed back into Perl. So I'm sure if Ruby hadn't existed Perl might not be quite so object-oriented as it is in Perl 6.

To be brutally honest, most of the languages that have arisen over the time period that we're talking about haven't really been all that different from the languages that came before them. I think what has changed our thinking [is that] some of the paradigms have become much stronger, and some of the understanding that we've gotten into the process of programming ­ what works and what doesn't work. To take an example from each of those, aspect-oriented programming has kind of burgeoned over that period. People realised that while OO had much to offer, a language wasn't going to be complete if there wasn't a mechanism which was orthogonal to the class structure that allowed you to do that mixing in of features that are needed but don't really belong in the hierarchy. And the great thing about that was we've been able to take that and run with it. We've said, "OK, Perl needs to support aspect-oriented programming, so we need to have the mechanism built right into the core of the language ­"

LXF: Is that where Pugs fits in?

DC: No, I don't think so. I think what fits in there is a component called PGE, which is the grammar engine. Where it really fits is the way that we've designed Perl 6 grammars so that you have the full parsing capacity of the very best breed of grammars that you'd find out there in the wild. Previously all of the grammatical constructs like Lex and Yacc in the very early days or more modern things like Antlr ­ they've always been separate from the language. They've always been a tool that you apply to the language or you apply to create the language.

We knew Perl had the best regular expressions on the planet, but it didn't have the next layer up: the grammatical layer where you could put those together to build language interpreters easily. So we put it in the language.

That kind of thing ­ the rise of our understanding of what was required to do good development and the rise of the various paradigms of language facilities (even though most of them were actually provided in most of the real-world languages) ­ allowed us to, over that half decade, say, "We can see that these are going to be good things and they're going to be important in the next 20 years, so it's not enough that we make Perl 6 able to do these things ­ it's got to support them."

That has been informing a lot of our thinking about how you design the language. It has to be designed so it's easy to test, easy to parse, easy to refactor. Yes, you need to test the software; and you don't just need to test that the functionality of it is correct, you need to test that the coverage of your test suite is doing that. So you need to have a language where you can have an internal representation of all the paths in the program to see that, and for that you've got to have an internal representation of the program, in the program.

LXF: How do you know when to stop?

DC: What an excellent, excellent question. [Long pause] You know, I've never been asked that, and it is the key question. How do you know when you've got enough in there and it's not too much?

One of the things that we've done over time is we've very carefully avoided putting in features that only serve one purpose. If something was going to go in to Perl, it had to solve a couple of problems. It had to make something that was already in the language better, and it had to provide extra functionality as well, so there was a net gain in putting it in. Because you had to offset the very problem you're talking about, which is, "Hang on, we've just made the language big."

But at a larger scale you're quite right, that there's this perpetual fountain of new techniques coming up in the programming world, both in terms of programming theory but also in terms of practical development techniques. So how do you decide when you just have to lock it down? To be perfectly honest, I think the answer is when you get tired! No, I think that once you're deep enough into the design of a language, it's fairly clear you can get to a point where you say, "Yes, it's all hanging together now: it now feels like one continuous cloth." And when that happens, that's the time to stop. Because if you go much beyond that you're going to be bolting extra things on that don't really need to be there at that stage.

That's the answer to your question: we stop when we have a feeling that it's all fitting together and we're covering all the bases that we wanted to cover. Sure, we're not covering this new thing that's been out for three months now, but the other thing that we've done is that we think we've made the new version of Perl sufficiently extensible that the stuff that's coming up now, in two years' time, in five years' time, will integrate very easily into Perl 6 so there may not need to be a Perl 7.

LXF: Is there much disagreement among developers as to what is important or where you should stop?

DC: No, no, we are all entirely in accord at all periods of time about everything. [Laughs] No, I think it's extremely healthy that there is a lot of disagreement about "what's the best way of doing things?" or "does this belong here or not?" If you watch the various mailing lists, either the core designer list or the broader language list where we have a lot of talented people checking our decisions and looking at them from a different perspective, you do get a lot of disagreement on the right way to do things.

So I think the disagreement that's gone on there has been incredibly valuable. Ultimately we also benefit from the fact that Perl is whatever Larry says it is.

LXF: That's a good thing.

DC: Well, you have to have someone with whom the buck stops, otherwise you never get anywhere at all. The lovely thing about having Larry as that person is that Larry is without ego in terms of this design. He wants to take good ideas from other people, and I've seen him adopt wholesale great ideas from someone for whom it was almost their first post on the [mailing] list. He never goes by who the person is who is suggesting the idea, he always goes by whether it makes sense.

LXF: Do you think that after Perl 6, Perl development will become more community-oriented?

DC: I think that is very likely. It's certainly what you see with Perl 5. Ultimately the Perl 5 porters now do most of the development on Perl 5. Although Larry still has his executive veto, he very rarely uses it. And he's usually very supportive of things that people want to do, because the Perl 5 porters know the codeso well.

LXF: What are your favourite parts of Perl 6?

DC: From a biased point of view, I think one of my favourite parts is the junctions feature. That's because I often think that junctions will be my only contribution to computer science ever. Junctions are these kind of quantum superimposed values that you have in Perl 6, so you can have a value that's equal to the value of all of 5, 6, 7 and 8 simultaneously. That kind of thing can be incredibly useful because if you can have a long list of things and you can superimpose them down into a single value, which you can then do a single test against, to see if you're equal to any of them.

If you had a list of unacceptable passwords, for example, you could create a single scalar value that is equal to any of them. Then if you say, "Give me your password, and if it is equal to this single scalar value, then we'll know that it's equal to any of [the unacceptable passwords]." The point is that because of the structure of it, we can at least theoretically parallelise that test. Because it knows, you just have to check whether you're equal to any of these, and they're completely independent, and you have the ability to short circuit if any of them proves otherwise. It's kind of the royal road to parallelisation. You can actually physically write in Perl: if $password eq any(@existing_passwords). And you get the ability to parallelise that; you get the ability to write it in basically natural English.

I'm very proud that I was the originator of that idea and I'm delighted by the way that it's been taken and improved by the other members of the team before we put it into the language.

In terms of favourites, it's the little things that I really love. It's the fact that we now have a `print line' function in Perl because 70% of the print statements I do I have to put a newline on the end of it. That drove me nuts! Now there's a little say function, and you just say say $whatever and it puts the newline on for you. Something like that seems trivial and you could put it in any language, but very few languages have put it in in that nice way.

LXF: Are you pleased object orientation is more prominent in Perl 6?

DC: Yes, I guess it was poor of me not to mention that. In terms of the practical utility of the language, the fact that there is a formalised object-oriented class structure now means you can do the kind of OO programming that's easy to do in Java, C# and C++, and now it's just as easy to do in Perl and just as clean... In fact we just added in a whole other thing that you don't get in all these other languages. I think that's going to make a very big difference. I perhaps don't think about it as much as I might do otherwise because I don't actually do a lot of object-oriented programming any more. But I know that for those that do, this will probably be the killer feature of Perl 6. The fact that now I can write proper classes with properly declared attribute components and methods ­ of all the things that we've talked about ­ that will probably impact more people more positively than anything else. So yes, I am very excited about that, but I guess I don't even think about it now because it's so fundamental to Perl 6.

LXF: What are the challenges that the open source community will face over the next five years?

DC: [Long pause] I think the challenge for open source over the next half to one decade is that we have to compete without becoming the competition. We have to have faith that the philosophical underpinnings of the way that we go about doing what we do are actually sufficiently correct that we'll survive if we do keep on doing what we do. We have to find ways of moving into the mainstream without adopting the trappings of the mainstream ­ without being trapped by the accepted practice. It doesn't surprise me to see companies closing down their source and not being as open as possible, and I think that's a failure of nerve more than anything else. I think it's eminently clear that enough start-ups have demonstrated that yes, you can proceed with an open source philosophy and stay open source, and still make a profit.

Open source really is becoming mainstream, now. Things like Firefox have done that in a huge way. Linux... maybe not even to the extent of Firefox, because Firefox is the interface where the non-geek meets geekdom every day, and an increasing number of people are doing that. I mean, we see Linux market share increasing as well, but perhaps not in the same overriding way. You can run Firefox on Windows so it's a little way to get in there. LXF