Julia: A post-mortem

Why I no longer believe my favourite programming language will save the world.

A little over half a decade ago, I came across a wonderful project: a programming language called Julia that was going to revolutionise data science and technical computing. It was going to run like C, seamlessly integrate with Fortran, do things R does without its clunkiness, look and read like Python and be homoiconic like Lisp. We were all going to heaven, and that right soon. The language wars would end, we would finally get a lingua franca for anywhere code performance mattered, Julia would take over the TIOBE Index, and we’d all be home for tea and medals.

I was so enthusiastic, I learned all there was to learn about Julia, and even pitched a book to Manning, who graciously accepted Learn Julia into their Early Access programme. Interest was lukewarm at best, which should have been a warning sign. Eventually, Manning felt there would not be a sufficient market for the book – and I don’t blame them. Much of the book’s contents are now available on Github, and are a pretty good introduction for getting into the language.

Fast forward to 2020: Julia is currently on the 34th position (narrowly missing the first tercile) on the TIOBE ratings. More people use FoxPro, a database language that reached end-of-life almost fifteen years ago, than Julia. Vastly more people use Scratch, a language that for all intents and purposes is a ‘learning’ language. Almost twice as many developers use COBOL and Fortran as use Julia, and both of those languages are nearing the retirement age in most Western countries.

Something has clearly gone wrong. While Julia is not dead, and there are some very committed developers who expand the ecosystem, it is clear that the initial promise of Julia is unlikely to be realised. Instead of becoming the language, it will become just another language. Here are my thoughts why that’s the case.

It’s hard to dislodge an incumbent

Programming languages compete not just for users but also for people who develop it, evangelise it, advocate for it. Almost all major modern programming languages are openly developed, if not for the core language, then definitely for the ecosystem surrounding it (about which more later). Languages depend on public engagement, and the public wants to back a winner – as much psychologically as they do not want to spend their weekends writing code for free for a language that will be forgotten before TikTok memes go the way of Vine.¹

¹ Do people still remember Vine? I remember Vine. Man, I feel so old.

² R is quite different from Python in that it makes a uniformly horrible ETL language. It was designed as a ‘batteries included’ statistical toolbox, and the further one strays outside that realm, the worse it gets.

³ This will inevitably help you gauge my age.

⁴ In the last case, please look for a better institution to transfer to.

In the field for languages to work with data – from data science through machine learning to just munging tabular data –, the incumbent is Python. There isn’t really a close second, except perhaps for R.² The former has remarkably low barriers to entry. Before Scratch and other ‘entry’ languages came around, it used to be the ‘gentle introduction’ to programming for many, along with BASIC.³ Python and R are some of the most widely taught languages to non-computer scientists. If you are in any science and they make you learn a programming language, chances are it is going to be Python or R, with a smaller chance of Matlab or SAS.⁴

All in all, this makes it shockingly difficult to dislodge an incumbent. It’s a self-reinforcing process: new developers, given a choice, will gravitate towards the language that comes with ‘batteries included’. Few languages have as strong an ecosystem as Python – anything that’s ever been calculated has probably been implemented somewhere in Python. Consequently, the contenders will have fewer people to catch up to the incumbent. And on and on it goes.

Ecosystems

In the heroic age of computing, when Fortran was still young enough to not have to worry about putting money into its Roth IRA, people expected to write most of their code, or at worst copy/paste it. In 2020, using packages is as ubiquitous as using functions. Nobody wants to reimplement the Laplace distribution for the n-th time. Consequently, a lot of what we do as programmers depends on the package ecosystem. Knowing Python is not enough to write good Python for a real-world application. In fact, it’s not necessary to know the internals of Python in any depth compared to how well one is expected to know the guts of Pandas or NumPy. As a data scientist, you can safely get by without understanding how Python does async – but you better know the difference between a PyArrayIterObject and a PyArrayMultiIterObject.⁵

⁵ I’m sure we all know all there is to know about IterObjects, but here’s a link to the NumPy reference to help you, uh, explain it to your dog.

Ecosystems are self-perpetuating (see previous section). Bigger ecosystems, subject to some boundary condition, tend to beget even bigger ecosystems. This stands to reason – it’s easier to build a house if you don’t have to make a stone chisel to make a tenon joint to make a brick mold, as this gentleman demonstrates. Nobody (except the yak wool lobby and, potentially, some yaks) likes yak shaving, and an ecosystem is really just a fancy way of saying the yaks have largely been shorn and shaven, and you won’t have to bother with them. That’s the kind of stuff that makes people want to write increasingly intricate code.

Python has one of the best data science ecosystems, if not the best. It has sufficient organisation (largely via NumFocus and to a lesser extent, Anaconda) to overcome mundane issues like, say, funding, but open enough to be still enjoyable for people who hate bureaucracy.⁶ It’s a tall order for any language, no matter how good, to beat this advantage. Which leads us to my next point.

⁶ The institution, not the season.

Cui bono?

Lucius Cassius, who was consul of Rome in the 2nd century B.C. and a notorious hard-ass as a judge, was known to ask who would benefit from a particular crime when looking for the culprit. Dollars to donuts, the people who benefit from a crime tend to be statistically the most likely to have had a hand in it. It’s an equally good principle, when considering the viability of a programming language, to ask whom it benefits.

Take R, for instance. For all its G–dawful syntax⁷ and wide range of horrific features, it is a great language for people who are looking for something that comes with all the statistical bells and whistles.⁸ It has ggplot2, which is quite probably the best static data plotting library ever written, and some rightly argue that Python still hasn’t caught up completely (although seaborn goes a long way towards that point). It has the Tidyverse/Hadleyverse, which is overall a pretty neat bundle of packages. There’s a clear profile of a user there – someone who needs to do some statistics, stat (no pun intended).

⁷ Here are some tips to make it suck less.

⁸ Many of whom have also previously experienced SAS or STATA and thus cling to R like a drowning man clenches a life raft.

Julia’s target user is harder to define. I have struggled with this while writing Learn Julia. Is this a general purpose language, or a technical/statistical computing language? Is this a language that people can easily get started with, or one that places a lot of emphasis on powerful features (mostly deriving from homoiconicity) that require a good bit of understanding? What’s the unique selling point?

Julia has a lot of great features – I love the homoiconicity related features, and I love built-in parallelism as a first-class citizen and I think C-like speed is really cool. The problem for Julia is that I can get all these features in Python (e.g. JIT compilation with Numba for speed), and get a library for every imaginable use case including antigravity, dynamic typing, a syntax pleasantly unconcerned with niceties and a range of tools from IDEs to logging platforms that’s exhausting to even think about.

In the end, this is the key point for me. The problem with Julia wasn’t that it wasn’t good. The problem was that it wasn’t good enough to make itself worthwhile in the face of Python &c.

Coda

These days, most of my work is in Python, with a smattering of bash scripts. I continue to follow Julia, but my laptop is probably at least half a year (if not more) out of date. I have poured a lot of time and energy into learning, understanding and popularising Julia, but I don’t see it living up to its initial promise anymore. There are some important, groundbreaking things being done with Python, largely in the machine learning field, and Julia did not make it past the fringes of that field.

Nevertheless, it is an important object lesson to anyone embarking on a new idea in software development. It’s hard to beat an incumbent, and even harder to do so without having a large target user community you can capture with a compelling use case. In the end, code doesn’t make software – people and communities do. And for Julia, that – or rather, the lack thereof – has made all the difference.

Edited To Add…

A few hours after this went up, I got hackernews’d, and my poor one vCPU machine running this site died a sad death. On the upside, it’s back up again with some additional beef, and there are some great comments on the Hacker News article. Let me respond to a few here.

There’s Python the language and “Python the dsl for tensor frameworks”, the former has arguably lost some ground/mindshare to newer, less “hobbled” languages, and the latter exists not by virtue of its own strengths but as the convenient vehicle for frameworks written by 2 enormous corporations.

– FridgeSeal

The whole comment of FridgeSeal is worth reading, even if I disagree with its gist. They make a great point: a lot of Python’s popularity is due to it being used as ‘the DSL for tensor frameworks’. The issue is that the consequence of Python being used as this ‘DSL for tensor frameworks’ is a better ecosystem, more support, great packages and a pretty much 100% dominance at the cutting edge of ML research. Reading any new paper about ML on ArXiv, I can be about 90% sure that there is a Python implementation somewhere that I can test drive, and if not, one is bound to come along soon. Tools like Jupyter Notebooks have had a massive impact on this, but let’s not forget that the first two letters in Jupyter stand for Julia.

Julia is a language that looks very simple, but the more you use it, the more you realize how complex and unpredictable it is. I think that Union types do more harm than good (why would you want a function to return Union of int and float instead of compile error? It totally slows down the program) Array{Number} is totally different from Array{:<Number}, and shouldn’t be allowed, as it is inefficient. 1-based indexing was a mistake, and I have seen it emitting inefficient code in the PTX compiler. But the worst and hardest part is the rule/heuristic for multiple dispatch: it’s so complex, that it isn’t even documented. It should probably throw more errors and be predictable instead of trying to be so smart.

– xiphias2

I am not going to get into the 1-vs-0 based indexing problem, because I treasure my head staying attached to my neck. However, xiphias2’s comment on multiple dispatch is spot on. I maintain that the success or failure of a programming language in 2021 is no longer intrinsic to the language itself (my whole ecosystem point), but at least partially contingent on extrinsic factors and, perhaps, even pure dumb luck. But if we were to look at issues with the language itself – why Julia doesn’t deserve to be the leading language rather than why it did not end up becoming the leading language, the difficulties of multiple dispatch would be pretty close to the top of my list.

There’s a very interesting article by DJ Passey that claims multiple dispatch is how Julia beats Python. I contend that if your Python code involves type contingent control flow, you’re writing bad Python, and you should feel bad. If your Python code is slow because of that, you should doubly feel that. The following is horrific Python I have seen a little too often for my own good:

def func(arg1: Any) -> Optional[float]:
    if isinstance(arg1, float):
        return arg1**2
    else:
       return None

And that’s not even the worst implementation of this error (I’ll let you guess). It’s better to ask forgiveness than permission is a core Pythonic principle. The kind of code that DJ Passey claims will be accelerated by Julia’s multiple dispatch vis-a-vis type contingent control flows in Python do not (or at least should not) exist in good code bases. On the other hand, multiple dispatch can get weird at best, terrifyingly unpredictable at worst. Any developer who can handle multiple dispatch without setting themselves on fire by accident should be able to write Python code that doesn’t suck.

I actually have very similar experiences. I want to like Julia and I look at it from time to time. The hype is always that it allows one to write C-speed code with the simplicity of python, without the idiosyncrasies of numpy. But every time I look I find just as many little things to look out for if one cares about performance, don’t use abstract types is another one. I understand why this is the case and I don’t have an issue generally with it, but considering that I know Python well, and know how to speed up Python using numba, pythran or cython, I don’t see what I would gain by investing in Julia. – cycomanic

Believe it or not, this is actually the most frequent complaint I hear from potential commercial users of Julia. Heck, I want to like Julia! I most definitely want something better than Python. The problem is, putting up with Cython and Numba and all their collective unpredictability is still preferable, in an operational context, to dealing with Julia. And that’s a problem. If you’re writing code for a living, you need to be able to justify any time you spend learning a new language – to yourself if it’s in your free time (you should be playing with your children/wife/dog), to your employer if it’s on the clock (most employers sadly expect developers to spend all their time producing new code and very little time learning new things – I’m lucky to work at a place you could consider an exception). It’s non-trivial to justify Julia in its current state.

Is the post only based on TIOBE? The same index that currently ranks JavaScript below Visual Basic and SQL below Assembly? That ranking is off by so much that anyone who takes it seriously loses at lot of credibility from the start. – superbcarrot

Nope, and I’m sorry if I’ve accidentally given that impression. TIOBE is a rough, inaccurate but quantitatively objective(ish) proxy for my own experiences (which are more accurate but definitely not quantitatively objective, nor do I know enough about every single language to be able to somehow rank or compare them). For much of the last five years or so, a key part of my day job was to make some very critical business processes run faster, from healthcare through financial services to fraud detection. To put it this way – my concerns with Julia are largely expressible through a proof by contradiction: were it possible to make Julia do what was initially claimed for it (replace Python with something faster and better), I would have seen many more success stories of it doing so in practice. I’m not arrogant enough to assume that if I can’t make it work, nobody can – I’ve met many people who have tried just that, many of whom are significantly smarter than I am, and none fared with much success. I admit to the potential existence of a sampling bias here, of course.

By calling it post-mortem the author already showed his level of stupidity (or absence of morality). Julia is a new language that have to compete with languages that accumulated a huge ecosystem of developments and continue to grow. Of course Julia is losing in comparison right now. – dandanua

There are quite a few comments that obsess – in my opinion, unduly – over the title. A post-mortem does not necessarily mean (outside the medical context, anyway) that something is finished (yes, it does mean that literally, but not semantically, i.e. how it is used in practice). A post-mortem is the examination of something at an endpoint (including, potentially, success – after-action reviews of projects are often termed post mortems even if the project was a resounding success). As far as I am concerned, I do not see myself spending a lot of time on Julia in the future (of course, I am happy to be proven wrong – I don’t think anyone would be happier if Julia defied my dim expectations and turned out to be the future indeed!). In that sense, it is more or less a bit of closure as far as my active engagement with Julia is concerned (though as said, I do not foreclose looking into it later in the future!).

“Give it time” is not how programming languages or software projects grow. If anything, it’s closer to “build it, and they will come”, which of course highlights a fundamental difficulty in open source, viz. that ‘they’ and the people who ‘build it’ are the same bunch. That makes for an untidy circular process. If we stopped being open to predictions based on the fact that the future has not played out yet, we would overall forfeit the epistemic validity of any prognostication about anything. If you are at 5,000ft and descending rapidly with a failed parachute, you’re not exactly dead – but it is rather unreasonable to ferociously contest any assertion that you might not exactly be long for the world, because ‘give it time’ (hey, the ground may just end up being made of marshmallows). In the land of cold, hart realities, sadly that is not how it tends to turn out.