Python "Surpasses" Perl?
There has been some buzz going around about Python "surpassing" Perl in terms of "popularity":
http://mail.python.org/pipermail/python-list/2008-November/517191.html
However, they're basing their results on the TIOBE survey, which is, in my opinion, even dodgier than my own, at http://www.langpop.com . It's really difficult to pin this sort of thing down, but I think that by utilizing more data sources, my own numbers bear some relationship to what my "ear to the ground" tells me about what's going on, and certainly more than TIOBE's, which place D, Logo and Lua above things like Tcl/Tk. I know Tcl is not the hottest thing out there these days, but...Logo? D is gaining in popularity, but how many companies are hiring D programmers, how much code is out there written in D? I'm not saying that D is "bad" or "not worthy" because of that - quite the contrary, I think it's on its way up, but there are a lot of things you have to look at in order to take a stab at creating some meaningful numbers, and even then it's important to remember that popularity isn't everything! In any case, picking a precise moment in time when one language "passes" another is also a bit more theater than science, in my opinion.
Getting back to Perl and Python, I thought I'd look at some numbers of my own. Unfortunately they only go back a year, but they already give us an idea of what's going on. In order to concentrate on something fairly concrete, let's look at Freshmeat projects:
Perl is still clearly more widely used. However, I think there's an important distinction to be made in terms of popularity. Since many systems that function ok stay around for years, there's a big difference between what's being used for new systems, and what's out there already. Perl has a lot of code out there already, but how fast is it growing in comparison to Python?
Python's definitely got the edge - it's growing, whereas Perl was very nearly static. To me that does indicate that Python has got momentum right now, where Perl is sort of coasting.
Trackbacks
Use the following link to trackback from your own site:
http://journal.dedasys.com/trackbacks?article_id=2116
about {{count}} hours later:
I've noticed Python momentum too. In fact I've written my very first program. There's something about Python that seems to actract Java developers.
My question is: will tooling for dynamic languages ever be on par with those for static ones? Is that even theoretically possible? How can people write large enterprise software in a dynamic language, which they seem they can do? (youtube is written in python)
about {{count}} hours later:
"How can people write large enterprise software in a dynamic language"
By writing a smaller program, in less time, that does the job ;-)
about {{count}} hours later:
This stuff has been discussed a lot elsewhere, but I don't think there's quite such a strong need for tools for dynamic languages. Think of writing code with notepad, for instance... it's the wrong tool for the job, but lots of people probably get started that way before the learn that it's wrong. With Python or Ruby, it's sort of doable, albeit painful. With Java and all its classes and methods, it's going to be painful. Same with Lisp and counting how many parenthesis to close at the end of a big block of code. Languages that 'scale down' are languages that can be written with bad tools, and don't require great tools to be productive with.
about {{count}} hours later:
i for one am a recent adopter of python (coming from perl) and have been thoroughly enjoying the experience. my feelings on this matter, could best be described as, perhaps... poetic - http://web-poet.com/2008/12/07/the-lure-of-python/
1 day later:
Like u say using Freshmeat in this way is "fairly concrete" so I don't think u can rely on your figures for Perl or Python being statistically representative elsewhere.
/I3az/
1 day later:
@draegtun - actually, I think freshmeat is a great indicator of what's going on in the open source world, and I think open source in general is often where it's at in terms of new and interesting stuff, because open source is often code people choose to work on, so if they're choosing to do Python, well then that's telling us something.
1 day later:
Freshmeat isn't a good indicator. Its a bit like me saying that because I get more Safari browser hits on my blog then there must be more Macs to PC's out in the wild! Unfortunately Freshmeat & my blog are eschewed universes ;-)
All u can say from your figures is "On Freshmeat... it seems to indicate that Python has got momentum right now, where Perl is sort of coasting".
I just checked for a few key Perl apps, frameworks and modules on Freshmeat and got nothing back. Both Freshmeat (and for that matter Ohloh.net) only show a "subset" of what is out there in the Perl (and Python) world.
For more accurate Perl figures then CPAN is (probably) the best place to look.
/I3az/
1 day later:
draegtun: I've been using freshmeat for nearly 10 years, and it seems to me as if it's a fairly neutral place in terms of languages used for the projects listed, as it's all about announcements. It's also very widely used, certainly more so than your blog or mine, by several orders of magnitude, which is going to smooth out some of the wrinkles. Incidentally, I ran the numbers for Amazon, too, and it shows growth of 1.38 for Python vs 1.13 for Perl.
In terms of CPAN, that's completely irrelevant as it's only for Perl. I use data sources that are language agnostic.
1 day later:
I've been using Perl for over 13 years now and I've never downloaded a Perl program or application from Freshmeat. And this despite the fact Freshmeat use to be my browsers homepage for many years ;-)
I'm afraid the Freshmeat comparison becomes eschewed because Perl & Python have completely different ecosystems and so their communities act differently causing statistical deviation in the outcome your observing.
Perl as the well established CPAN so it doesn't need or use Freshmeat like most languages so the eschew effect is probably far greater. So if u want Perl statistics on number of apps, modules, authors, lines of code, etc then CPAN is the place to go. For Python there may also be a better place than Freshmeat.
BTW.... Just checking Freshmeat and I'm amazed to find the some of the programs listed under Perl aren't Perl at all. And the same goes for Python. In lots of cases they're listed simply because there is a library or API available for that language.
This means the figures cannot even be relied upon in themselves let alone in comparative terms within the confines of a Freshmeat universe.
/I3az/
{{count}} days later:
(BTW, I don't think "eschewed" means what you think it does).
You're correct that there are various things that might cause problems, but you're not demonstrating that Freshmeat does exhibit those problems. For instance,
Well then that points to more popularity for that language, doesn't it? That is, after all, what we are after. If you can demonstrate that that happens a lot more for one language, and not others, that's a good sign of bias, but if it tends to happen for all languages, then it cancels itself out, doesn't it.
{{count}} days later:
re: eschewed - Sorry that's a colloquial term we use in Market Research industry here in UK (at least in some of the companies I've worked for) and isn't a direct reference to "eschew" definition but a "slang" of something else! However I think u get the meaning ;-)
re: Library / API - Yes u can argue your point but u can also argue it in other ways which only shows it cannot be relied on to be used as a statistical mark for comparison.
I'll give u an example... Project A came out a few years ago with Perl bindings. Recently Project A added Python bindings. Python obvious got popular enough to get bindings but how is Perl going to improve on somethings its already got.
So the point is neither of us can demonstrate that Freshmeat does exhibit these problems or not. Without statistical proof of what Freshmeat is then its a flawed sample to use it outside of the Freshmeat universe.
/I3az/
{{count}} days later:
Java is Dead! Long Live Python!
{{count}} days later:
draegtun is spot on.. almost Perl libraries and many applications are on CPAN there is little point faffing around with Freshmeat when you have CPAN.
I should know I have several open source projects, and keeping freshmeat, sourceforge and the like up to date is a real chore and I can't be bothered with it, when CPAN takes 30 seconds to update, especially as you can install from cpan with a single command.
I'm also not quite sure how growth of 1.1 percent is coasting while 1.4 percent is skyrocketing as you claim.
I had a look at langpop, I noticed that in the last 'normalised' charts python came out ahead of perl despite being well behind in all but one of the charts being normalised. bit of a skew there I think.
You might be interested in the metrics page on the perl 5 wiki (that's grown faster than the Python one - easy to grow faster when you're smaller ;) at http://www.langpop.com/
{{count}} days later:
TeeJay: no one talked about python "skyrocketing" - you need to tone down the defensiveness. What I said is that Python has more momentum than Perl right now.
Even if you don't use Freshmeat yourself, someone does, because there are more Perl projects there than Python ones. To me that's indicative that people are using it. CPAN has been around longer than Freshmeat (I remember using CPAN in something like 1997), so clearly people have gone and posted code to Freshmeat despite the existence of CPAN. And keep in mind that other languages have other systems as well, Python isn't unique in that regard.
So, what is interesting about Freshmeat is not that it has every scrap of Perl or Python or Cobol or whatever ever written, but that it is a neutral place where many authors do publicize their projects.
In terms of biases it does have, it's pretty clearly biased towards open source projects, as that's the environment it came out of. That hurts languages like Cobol.
Furthermore, it turns out there are some issues with the delicious results, which is where Python is above Perl - when I rerun those, I bet that Perl will once again be on top on the normalized results. However, I stand by my assertion that Python does have more momentum.
{{count}} days later:
@david: re: TeeJay response - Its unfortunate that you blog title is misleading then. If you had used your "Python has more momentum than Perl right now" instead then it might have drawn less fervent responses ;-)
Anyway my beef is representing Freshmeat data out of context. I still feel in your reply to TeeJay that I've clearly been unable to convey to u exactly what the problem is.
So let me point out to u what sticks out like a sore thumb... the Perl 2007 & 2008 figures being flat (or "coasting" as u put it). This is clearly incorrect outside of Freshmeat when u look at these figures from CPAN...
2008 - 14793 modules
2007 - 12761 modules
The above shows that Perl code as increased by 16% this year and proves that Freshmeat is a Nonprobability Sample and therefore produces "screwed up" figures!!
/I3az/
{{count}} days later:
draegtun, did "u" miss that the title of my post has a question mark after it? I was questioning the TIOBE-based discussion, show that Perl still has quite a bit of software out there, and yes, also conclude that Python is more active. If you look at some of the other stats, such as jobs, you see that Python is definitely doing better than Perl.
CPAN just isn't relevant to this discussion as it's a Perl-only archive, and thus completely useless for comparing Perl to other languages. Freshmeat is a fine source of data, except for the open source bias, which affects Perl or Python equally (they are probably a bit more popular than in industry). You have failed to demonstrate why Freshmeat would see declining Perl usage at a rate that differs from that of Python, which you would need to do in order to demonstrate that it's not a good data source.
{{count}} days later:
@david
re: blog title & seeing question mark after it - Did u not see the smiley after my reply!
re: blog title - You chose the title for a reason. Whatever the reasons u have to accept responsibility for it.. ie. publish & be damned ;-)
re: TIOBE - Yes tiobe is flawed and so any declarations on the results are therefore also flawed. Don't fall into same trap!
BTW.. have u seen Dec 2008 figures on Tiobe? Python as dropped one place and Perl as moved up just below it! They fixed (at least a bit) the Delphi flaw with using Google Chrome in its search ;-)
re: CPAN relevance etc. Unfortunately u're still missing the point. Freshmeat is "not a fine source of data" when used out of context. If u want to compare Perl & Python development then u have to go to their relevant sources (ie. CPAN for Perl).
re: "failed to demonstrate why Freshmeat..." - I have given lots of clear reasons why Freshmeat data cannot be trusted in the reports you've given. Going thru my replies in order...
1) Freshmeat does not reflect (and probably never as done) what is developed in the Perl world because of its CPAN centric ecosystem. Also I've given myself as a clear case study of a Perl programmer that used Freshmeat but stopped doing so a few years ago.
2) Freshmeat language tagging is not concise.
3) CPAN shows growth. Freshmeat shows "coasting".
I have clearly demonstrated that u cannot trust Freshmeat data for the comparative analysis u're putting forward.
Even without these clues I provide any statistican will tell u that the mechanisms of what Freshmeat is will near enough certainly qualify it as a Nonprobability sample.
I've worked in Market Research for over 25 years and whenever I've published findings to clients the onus as been on me to statistically prove they are correct.
Thus the onus is on your to prove the Freshmeat isn't a a "screwed up" sample and that your comparison figures you've produced have relevance outside of a Freshmeat universe.
/I3az/
{{count}} days later:
draegtun, I don't want to compare apples to oranges, so it's useless to look at CPAN, which (obviously) has no data for Python or any of the other languages. Freshmeat has data for almost all the languages on my list, so that makes it a much better source of data than trying to look at repositories for many different languages. If CPAN shows some growth for Perl, that seems normal to me - if there's no growth there, the language really would be dead. Freshmeat indicates that, relative to Python and other languages, Perl isn't seeing as much code released. The point that perhaps people are less interested in Freshmeat these days isn't a bad one (although "myself as a case study" isn't great statistics either), but why should that be different for Python, Perl or, say, Lua users? If it's going to drop off, it'll drop off for everyone. Python, however, is growing, whereas Perl hasn't. I'm sorry if that upsets you, but those are the relevant facts. Does that make Perl a bad language? Absolutely not. There are still many valid reasons for using it. However, it is not as popular as it once was, in terms of new code being written.