Archive
jan@apache.org
twitter.com/janl
mastodon: @janl@narrativ.es
github.com/janl
GPG Fingerprint: D2B1 7F9D A23C 0A10 991A F2E3 D9EE 01E4 7852 AEE4
OTR Key: 83DFF1C9 95252513 D430604F 5E244CCC C8795B4B

Sustainable Open Source

November 20th, 2015

Recently, two major open source projects in the JavaScript world have started pleas to contributions, because they were running out of people to move the projects forward.

Reading the issues and tweets around this, I accidentally a Twitter rant (*). Now I want to write this up with a little more detail.

All of this is based on the premise that the lifeblood of an open source project are the people who work on the project. If you disagree with that premise, you don’t have to keep reading (I’m not saying you are wrong, but you won’t get anything out of this).

Another premise here is that open source projects are largely contributed to by people in their spare time. Of course that’s not all open source projects, and those who have paid full- or part-time staff still benefit from this discussion, but I want to focus on the scenario where people contribute to projects, and create a benefit for the larger community, in their own time.

In the twitter rant, I postulated a list of priorities for open source projects, if they want to be run sustainably. Before I go into the priorities, let me explain what I mean by a sustainable open source project.

Sustainable Open Source Project

A sustainable open source project is one that, during its lifetime, does not run out of contributors. There are two factors here that make projects run out of people:

1. Existing people leave the project.
2. No new people join the project.

To keep people who are already contributing, there are few basics things you need to get right.

1. Care about them as people, rather than producers of contributions (whether it is code, or docs, or designs for the website).

For example: if somebody promised to do something at a certain time, but then life gets in the way: don’t chastise them, and don’t have a culture of where they are even afraid of telling you that they are missing the deadline. Support them in their hardship, be open and understanding of people’s limited time. You don’t have to become their counsellor, but caring about your collaborators as people instead of just the code they commit makes all the difference.

2. Make sure your contributors don’t burn out,

In open source projects there is always a lot to do, more than could ever be done in a reasonable amount of time, even if paid for full-time. That creates pressure for the people who care about moving the project forward, respond to all issues, review all pull requests, write all the docs, etc.

As the people who run an open source project, it is your responsibility that this pressure doesn’t eat up contributors. For example, observe whether people take a break from contributing. As in everything, you must take breaks to be successful long-term. Maybe have a policy where people have to take a week or two off from participating in the project every three months. I’m sure you can think of other ways that would work for you.

3, Make sure there are enough people who, together, can take on the workload of the project, even with people taking regular breaks. If you can’t get enough people together, reduce the scope of the project until it matches what your contributors can handle.

This specifically includes not only writing code, but writing docs, supporting end-users in issues, on Slack/IRC, StackOverflow etc, project marketing (updating the blog/twitter/tumblr), keeping drive-by contributors happy by reviewing pull requests and issues in a timely and friendly manner, and so on. There are ton of time-hogs that are not obvious in open source projects.

For larger projects this even includes press inquiries, legal topics (licensing and naming are just two of the very hairy topics that eat a lot of time, in my experience), coordination with other projects in the space, work on foundations (hello Node.js) etc.

tl;dr: Open Source projects are a lot more than just committing code to a repository somewhere. Make sure you have enough people to handle what you want to achieve.

4. Have a Code of Conduct and be prepared to enforce it. As with all human things, people are bound to disagree. A CoC will bind everyone to keep disagreements civil and productive. And yet, some people won’t play by the rules, or understand the rules in a different way. And with a potentially global audience, cultural difference will make this doubly complicated.

But: you have to have a Code of Conduct that explains what happens if somebody repeatedly violates it. And when that happens, you have to act on what you wrote down. In most extreme cases, this means expelling an individual from the community. This is one of the hardest things I’ve ever done while running an open source project, but it has to be done.

Getting new people to join the project

All of the above will help you retain people, once they are engaged with your project. Here are a few tips on how to make sure you have enough new people coming in to shoulder all the work, account for people taking breaks, and account for others losing interest, or having other commitments.

1. Always be recruiting.

As an open source project, your scarcest resource is people’s time and attention. In order to get anything done, you will need more people. And as everywhere else, recruiting is hard, so it has to be your top priority.

For example: there’s a tough bug in your code, you could sit down for an hour and fix it. Or you could sit down for an hour and write an issue that explains enough of the context of the project so someone else would be able to fix it. Do the latter and you have a chance of recruiting someone, while at the same time more than one person can learn about the project’s internals from the issue, maybe take that part of the issue and put it into your README under “developer documentation.”

There are many, many more opportunities on how to involve new people rather than “quickly doing it yourself”, try to find out which ones apply to your project.

2. Again: Care about the people first and their contributions second.

Naturally, newcomer’s contributions aren’t as complete or far-reaching than those of experienced contributors, so it is doubly important for you care about the people and their enthusiasm about your project more than that typo-fix they put on the website. We’ve turned someone who fixed a single typo on the website to a steady contributor and well respected community member that now helps out all over the project at Hoodie, for example.

At the Apache Software Foundation, where the CouchDB project is run, we have a mantra: Community over Code. — It is a hard learned lesson from running high-profile open source projects like the Apache Web Server, Hadoop or Lucene, for over 20 years now. The reasoning is simple: if the right people rally around an idea, the right code will follow. But without a community who cares, no ever so brilliant piece of code, is worth anything.

This takes very little of your time and makes all the difference.

3. Document your processes.

Have you ever been to a (tech-) event where you didn’t know anyone, but everyone else seemed to stand in groups and chat like old friends? — I’ve been to tons of them, and as an introvert, this is not a lot of fun, and even as an extrovert, this is a challenge.

Now imagine you want to contribute to an open source project where you see some people doing some work, but you don’t see why or how this is happening and where people talk about things.

Here are a few things that happen in your open source project that are handled implicitly. You personally have no problem with this, because you either invented the practices, or were around when they were established, but for a newcomer, these are impenetrable barriers to participating.

Releases. Someone somewhere puts a tag into git, a tarball on a website, or a package into your package manager. Who decides when that happens? What objective criteria need to be fulfilled before a new release is made? Who or what decides whether a particular contribution, feature, or bugfix makes it into the next (or any) release? Who decides on the tag? Who publishes the tarball or package?

If you don’t document all of this in detail, you won’t get anyone ever helping with this. Usually, much of this is very implicit, and by making it explicit by documenting it, you often find that you have some flaw in your procedure, that you can simplify or automate a certain aspect, but most of all, somebody else can join and help.

“Hey, we are stuck on step 3 of the release process, does somebody have time to do 4 and 5?”

Accepting and declining Issues, feature requests and pull requests. Who decides whether an issue is an issue? Who decides whether a particular feature is merged from a pull request, or against the direction the project wants to go in? What are your commit message, code-listing and test-coverage requirements (use GitHub’s CONTRIBUTING.md)?

Put as much of your project’s vision and direction in writing. Of course a lot of this will be in flux as you go along, but if you copiously document this, anyone can jump in at any time and make the right call.

At CouchDB, we even went so far as to document our decision making process as part of our bylaws, which explain how the project operates. If this seems overly tedious bureaucracy to you, you do have a point: this takes time, especially agreeing on everything in large community, but it is very worth it. Since establishing the bylaws, we’ve had numerous new contributors come in and work as part of the project as if they’ve been around forever, following the right procedures to get their initiatives to a successful conclusion and thus making CouchDB a lot better.

Make your communication channels explicit*. At the ASF we have a rule that every project decision must happen on the dev@ mailing list in order for everyone having a chance to read about it in their own time and join the conversation at a central and transparent point.

At the same time, we have a busy IRC channel where we discuss a lot of ideas in a higher-bandwidth environment. However, once we get out of the half-baked stage for anything, we write it up on dev@ and have everyone else who couldn’t e.g. be on IRC at that moment (because they are on the other side of the globe, or at work) catch up with everything and partake in the conversation. Yes, this slows things down a little, but it makes sure everyone has a chance to see everything and be heard.

Whatever your communication channels are, make sure they are documented in your README, on your website and documentation. Make it easy (and free) for people to join and explain the etiquette of the medium.

Special shout out to the FreeBSD project, which is outstanding at documenting their inner workings <3

Specifically reach out to people for “positions” you want to see filled. At Hoodie, we’ve recently decided that we’d like to have a dedicated editorial team that handles our website, twitter, tumblr, etc. because the rest of us didn’t really have the capacities to maintain them all, and previous attempts made us lose two community members getting overwhelmed with the responsibility, so now we are recruiting a team (c.f. taking on only so much that you have enough people to handle it, see above).

As a result, we published a dedicated “job ad” for the position. We are just getting started, but the response has been so overwhelmingly positive that I strongly believe we’ll be very successful going forward with this for other positions. Try it.

Culture

In an earlier life I was deeply involved in the PHP community. One of the prime places to get support for problems with PHP was the IRC channel #php on EFnet. IRC and EFnet in particular are notoriously hostile places (I know there are worse IRC networks, I don’t need to be reminded of them), and that culture translated to the #php channel.

I was young and just thought that’s how things are. So I helped people out, and made fun of people who didn’t seem to know there way around (“press F4 to make this chat go faster”). I’m not very proud of being part of a hostile culture in hindsight, but at the time, I thought that’s just how things are.

Over time, I learned how beneficial it is to not antagonise people who are trying to use your open source project. Maybe in the context of PHP it didn’t matter because it was so popular, but later on, this got me thinking.


BIG FAT DISCLAIMER INTERMISSION: This is not a generalisation about the PHP community. As any community, the PHP community goes through its ups and downs and has its good and bad actors, but overall, the people who work on and with PHP are a great bunch that I have learned from half a career’s worth of knowledge and experience. Senior developers have mentored me (a high school student at the time) and helped me understand a lot about the web, not just PHP. I’m eternally grateful for this. INTERMISSION END.


When I started out at CouchDB and we finally joined the ASF and it was standard procedure to have a user@ mailing list for end-user support, I remembered my days in the #php channel and decided that that’s not the culture I want to have there. So for the first three or so years, I tried to be involved with every thread on the mailing list, and showing exemplary behaviour, being nice to people, taking their issues seriously and trying to be helpful overall. After a while, people stuck around not to only ask questions, but to help with the answering as well, and to my complete delight, they mimicked my style.

My responsibilities since have shifted quite a bit and I’m no longer checking on the user@ mailing list day-to-day, but every time I join and read, I see the culture that I helped plant there seven years ago and it makes me very proud.

Culture isn’t just what happens to you, culture is what you make of it.

Whatever you want your culture to be for your open source project’s community, as the people who are most visible, it is up to you to lead by example.

Money

If you follow my premise of people being the biggest asset in open source and recruiting and retaining them is your first priority, then it follows that it would be really easy to grow your contributors.

The ratio of women in IT is about 17-25% (depending on the study), but the ratio of women in open source are 0.1-5%. For me that means: we have spare talent that isn’t participating in our projects that could help with all of the above.

Sadly, we don’t have good numbers (as far as I know, if you have them, please send them along!) for other groups of people who are underrepresented in technology and open source specifically, including, but not limited to, people of colour, LGBTQI+ people, people with disabilities, people who do private care work (for children or elderly family members), or people of lower economic status. All of this section equally applies to all of these groups and I’m just taking women as an example because we have the numbers there. Rest assured, the numbers are worse than they are for women in open source, so we have some work ahead of us.

The summary here for why this disparity exists is that most people in open source are young white guys from a upper-middle-class (in the EU sense) and above, like myself. We all have the privileges that allow us to spend our spare time, or even paid time, on open source projects, but as is evident by the numbers, that privilege doesn’t extend to your average single mother of two, or person of colour who couldn’t land a job in IT, despite the qualification.

One way out of this would be a way for people to get paid to work on open source projects, and especially those people who are underrepresented in tech, because their time needs to be paid for. But this also includes the dedicated individuals who were privileged enough to have enough time to work on an open source project that is now so popular that it demands so much of their time that they can’t do it in their spare time sustainably.

Unfortunately, I have no guidelines for this. Financing open source is an open topic and people have far-reaching opinions from wanting to keep money out of open source at all cost (rimshot) or not, and people who are already raising money for tech projects by marginalised people.

However, this is not a mainstream discussion yet, although smaller initiatives exist, even after the Heartbleed disaster. Even though OpenSSL is comfortably funded now, this was a purely reactionary initiative (that worked well, no qualms about that), but we have no strategy about how to address this and it’s about time that we started having this discussion.

So much of our work and infrastructure depends on Open Source, and it’s time we took the people doing this work as seriously as as the highly financed startups and corporations that depend on of all this freely available work.


Let’s make open source sustainable!