If I was to rebuild Drupal.org, what would I want it to look like?

Over the last few weeks, I have found myself thinking more and more about Drupal.org and the various websites and services that surround it, such as groups.drupal.org, events.drupal.org, git.drupal.org, api.drupal.org, jobs.drupal.org, Slack, drupalchat.me, drupical.com, drupal.tv, drupalcontributions.opensocial.site, and all the others that don’t immediately spring to mind!

Awesome services by a tiny team

As the Drupal project has matured over the last twenty years, we have much to be proud of about the services we provide, both those provided to the community by the Drupal Association and those created and maintained by people in the community itself. Indeed, looking at main Drupal Association services of *.drupal.org we can see how many of the functions they provide; identity, project management, contribution recognition, documentation, community growth, event management, and more, are providing many features at the cutting edge of their areas. Not only that but *.drupal.org has an enviable reputation for reliability, especially given its size and extreme level of openness to community-created content.

That all those services on *.drupal.org are created and managed by a Drupal Association team of four (yes, FOUR!) engineers, is almost unbelievable. Seriously, Tim, Neil, Ryan and Brendan - I’m constantly amazed.

But...

There’s always a but, isn’t there?

I spend a lot of my time talking about the incredible flexibility of the Drupal content management framework and how it allows websites and services to be built quickly because of the technology it implements and makes available to the developer and user. The irony is, though, that with a website as big and complex as Drupal.org, and especially one so mission-critical to every user of the Drupal software, actually making improvements is challenging. Making even relatively small changes to Drupal.org can be a significant change management issue, as the updates to the Contribution Guide has demonstrated. Not only that but keeping the services we use for things like community group management can be difficult to keep up with the relentless pace of developments in the software itself. (I’ll let you in on a little secret - groups.drupal.org is running Drupal 6! 😱)

So, what do we do about this?

If we recognise that, to be able to help promote the growth of the Drupal community, we need to be able to deliver the very best services “just in time” for when they are really needed, then we need to think about how we change how we build *.drupal.org so that we:

1. Protect the reliability of the mission-critical services it provides
2. Enable the rapid development of services that support community-growth
3. Stay within reach of the latest releases of the Drupal software we make
4. Present the very best case study of Drupal-in-action there is around.

Small ambitions, eh?

Thinking from first principles

We’re not going to meet those ambitions by simply taking what we have and pressing the magic “Upgrade to Drupal 9” button. We’re going to have to get back to first principles and really question what *.drupal.org is for. What is it for, where does it excel and where can we think differently.

Peeling back the layers

When I think of the Drupal project, the thing that I always think of first is not the code, it’s the people. Each member of the Drupal community brings something different and valuable. Being able to recognize and reward that value is the single most important thing, in my mind, that Drupal.org does. Above everything else.

Now, when I talk of people, I also recognize that people are grouped together in multiple ways - as organizations, as geographical and common interest groups, and as roles they perform in the community (like contribution mentoring, project maintainer, etc).

I don’t think there is anything more important that Drupal.org can be doing than providing a core service to identify, recognize and reward the people in the community. So, here goes…

Proposal 1: The main website, Drupal.org, performs one role, upon which everything else depends: To maintain the very best possible system to manage the identity, recognition and rewarding of members of an open source community.

This role is mission-critical to the Drupal project and changes to this core system should have the highest levels of change management friction of all with the express intention to keep the system small and focussed on its key role.

Adding Services people need

But Drupal.org does far more than that right now - what about everything else?

When we build on top of the core Drupal.org, we should build on a new platform for each service we provide. This allows us to apply different levels of change management to different services depending on how critical they are to the community.

When NASA was building the computers for the Voyager probes, of which there were three in each spacecraft, they needed to be sure they worked in the harshest of environments, without fail, for five years. They applied three principles to their design: Reliability, Redundancy, and Reconfigurability. We can use these to think about adding services:

Reliability

In the Voyager missions, they applied different requirements of reliability to the three different computers - so the main computer was super, super basic and built using highly tested components. The Camera computer wasn’t as mission-critical but needed more speed so was built using less reliable MOS components.

In Drupal.org’s case, the mission-critical components should be built with reliability as the highest concern, even if that means they are less configurable and more resistant to change. Parts of *.drupal.org that are less mission-critical on an hour-by-hour basis, like whatever replaces groups.drupal.org, can have a lower requirement for reliability applied and, therefore, be more open to configurability and future enhancements.

Redundancy

Voyager had a concept of “system voting” where each component in the main computers had redundant replicas and a vote was taken on each result, still used in many aircraft fly-by-wire systems today.

We might not need quite that level of redundancy in the Drupal community but we should always encourage alternatives to systems that exist. So, for example, we have both Slack and drupalchat.me - this is definitely a good thing and something to encourage.

Reconfigurability

Again, Voyager engineers really thought ahead and built the main computer so that the fundamental ways it worked could be reprogrammed. For example, they changed the data parity algorithm from Golay to Reed-Solomon to reduce the size of data transfers as the more distant signal meant slower data rates.

For Drupal, we should always be building services that we can pull out and replace. The main core should be stable but everything else should be “easy” to replace because it talks with the main core using a published API. So, we should be able to replace our “Hey, this is Drupal, isn’t it fab?” marketing site for new users with a new version very quickly and easily. We should even be able to replace our project management / issue management system with an alternative, if a better one comes along. This allows us to implement whatever is “best example of its kind” at any point in time.

Concentrate on “unique value”

A good organisation spends time to really understand where to add “unique value” to its operations. Many parts of an organisation are “standard, out of the box” things that can be repeated across lots of similar businesses which means that staff know how they work without learning from scratch. Finance and Accounting can often be an example of this. Where the organisation has something unique, that really defines what makes that organisation better than everyone else, that’s their unique value.

For *.drupal.org, I think our unique value, what we do better than any other organisation, is how we recognise and reward contributions to the project. We have a really quite mature process that recognises contributions are far more than code and that it is not just the quantity of contributions but the way in which they are made that should lead to reward.

Proposal 2: Whilst concentrating our resources on building the custom systems of manage the people, recognition and reward, we should look for the best “standard, out of the box” systems to fill the other functions.

Let the Drupal.org core handle identity, recognition, and reward

Of course, if we are building a core around people, the other systems need to be able to interact with that core and “talk” about the people!

We need a published API for authentication and to be able to both get and post updates recognising the contribution members of the community are making on all these other systems.

So, for example, if we had a “standard, off the shelf” system for issue management, like GitLab (!), then every time someone posts a new issue, GitLab calls the core API and says “hey, this person is fab and they told us about a new issue - give them credit”, when the issue continues to progress, at each update, GitLab sends further API calls giving further credit.

Proposal 3: Credit is given to community members on the core Drupal.org via a published API.

Not everything has to be built “in-house”

Equally, other systems can hook into this - suppose we had an Open Social site as a replacement for the ageing groups.drupal.org - people could log into that Open Social site using their Drupal.org credentials because that is a published API (OpenID, perhaps?) and Open Social could query Drupal.org API to show relevant information about that community member. They could equally give contribution credit when that member answers other member’s questions or performs other worthy roles on the platform.

Because we have a published API, so long as a service is built and authorised to use it (something like building a GitHub app?) then they don’t need to worry about how we allocate credit and reward internally, only that they can communicate with the API.

Proposal 4: Authorise 3rd party services to work with the core Drupal API to give identity and recognise contributions.

Between here and there

Well, how do we get from where we are today with a mostly-monolithic *.drupal.org to one of what could almost be described as micro-services?

I don’t know – yet. Maybe that’s another blog post…

What do you think?

Hey, I’m only talking as a Drupal community member here - this isn’t something I’m doing in close collaboration with the Drupal Association Engineering Team, mainly because I wanted to let my mind wander free before doing something like that. Obviously, I’ll be chatting with Tim et al about it. They might even comment here!

If only they could “Login with Drupal”, to do so - and then get contribution credit for it!!