Spare a thought for contrib

Just ponder these statements for a while. No need to over-analyse:

o Drupal core has about 50 modules.
o The D7->D8 core transformation has been a mammoth effort by a wonderful team of passionate individuals -- and they're still going.
o Most of their time is funded, although perhaps not sufficiently, through various sources.
o At D8's release, hopefully early 2014, the transformation from the day that the D8.x branch was created, will have taken 3 years
o Drupal is NOT backward compatible. If one module important to you or your client has not been ported to D8, you're stuck on D7 for your entire web site.
o Contrib has over 20,000 contrib modules, 400 times the amount of core.
o At this point in time, close to zero of these modules have D8 releases. Some popular modules don't even have a stable D7 release yet. Examples: Admin menu, Boost, Context, Field Permissions, Lightbox2, Location, Metatag, Redirect.
o D8 contrib development is harder than D7, at least at first. Read this or eaton's prophetic words in these Predictions for 2013: "The next generation of Drupal devs will find older D7 code as baffling as today's devs find Drupal 3 code. There will be a long lag cycle as contrib modules are ported, replaced with new alternatives, and so on."

Now ponder these questions.
- How will the bulk of a body of work 400 times the size of D7 core, progress to D8?
- How many people will be needed to make this happen and who will do it?
- How will they be rewarded for their time? Time is a precious resource and working for free indefinitely is not a viable business model.
- How much will not doing this in a timely fashion cost Drupal-based businesses all over the world?
- How many businesses as a result, will think of the above when contemplating switching to competing systems? eaton: "We'll inevitably lose a number of [those] sites to other frameworks". What do you think?

Drupal's end-of-life policy has been the elephant in the room for some time. This time though, with Drupal and its number of modules being larger than ever before, has the elephant grown so big it will crush us?

Do you, or the DA, or Acquia, or "the powers that be" even care? If so, what are the strategies or suggested strategies?

Any answers or suggestions? Please comment below.

File under: 

My first D7 -> D8 module port

Read More »

Comments

I thought all the Symfony devs were supposed to come over and help us.

i think because open source involves free community contribution, then it requires a lot more planning and designing ahead than commercial closed source projects do
someone needs to orchestrate the whole process, and that someone needs to rely on scientific engineering and not popular opinion

It is definitely true again that growing up with Drupal 8 itself makes it much easier to take the changes. So you get accustomed to changes as they happen. That is how I learnt new Drupal 8 things, and there are always areas where I did not yet work, so I don't know the best practices yet.

Most developers working with Drupal 8 decided taking the changes throughout the cycle would be more expensive for them compared to taking it all in at once. There is some value to that too, since they did not need to learn and accommodate to interim states. However, there is no denying it will be lots of things at once.

I think what can help there are the usual things essentially.

1. Docs. I wrote docs for new systems like configuration context and overrides and configuration schemas with "beginners" in mind in such areas. We do need more people write more docs for things as pieces evolve. Books might help later, but those are not even in writing yet, since Drupal 8 is still so much in flux. (Or if they are in writing, that is way too early, and such books will not be useful for 8.0).

2. Examples. Early ported modules as well as simple things can serve as examples. You can look at existing Drupal 8 modules like http://drupal.org/project/config_translation. Note that these not necessarily follow all best practices yet. Eg. many Drupal 8 modules use hook_menu as before while we figure out how to migrate to more best practices. As of now, many Drupal 7 ways still work, you can define, document and use hooks, etc. It is hard to tell what will be left when Drupal 8 comes out, joining the learning process now is still valuable :)

3. Tools. I think tools that have been available before like coder_upgrade will face an uphill battle now. Converting module hooks to PSR-0 classes outright automatically is pretty much impossible, so I fully agree it is much harder to write tools for Drupal 8 that could serve this purpose. I'm not envious of people who will work on coder_upgrade although I'm sure they will find some tricky solutions for some of the conversions.

Summary is you cannot really avoid learning a lot. Getting involved earlier helps cut down the investment into smaller incremental steps. Attempting to port simpler modules now is already a very good idea.

The missing piece in these conversations is always the same: Backward compatibility isn't a feature. Forward compatibility is. Drupal 7 and earlier didn't have it, and architecturally it was impossible to add. Drupal 8 is a massive, radical departure... but for the first time we have the potential to actually have forward compatibility: http://www.garfieldtech.com/blog/backward-compatibility

Is this going to be a problem? Yup. Is it going to be painful? Yup. Is it necessary? Yup.

Does this pain now at least offer the potential for us to have less pain in the future, finally, by adopting techniques and approaches that have been long-understood to be better for dealing with exactly this situation? Yup.

That's what happens when we lag in PHP 4 land for long...

Where did you get the information for most Drupal 8 work being funded? A lot more is than previous releases, but mine for example isn't and I very much doubt that 'most' is accurate.

but there are "only" 6228 Drupal 7 modules and not all of these will require that much re-working to make them Drupal 8 compatible (I hope).

I agree with the sentiment, but using hyperbole detracts from the credibility of the argument. Certainly there's plenty of work to be done in contrib but it isn't 400 times the amount of work done in core.

That said, if you factor in the total amount of effort required to understand and re-skill in a new Drupal architecture that maybe balances things out.

"...not all of these will require that much re-working to make them Drupal 8 compatible..."

have you been following the changes? This is pretty well an impossible statement and it's even likely that all the help the coder module provides won't work this time around either. I'm likely to just abandon any contrib modules I maintain that I don't actually use.

>have you been following the changes?

only somewhat. I was basing my statement on the effort required to migrate D6 to D7 modules.

The statement / question article was a great way to make these points in a subtle yet powerful manner.

I think the issues that you raise are real and will make a big impact on the future of Drupal.

To me, there are a few "quick win" solutions that will help a lot:
- Supporting Drupal 6 for a substantial period of time after Drupal 8 is released (security fixes only)
- A big push to get modules upgraded, with (most importantly) support and tutorials for common upgrade pains (an even bigger effort than the D7CX movement).

Overall, we need to ease the pain of those who are upgrading, which will in turn help justify the "no backwards compatibility" policy that aids innovation in new core releases.

Thanks for all the food for thought everyone! Clearly this topic is very much alive in the Drupal community. I will work your contributions into a follow-up post. Cheers, Rik de Boer founder flink

Contrib is not the be all and end all of Drupal module. An awful lot of sites would have rolled their own custom modules and themes. Even if all of contrib is upgraded with a painless upgrade path, every site that has such a module must invest in re writing their own code if they want to upgrade.

If they have to do that, it will at the very least mean that they will investigate other options, and some may be less work for them to move to.

While it is good to be forward looking, it may cost you a lot of your existing user base.

We loose people/sites either way. If we build all the backwards compatibility layers (if it would even be possible), that would bloat Drupal so much that we'd lose sites who need a leaner environment without unneeded piles of code just to support older setups migrating onwards. The Drupal approach is we bring your data forward but your own code you need to take care of. You get what you paid for.

The goal is obviously that we gain more by including the new solutions, cleaning up existing ones, etc. then what we loose by breaking backwards compatibility so much. I think its natural that we loose sites/people either way.

You said: "Most of their time is funded, although perhaps not sufficiently, through various sources." -- Uh. Where did you hear that?? Because it's quite false. I know of exactly nine people who've gotten paid for part of their time to work on D8 regularly, plus the four-person Spark team at Acquia who are working on authoring improvements for D7 and contributing them forward to D8 as well. Drupal 8 probably has at least 1000 contributors. I don't know how 1% of developers having some funding (mostly through their own fundraising) can possibly qualify as "most".

Excellent -- just the reaction I was hoping for! Thank you xjm for confirming and putting a figure on what both I and catch suspected: not enough Drupal contributors get paid enough for all the time they give so that others can make money from it. Sorry, I may be in a cynical mood tonight... I do love Drupal and truly admire all the people, like you and catch, that spend so much of their time on making Drupal even better. I just wish more of them, i.e. catch and you and all the ones that deserve it, got rewarded -- is that a bad thing for me to wish for? So apart from that word "most", how did you like the article?

The last two paragraphs are what bother me:
"Drupal's end-of-life policy has been the elephant in the room for some time. This time though, with Drupal and its number of modules being larger than ever before, has the elephant grown so big it will crush us?

Do you, or the DA, or Acquia, or 'the powers that be' even care? If so, what are the strategies or suggested strategies?"

I don't think the EOL policy is any elephant in the room, since it's been discussed throughout release cycle as it was in the D7 cycle before it. No one I know of tries to ignore its impact. I also think remarks like "has the elephant grown so big it will crush us?" and "Do you even care?" are unnecessarily inflammatory.

Finally, with regard to the specific entities in the last line:
- "Do you care?" You bet I do!
- "Does the DA care?" The DA's charter explicitly prohibits them from trying to direct Drupal development, so no, they shouldn't.
- "Does Acquia care?" You bet Acquia does! I started in the Office of the CTO at Acquia a month ago, and facilitating the D7 to D8 transition is definitely on everyone's radar. (Watch for Drupal 8 resources from Acquia through the spring, summer, and fall.) However, why is Acquia called out specifically? There are lots of commercial entities whose bottom lines depend on Drupal, and Acquia is not the largest of these companies.
- "Do 'the powers that be' care?" I'm not sure who you mean here. The notion that there are "powers" seems like a misconception to me. Do you mean core devs? Yeah, we do--and also, there's no membership card. You can be one too. ;) Do you mean initiative owners, core and component maintainers, and so on? For sure they do. Most of them have to address the upgrade issue in their own jobs, after all, and most of them have become leaders precisely because they are. But I really don't buy into the notion that there are "powers." The Drupal project is driven by code and consensus, with thousands of people each addressing their own concerns and collaborating with each other. That's what makes it amazing.

As for strategies, Gábor already said most everything I would. Right now, the most critical thing is to get as many people looking at D8 as possible. Individual developers and companies that get involved now will be more ready for it come 8.0, and the closer we get to code freeze July 1, the more critical it becomes. Getting involved now also means that you can pay forward what you learn to help others make the transition as well. :) https://twitter.com/xjmdrupal/status/313491831977549824

And despite you elaborate answer those two paragraphs still bother me too. Thank you for attempting to answer my question regarding strategies. I think we all welcome any help any individual or organisation can give. This article was not about core, core developers or core documentation. It was about how we get a huge number of D7 contrib modules on the D8 road a bit quicker than we have in the past. Answers so far are still lacking. Sure documentation is necessary and you and Gabor and the team do a great job on it while others chip in where they can. But documentation is only the beginning. Being a developer yourself I don't have to convince you that converting a module to D8 is a laborious, tedious journey from one WSOD to the next. It consumes time and it can be draining. And my fear, dare I say reality, is that many developers do not have that time and won't do it in a hurry. Just look at the issue queues of some module to see how little attention some modules get. Multiply that by however many developers you feel are part of this porting issue. Doesn't smell like a bed of roses to me. "facilitating the D7 to D8 transition is definitely on everyone's radar". That's great to hear, but what does it mean, in concreto? Do we have to wait till summer or fall for the details? Your comment about the DA in this context is interesting. So the DA is prevented by charter from caring?