WordPress Needs Formal Pre-Release Testing
The debacle that has been the WP 2.0 release cycle to date—look at the comments on Owen’s “Why aren’t you upgrading?” entry if you haven’t seen the furor—has just driven home to me the need for a structured testing program for WordPress releases.
I especially want to point to Craig Hartel’s comment, which is what inspired me to write this entry today:
I see this same discussion after every major release, and always the answers are basically the same.
What can we do to address this repitition? I suppose one major thing would be to establish a formal testing program. I would certainly participate in that, if I knew how. I feel like I’m a WP user that kind of falls through the cracks within the WP community. I know enough to get myself into, and usually, out of trouble, but I can’t code my own plugin or do much more than change some settings within an existing one.
I’ve been with WordPress since the pre-1.0 days, and I think its development has been amazing. Sure, there are plenty of things to be critical about, but what good project doesn’t have things to work on?
Somebody, please offer your skills to mentor users like me who want to gain more knowledge and understanding to help contribute to a fantastic open-source community.
FWIW, I’ve never really hesitated to upgrade, and I’ve never really had any major issues either (knock on wood). My biggest concern with WP has always been security, and I feel that by and large, security issues have been dealt with in a reasonable fashion.
Upgrading for its own sake is not a good reason, IMO, but I did just that for WP 2.0, and I’ve not regretted it.
I’m right there with Craig—I’d help if I knew how. Help me help you!
Here’s what WordPress needs in terms of a testing system:
- A release candidate period.
If we had an RC period of, say, a week for users to download, install, and run test cases on a build of WordPress, then note all the bugs, things would be a lot better.
What this takes, though, is a willingness to call for an RC, open it up to testing, and a specific bug-reporting environment for that RC alone. Nightlies and all can happen while the RCs are out, but you have to encourage your RC testers to stay away from the nightlies while they’re doing RC testing—everyone needs to be on the same page.
- A set of test cases to run against.
What’s broken WP in the past? I18N? HTML characters? Write up a fault tree. List every core feature to be tested, and give people targets to shoot at. If you have a ton of users running a bunch of test cases on all your features, you’ll likely find all the bugs you’ve got in there, including those that are server configuration-dependent.
Since I mentioned config-dependencies, I think a lot of WP users and testers are on a LAMP structure, but I know folks run WP on various versions of PHP, MySQL, and sometimes on HTTP servers other than Apache. So, let’s encourage the people that have bugs in the past to help with testing! They’ve got an interest in helping you.
- Patience and understanding on everyone’s part, and no date-driven releases. Goals and targets are great things, but sometimes you need more time.
What else is needed in coming up with a good test suite for testers? Testing needs good structure for us to have a clean .0 release.