Archive for the ‘WordPress Testing’ Category
[Hi. I write here when it suits me.]
Jane’s post on setting the scope for future releases shows that WordPress’s process is continuing to mature. Notably:
- “Future release” goes away, and features will get slotted for specific releases.
- “As long as we’re not in freeze” goes away as a mentality.
- Project planning.
I have a prediction—the first few plans will suck! But that’s true of every project plan. [I know; I'm a project manager.] But I have hope that it will get better.
The embarrassing bug behind 2.9.1 getting so quickly moved out the door has me thinking about testing again. Before I expound on this topic, I find that I better just dive in before I sit here and pontificate…
Back when I saw Wincent Colaiuta’s strident slamming of the security holes in WordPress 2.2, I commented, “I think Colaiuta overstates his case here, but the point is taken: this should have been pushed out faster.” I brought it up with Stephen at lunch yesterday, and we talked about some of the issues at play here. I think that they are:
- Self-hosted WordPress installations not making a big enough deal to the end-user that their install is out-of-date.
- A lack of a regimented test case suite on multiple platforms [Apache, IIS, etc.] to root out any bugs in release candidates.
- The delay between discovery and patch release.
WordPress founder Matt Mullenweg has angrily responded. I’m a bit disappointed in Matt, because he takes every opportunity to take digs at Colaiuta’s platform of choice, rather than calmly and rationally dissecting the argument that Colaiuta made. This is perhaps understandable, as WordPress is the foundation of Matt’s career, but for him personally and WordPress in general, I wish that he’d responded with more grace.
- # The SQL problem in 2.2 requires both registration to be enabled (off by default) and the blog to be upgraded to 2.2. It is a serious problem but I’ve heard of fewer than 5 exploits from the flaw. Even if you assume there are 100 blogs for every one we heard about, that’s still an incredibly small percentage of the millions of WordPresses out there, especially considering, as Wincent points out, the problem has been in the public for a while now.
- Getting people to upgrade web software is hard. We work as best we can with hosting companies, but a consideration is that it’s best to roll several security fixes into one release. It’s not responsible to do a release if we know of another problem, so sometimes there is a lag between an initial report and a final release, not to mention the testing required of a product used as much as WP.
Those are the salient points here. The first is pretty important: the flaw was only vulnerable if you go away from WordPress’s defaults. The chances are that the users who are too lazy or unaware of the risks to keep their software up-to-date aren’t going to have changed many of the defaults. I run a community of Webloggers using self-hosted WordPress installs, and I can tell you that very, very few of them—maybe 5%—have ever done anything different than the default options.
The second point is important as well, although there’s not enough apparent testing to me—as a power user who sorta half-monitors wp-hackers, albeit with less frequency right now because I’m covered up with work—to be sure that this is fully the issue here. I don’t have a problem with a quick #.#.X release for a bug such as this, because good tools to keep your installations up-to-date exist. When 2.2.1 was released the other night, I patched 30+ installations in under a half-hour across as many domains on my server, all by hand. [The "40 minutes" comment is how long the 2.2.1 release had been out in the wild.] When I finished upgrading all my RMFO-Blogs users to 2.2.1—the re-architecture took quite some time, and had bitten me in the ass by running 2.0.2 [!] for months and getting my server 0wn3d—today, I went and upgraded about 30 users whom I’d already gotten up to 2.2 in about 15 minutes—again, by hand. I’m working on a shell script to automate this and make it much, much faster for me.
The big thing that’s come to mind for me lately is that WP needs to set an annoyance factor for release updates. At present, WP includes your version at the bottom of every page in the admin—which is good. That’s not enough, though. If you’re running an outdated installation, WP ought to bitch at you every opportunity it gets. How? There ought to be a banner at the top of every single WordPress admin page, using that lovely yellow-fade-to-blue status notice, that annoys the crap out of the user by letting them know that their installation is outdated. If I had that for my RMFO-Blogs users, I can guarantee that this would have them banging on my doorstep to do an upgrade for them. Another possibility is that any WP installation that’s out-of-date could, using WP-Cron, send a daily email to the administrator email address, advising them that their installation is out-of-date, and providing them URLs on where to download the files to upgrade and providing help pages for upgrading.
Security is a serious issue. It should be treated soberly. Matt’s angry, unprofessional response—choosing to sling a lot of mud at Movable Type—simply muddies the conversation. I had hoped for better from him. I do hope that we’ll see better in the future.
In a larger post about keeping the scripts that produce WordPress’s syndication feeds up-to-date, Owen Winkler says the following:
Changes in significant areas of the software such as [comment- and category-focused APIs] would require exponentially significant testing. And as might have been noticed during our pains from the 2.0 release, WordPress could stand to have a bit more thorough testing generally. It doesn’t need the strain of testing on formats that we don’t even have an developer advocate strong enough to volunteer for coding and testing. Not only would someone have to adopt each of the four feed files, but we would need testers to focus on the changes in those areas. I don’t see enough people who care – Maybe you’re out there?
Owen, I don’t know enough about the ins and outs of these feeds to be of much use, but there are people who do. [Phil Ringnalda seems to care, and you could probably talk Sam Ruby into helping with test cases---and if nothing else, Sam's focus on it would get some interesting comments from Mark Pilgrim on the subject, and that would make enough of a kerfluffle to get some attention. ] But I am certainly arguing that we need to take time and figure out what testing needs to be done—fishbone diagrams, whatever.
Matt‘s put out a request for folks to sign up to help develop and test WordPress in particular areas. I need to look at where I can contribute, seeing as I’ve been arguing for this kind of thing.
I’m not sure that a wiki-based approach is the best way to go—I’m thinking something nice and hierarchical like Tasks is a better approach—but this could be because I don’t use wikis much but seem to live in my Tasks installations.