While catching up on some old podcasts, specifically Episode 82 of WordPress Weekly, I came across a discussion about WordPress beta testing. The discussion centers around the problem of bugs not being caught during beta testing because there just aren’t enough beta testers.

To me, the solution seems straightforward – but that may be because I worked in the software industry for 10 years and have experience in software release management, so I’ll take the long path and set the scene properly.

That makes it a long post, but stick with it – you need to read it all to fully understand my reasoning. First we need to consider the following question:

Should You Upgrade WordPress Immediately After A Release?

Short answer: Sometimes, but not always.

Over the last couple of years, there have been repeated calls for WordPress users to update their blogs immediately after a new version has been released. At times, the call to upgrade has almost become a frenzy.

I’ve taken this with a grain of salt and held to my belief that upgrading immediately isn’t always the correct choice. Sure, upgrade immediately if the release fixes a critical security issue – but most of the time it’s better to wait and see whether:

• there are any major bugs in the new version
• your theme works with the new version
• all of the plugins you’re using work with the new version

I’ve developed this ‘hold back’ approach while working in the software industry over the last 15 years or so – I’d contend that it’s ‘best practice’. Upgrading to a new version without testing the new version in your environment (including server, theme and plugins) is the sign of immature processes.

Some may argue that this approach is overkill for a simple blog, but there are many sites out there running WordPress that are a little more complicated.

Even simple blogs are often not that simple, especially as so many are now monetized. Ask yourself, will I lose money if WordPress doesn’t work after an upgrade? If the answer is yes, you should probably be testing new versions before upgrading.

You’re probably asking how does this relate to beta testing? Well, in a couple of ways:

1. Bugs In New Releases Mean Less People Will Upgrade

Let’s leave aside the fact that I don’t personally subscribe to the ‘upgrade immediately’ advice and assume that it’s a good thing (which it would be in the ideal world).

Bugs in new releases undermine this advice. After the recent problem with scheduled posts, introduced in the WordPress 2.9 release, there have been people questioning whether upgrading immediately was a good thing.

If beta testing was improved and caught more of these problems, then there’d be less of the ‘hold back’ people out there and more people would follow the advice, in theory making WordPress more secure. There’d be:

• less people (such as Robert Scoble) complaining about WordPress security.
• less need for Matt to write public responses outlining why people should upgrade.
• less complaints about bugs and the loss of functionality.

There’ll always be bugs in software and there’ll always be people who complain – but more comprehensive beta testing would lead to less bugs, leading to less unhappy users, leading to more sites being updated immediately, leading to less hacking, leading to even less unhappy users.

Better beta testing begets better security.

2. Balance Between Upgrade Speed Vs Security

Alternatively, if we don’t leave aside my beliefs, if we can moderate the unthinking update immediately advice, then we’re provided with a tremendous opportunity:

On a case by case basis, we can decide if an upgrade is in the “update immediately” category or in the “can wait for critical bugs to be fixed” category.

My solution, way down the bottom of this post, is going to propose delays to upgrades that are not for critical security issues, so that user experience less critical bugs. But more on that later.

The Problem: A Lack Of Beta Testers

The problem, which has been mentioned in various places including the podcast, is that although WordPress has a beta testing phase, there aren’t enough end users testing it to catch all the critical bugs. The software appears to runs fine during beta testing, but bugs rise to the surface when the release is rolled out to everyone.

First, some facts:

• Software is always going to have bugs
• Beta testing will never find all the bugs

There will always be some bugs that are only triggered when the software is run in an obscure environment or is used in a way that’s slightly different from how most people use it.

So it’s not about eliminating bugs, just about reducing the number of critical bugs that get through beta testing. The only way to do this is to get more people, using the software in different ways, on different server environments, to test the software.

How do we get more people to test the software? The answer’s coming soon (honest).

Software Release Management Anecdotes

First, let me say that I worked in the software industry (for a library management system vendor) for 10 years. For quite a bit of that time, I was the person who authorised the release of software to clients and/or agents.

I could bore you with software release management anecdotes going back to 1994. Luckily for you, I came to my senses and edited them out! I’ll just say that making sure that major releases were properly tested and free of bugs was extremely important for the following reasons:

• Loss of reputation, harming future sales: Upset users are significant in the relatively small library software marketplace.
• Direct financial loss: yes, it cost a significant amount of money to burn 5000 CDs back in 1994 (and then post them out).
• Lots of additional work: printing letters, packaging CDs and supporting users who were upgrading, etc.

Obviously, WordPress operates in a very different environment from library software industry of old and many of these points are less of an issue:

If WordPress lose one user, or even ten thousand users, because of loss of reputation, they don’t actually lose any money in sales (actually there is no they!). If they have to release an upgrade, there are no costs for CDs and postage, people just download it. There’s no surge in the amount of support they need to provide, as support is crowd sourced.

Does that mean WordPress can afford to ignore this issue?

I say No – they should care about their reputation (and their users), even if there isn’t the same financial emphasis that there is in the closed source software industry. In fact, they should look to the closed source software industry and learn from it. Solutions to this very problem have been worked on and refined for decades.

In my particular case, we had several major incidents that led to us refining our testing and release processes. We ended up moving to a process that involved three phases of testing – I’ll outline these and relate them to WordPress in the next section.

The Three Phases Of Testing

1. Developer Testing

I’m sure this is done comprehensively by the WordPress developers. Ultimately, we employed a couple of full time testers, which Automattic could possibly consider funding. They may already have some testers (besides the Chief BBQ Taste Tester), although it’s not obvious from their About page.

But the problem isn’t here and full time testers would only have limited impact, so let’s keep moving.

2. Beta Testing

This is being done, but the problem is that not enough end users are involved. WordPress could look at ways to entice more users to join the beta testing program, but many of the methods that the closed source software industry use, such as offering discounts to beta testers, won’t work in the open source world.

One idea could be to build in a "do you want to upgrade to the beta version (for the greater good)" message into the automatic update function. When a beta version is released, everyone gets this message.

It would have to be very clear that it was a beta version and that there may be some problems and it would need Yes, No and Never Ask Me Again options, but it could work. Not many would choose to upgrade, but 1% of 3 million is 30,000, which is a lot more than the number of current beta testers.

Once again, though, I don’t think the answer is here. It would be a significant improvement, but could cause a lot of negative publicity: there’d be too many people who’d choose it accidentally, then blame WordPress for upgrading them to beta software.

3. Live User Testing / Staged Release

So we come to the solution at last (I told you we would) and it’s not in the beta testing phase.

Instead, the answer lies in a staged roll-out to users: limiting new releases to a relatively small number of people until it’s proven that there are no major problems.

If any major problems are found, a decision can be made on whether they are serious enough to fix immediately, before making the software available to the rest of the users. Even if it’s decided not to fix the problems, users can be made aware of issues that may affect them before they upgrade. Even that small improvement would make a real difference to users.

The staged release approach been used for decades. I was using it 15 years ago. Google uses it today: they roll-out new features in Google Analytics or Google Adsense slowly.

How Would A Staged Release Work?

A staged roll-out of WordPress would work something like this:

1. Once the beta testing phase is complete, the automatic upgrade notice would be sent out to say 100,000 people.
2. There would then be a period of say one week, where the feedback would be monitored and any critical bugs identified and investigated.
3. At the end of the period:
   • if no critical bugs have been identified, then the automatic upgrade would appear to everyone else.
   • If critical issues have been identified, a decision would be made on whether to proceed with the release or delay it until the problem is fixed.

It goes without saying that urgent security releases would bypass this.

Now I’m not talking about physically denying the software to anyone, just manipulating who gets it through the automatic upgrade process. Anyone would still be able to visit wordpress.org and download the lastest version.

How Would A Staged Release Be Achieved Technically?

I’m not the best person to answer that, but I’d imagine it wouldn’t be too hard to do. It would require the automatic upgrade function to be modified: instead of WordPress installations asking "is there a new version", they’d ask "is there a new version and can I have it".

On the server side, it would check the download count and, when it reached the threshold, start saying no. You could make it complicated (checking whether it’s been downloaded from this IP address before), but it would be best to keep it simple.

There exists the potential that some users may be unhappy about either getting the new version before it’s been through live testing, or having to wait until after. The key here is that users are not getting the beta version, they’re getting the real version that’s been through beta testing, just as they would now. It’s just that it’s being rolled out slowly to further protect users.

This would need to be well communicated.

Of course, it could be made it opt-in. The first 100,000 users could be shown the message: "WordPress 3.1 is available! Update or Wait". If they choose Wait, it won’t ask them again until the live release testing is complete. The rest of the users would be shown: "WordPress 3.1 is available, but we recommend you wait for moment. Wait or Upgrade anyway".

Final Thoughts

So the solution to the beta testing problem isn’t actually within the actual beta testing phase. It’s just to roll out the upgrades slowly, giving time for critical bugs to be caught before they affect too many people. This would protect the majority of users from any serious bugs.

Personally, I have doubts that this would ever be adopted: I think it would be seen as too complicated. But really, it wouldn’t be that hard to do and would make life better for the vast majority.

What do you think? Am I crazy? Do you disagree with me? Do closed source practices have no place in the open source world? Do you have a better idea? Let me know!

Earlier today I received an email from an irate commentator, accusing me of spamming him and threatening to report me. He was receiving emails from my blog, via the Subscribe To Comments plugin, but he thought couldn’t unsubscribe. The cause: my wp-admin folder is password protected.

As it happens, the commentator had successfully blocked notifications from my site. However, he thought he hadn’t because he received a username / password prompt when he clicked the Unsubscribe link in the email.

Cause – Subscribe To Comments Calling Wp-admin.css

I wasn’t sure why this was happening. The URL used for the Unsubscribe link didn’t appear to be going to the wp-admin folder and I couldn’t see any reason why it would need to. I rolled my sleeves up and jumped into the Subscribe To Comments code. I quickly found the reason on line 951 (in version 2.1.2):

@import url( <?php echo get_settings('siteurl'); ?>/wp-admin/wp-admin.css );

The plugin is calling a CSS file from the wp-admin folder, which invokes the password prompt. As the user doesn’t know the password, they will probably click Cancel and the CSS file will not be served.

This CSS file is only used to style the Unsubscribe page. It does not affect the functionality of the Unsubscribe / Block function. It will continue to run and will unsubscribe the user. The only negative outcome will be it won’t look quite as nice. Well, not the only negative outcome:

The user will be confused as hell because of the password prompt.

Solution – Excluding Wp-admin.css From Protection

I had to resolve this issue. The easiest solution would have been to just hack the code of the Subscribe To Comments plugin and remove the call to the CSS file. However, if the plugin is ever updated, then it would have overwritten my hack and we’d be back where we started.

The sensible alternative seemed to be to exclude the wp-admin.css file from the password protection. A CSS file is highly unlikely to be used in any attack on my site.

There didn’t seem to be anyway to exclude the file via CPanel, but I knew there’d be a way to do it by editing.htaccess. I’m no .htaccess expert, so I did a search on the topic, finding the answer in Brett Batie’s Password Protect All but One File post.

That post tells you how to exclude a file in general terms, so here are the instructions for excluding wp-admin.css file.

Go to the wp-admin folder (make sure it is the wp-admin folder) on your server and edit the .htaccess file. It will probably look something like:

AuthType Basic
AuthName "Authorised Only"
require valid-user
AuthUserFile "<path-to-site-root>/wp-admin/passwd"

Leaving the first 4 lines exactly the same, add the following 4 lines directly after them:

<Files "wp-admin.css">
  Allow from all
  Satisfy any
</Files>

That’s telling Apache to allow access to wp-admin.css. The final .htaccess (in the wp-admin folder), should look something like:

AuthType Basic
AuthName "Authorised Only"
require valid-user
AuthUserFile "<path-to-site-root>/wp-admin/passwd"

<Files "wp-admin.css">
  Allow from all
  Satisfy any
</Files>

Problem solved. My visitors can now unsubscribe again (though why would they want to!). I still have the hardened security provided by password protecting the wp-admin folder. I haven’t had to hack Subscribe To Comments, which would cause problems when the plugin is upgraded.

Final Thoughts

Mark Jaquith, if you read this, many thanks for a great plugin - but could you consider copying the wp-admin.css file into the plugin folder and calling it from there? I’m fine with you making this post redundant!

Tonight I was browsing the Internet, when my virus software notified me of a potential threat from openstat.ws. None of the websites open in Firefox had a link to this site in the source. After some investigation, it appears that the potentially malicious site is called by Google Adsense.

Avast Anti-virus Warning Message

I use Avast Antivirus on my computer and tonight it gave the following warning message while I was browsing the Internet:

Sign of "HTML:Iframe-inf" has been found in "http://openstat.ws/top.php\{gzip}" file

The inclusion of a URL made me suspect that one of the sites I was browsing was linking to a dodgy website (ie openstat.ws).

The obvious thing to do was to check the source of the sites open in Firefox, to see which one was the culprit. However, openstat.ws did not appear in the source of any of the pages. Not to be put off, I used the Web Developer toolbar to examine the generated source. Still nothing.

Google Says Openstat.ws Is Suspicious

Next stop, a Google search for openstat.ws. The number one result was the Google Safe Browsing diagnostic page for openstat.ws page. Because the nature of this page is that it may change often, I’ve grabbed a screenshot of what it’s showing tonight:

Okay, so Google are saying:

Site is listed as suspicious – visiting this web site may harm your computer.

They say the site was only listed for suspicious activity once in the last 90 days, but they also say:

Of the 6 pages we tested on the site over the past 90 days, 3 page(s) resulted in malicious software being downloaded and installed without user consent.

I’m not a security expert and I may be reading this wrong (please let me know if I am), but that seems to be indicating that there’s a 50% chance of malicious software being installed from openstat.ws.

Norton Say Openstat.ws Is A Threat

The third result in the Google search was Norton Safe Web’s page on openstat.ws. Let’s see what they say about openstat.ws:

Norton are saying that there are two threats found on openstat.ws, one of which is:

Threat Name: Direct link to HTTP Malicious Toolkit Variant Activity

Location: http://openstat.ws/top.htm

The file Avast picked up on my computer is top.php, but top.htm is pretty close. HTTP Malicious Toolkit Variant Activity sounds pretty nasty. Norton say:

Severity: High

This attack could pose a serious security threat. You should take immediate action to stop any damage or prevent further damage from happening.

Okay, I’m convinced now that I don’t want openstat.ws being called on my computer. But how can I stop it where I can’t find where it’s being called from.

Looking Under Firefox’s Hood – Sessionstore.js

If openstat.ws wasn’t being called by the websites I was visiting, perhaps it was being called by Firefox itself. I started thinking that Firefox or one of the extensions I run must have been compromised. I started looking through the Firefox files – admittedly without much of an idea of what I was looking for.

I started by looking in the \Documents and Settings\[username]\ Application Data\Mozilla\Firefox\Profiles\[profilename] folder. I ordered the files in date order and started going through the most recently modified files.

I soon came to sessionstore.js. It gave me the answer, although it wasn’t the answer I was expecting. Sessionstore.js seems to store the current session, presumably so it can be restored in the case of Firefox crashing. I’m not sure if this is default behaviour or part of the Session Manager extension.

It consists of a series of entries tags, one for each tab that’s open. In examining this, I found the following:

EDIT: Due to Syntax Highlighter performance issues, I’ve moved the sessionstore.js snippet into a text file.

That’s not particularly readable, but it’s saying that I’ve got Ozh’s Handling Plugins Options in WordPress 2.8 with register_setting() post open. Inside that there is a child URL open (http://googleads.g.doubleclick.net/etc) which is a Google Adsense ad. Inside that, there are some further children, down until we come to one for http://openstat.ws/top.php, which is our suspicious site.

At this point we are still inside the Google Adsense child, meaning that the site that Google lists as suspicious is actually being served through Adsense. This is a little worrying to say the least!

Note: There is absolutely nothing wrong with Ozh’s site apart from the fact that he is running Adsense – as do I and hundreds of thousands of other sites.

Final Thoughts

As I said, I’m not a security expert, so I’d love some feedback from some more knowledgable. I’d also love to hear if anyone else out there has come across this problem.

I recently listened to the WordPress Podcast – Episode 44. Although it’s a couple of months old now, it was quite interesting and one issue really caught my eye ear: the security related question for Matt Mullenweg at around 1:13:30 of the podcast.

A listener, Simon Jones from beforeiforget.co.uk, talked about the difficulty of changing the default WordPress user name from admin to something a little harder for hackers to guess. The only way to do this is to change it in the database itself – it can’t be done through WordPress.

Simon’s main question was "Why isn’t there an easier way to change the default user from admin to something else?". Matt didn’t actually answer this, presumably because Simon’s question was fairly long and to be honest, could have been a little more to the point.

However, Matt did mention that it’s very difficult to brute force WordPress because such an attempt would need to submit thousands of requests per second and web servers won’t allow that many requests. He also mentioned that he’d changed his user name, though obviously not for security reasons, because he told everyone what it was!

Anyway, the podcast has inspired me to go on to discuss a couple of points related to WordPress security:

Password Strength

What Matt didn’t say (because it’s elemental) is that having a strong password is essential. A user name / password combination of Admin and dS35Hg68p1d will be much harder to break than one of admin and WordPress.

If you have a strong password, leaving user name as admin is less of an issue. So make sure your password is reasonably strong.

Protecting The Wp-admin Folder

For the paranoid, such as myself, who are worried about their WordPress login being hacked, it’s possible to add an extra layer of security to the wp-admin folder. This is more effective than just changing the default user. There are different ways to do this, including:

• only allowing users from a certain IP address or range to access the wp-admin folder
• using the AskApache Password Protect plugin to password protect the wp-admin folder
• using CPanel to password protect the wp-admin folder (here’s a general tutorial, but one that could be applied to wp-admin)

The second two methods will present the user with an additional user name / password prompt before the normal WordPress login screen can be accessed. This obviously takes longer to log in, but it also makes it much more unlikely that your site can be hacked.

Problem With Password Protecting The Wp-admin Folder

When I first tried to password protect the wp-admin folder, using the AskApache Password Protect plugin, I ran into a serious problem. I wasn’t given the “Authentication Required” window, I just got a 404 File not found message. This meant I was unable to log into my WordPress system and couldn’t access the plugin screen to turn off the password protection.

So how did I get access to my site again? I logged into my host via FTP and removed the following files:

• .aahtpasswd from the public_html folder
• .htaccess from the public_html/wp-admin directory (not the one from public_html)

I later experienced this same problem when password protecting the wp-admin folder via Cpanel, but of course, I could just turn off the password protection again via Cpanel.

I found a permanent solution to this problem at Developed Traffic’s WordPress admin password protection 404 post. The solution is to create an empty file called myerror.html and upload it to your public_html folder, then add the following to your .htaccess file (in public_html):

ErrorDocument 401 /myerror.html
ErrorDocument 403 /myerror.html

If you want to store the myerror.html file in a folder, rather than in public_html, then simply add the folder’s name to the two lines, ie:

ErrorDocument 401 /foldername/myerror.html
ErrorDocument 403 /foldername/myerror.html

That should fix the problem – although I’m not sure what impact it may have on other things (ie by having 401 and 403 go to the new file rather than WordPress handling it). If anyone out there know this please let me know in the comments.

The Real Lesson About WordPress Security

This post is all about trying to minimize the slight chance that someone may be able to break into your WordPress system via the login screen. I’m willing to bet that out of all the WordPress blogs ever hacked, very few of them would have been hacked via the login screen.

In the previous section, I mentioned how I locked myself out of the WordPress login screen, but got around it by FTPing in. All that security on the login screen is worth nothing if someone gets access to your host account via FTP.

From what I’ve heard, most hacked sites were compromised via:

• their host login being stolen after their email account was hijacked and used by the hackers to get the login details from the host service provider
• through an XSS exploit in WordPress

So the real lesson is to keep your computer free of viruses and spyware and your WordPress installation up to date with the latest security releases.

If you have any further thoughts on WordPress security, please let me know in the comments.