Testing Web applications with multiple browsers

One of the messier aspects of delivering Web applications to the Internet is comprehensive testing to ensure a consistent user experience with different browsers. Given the wealth of browsers and versions along with operating systems, this is easier said than done. Here's a look at various avenues for proper application testing.

Who will use it?

A key ingredient when approaching the testing phase of a Web application is deciding what browser platforms will be used to access it -- or, more appropriately, what browser platforms will be supported. With intranet applications, the browser is more easily controlled, but the public Internet is wide open, as users are free to use what they want.

A quick glance at browser statistics for December 2007 on TheCounter shows Internet Explorer with a commanding lead in browser usage (version 6.x at 44% and version 7.x with a 35% share) and Firefox and Safari with smaller shares. You may examine such statistics and decide to test an application with the top four browsers, or the client may decide what browsers will be supported. (It is worth noting that the growth in the use of handheld devices like cell phones and PDAs means you may need to test these as well -- depending upon the application.) Once you decide what browsers are supported, you need to decide how to actually test with these browsers.

Testing platforms

You need to decide how to properly test with a set of browsers. The simplest, and most costly, solution is to set up test machines with each browser installed. Or, you may choose to install each browser on the same machine; however, this can get hairy when dealing with multiple versions of the same browser platform (like Internet Explorer 6.x and 7.x). One issue with using multiple browser versions is actually getting copies of older browsers. A great resource for locating older browsers is evolt.org.

One browser you may not want to ignore is the text-based Lynx browser, which is still available. It is good for testing how a site looks to nongraphical browsers like search engines. Also, it can help with testing accessibility issues because it shows how the site appears when presented as text -- with this text processed by screen readers and so forth.

Along with using multiple browser versions is testing with the numerous operating systems in use today. You may test Internet Explorer with Windows Vista, Windows XP, and Windows 2000, while using Safari with the various OS X versions like Leopard, Tiger, and Panther. Also, you may test Firefox on these platforms along with Linux.

It's costly to set up individual computers for each browser and operating system configuration. Dual booting and virtualisation provide alternatives that allow you to consolidate testing environments and reduce costs. Dual booting can be time consuming because you have to reboot every time you switch to a different operating system. Virtualisation allows you to run multiple virtual machines with heterogeneous operating systems at the same time on the same physical machine. You can switch to the different machines without any lag time with rebooting. Some popular virtualisation platforms are VMware and Virtual PC.

You can get the most control by conducting all application testing with multiple platforms in-house, but this may be out of the realm of possibility for smaller organisations. Smaller shops may turn to a set of users or use third-party services.

Another path

I have worked on numerous projects where an established set of users outside of the organisation are tapped for application testing. In addition to using various platforms, it also provides the opportunity for real-world testing where users have their own Internet connections, and testing does not rely on high-speed corporate connections.

These users can provide valuable feedback on application behaviour and performance. In addition, organisations often use this type of setup even if testing is conducted in-house. These users may be viewed as beta testers where they offer a second wave of testing to ensure proper functionality in the real world.

Another path that may be followed is using a third-party service to test a Web application via multiple browser platforms. You could choose an offshore company to test with various platforms or use a free service like Browsershots or a paid service like BrowserCam.

The mobile world

The boom in mobile device usage means this ever-expanding user community should not be ignored. Like personal computers, you can assemble a group of mobile devices to use for testing, or you can use third-party services and products to assist with mobile testing. A great resource is the DotMobi Virtual Developer Lab, which provides access to hundreds of mobile devices for testing.

Make sure it works

While most developers think an application is ready once their work is done, you still need to conduct extensive testing to ensure the product delivered actually meets project expectations and behaves consistently within the target set of browsers. There are many ways to go when testing with multiple browsers as you may choose to set up multiple machines, use virtualisation, or even go with a third-party service or organisation.

The key issue is to test an application so it functions properly within supported browsers. How do you test your Web applications? Do you get users involved or keep it in-house? Share your thoughts and experience with the Web Developer community.