Taming The Beast: How To Test an AJAX Application (GTAC 2008)

This was one of the talks at GTAC 2008 that I was most looking forward to before the conference. It was excellent, I was not let down. The talk was given by Markus Clermont and John Thomas who work at Google. Since the talk was right after lunch they decided to take a Q & A approach. It sort of went off in tangents at points but overall the format seemed to work.

In my own work I've been struggling at maintaining a now bloated test suite for an AJAX website but their approach made something click in my head. I'm already working on a refactoring plan.

Here is my abbreviated interpretation of the talk:

• AJAX web applications are hard to test
  • They are asynchronous in nature
  • There are multiple components
  • They need to run in multiple web browsers which all vary slightly in how they run JavaScript
• They show an example application using Google Web Tookit (GWT): a dynamic grid of three columns (Location, Product, Stock). I.E. Location: Mountain View, Product: Software Engineers, Stock: 85; Location: Mountain View, Product: Chefs, Stock: 25; Location: Zurich, Product: Software Engineers, Stock: 25; et cetera. The numbers are made up.
  • You can filter the grid by location using checkboxes on the left. There are up and down arrows for the stock count.
• Asks audience: How do you test this? What tests would you write? Responses:
  • Try different combination of checkboxes, assert that the grid is updated accordingly.
  • Try to enter a negative number into the Stock column
  • Sort the grid by Stock then change one of the numbers and assert the grid is re-sorted.
  • And so on...
• The tests must be automated otherwise the test suite will be hard to scale as the application grows in features
• The tests must run as fast as possible
  • Thus developers will be less likely to commit untested code
  • Developers can run all tests instead of selecting partial tests to run during development
• It must be easy to interpret failures
• The time between when a developer accidentally commits buggy code and when that bug is discovered must be as short as possible
  • Mentions continuous integration
• Shows a diagram of what makes up this example application. An HTTP server delivers HTML and JavaScript to the web browser in response to a GET request. The JavaScript code runs in the web browser to power the grid; the JavaScript makes asynchronous POST requests to a Remote Procedure Call (RPC) server for fetching data. The RPC server reads from a database.
• What kinds of tests are there?
  • Small tests
    • AKA Unit tests, functional tests
    • Tests for each component in isolation
    • I.E. you might instantiate a checkbox filter, attach an event handler to it, send it a click event, and assert that the handler is called.
    • You might make a POST request to the RPC server and assert the output (XML, JSON, whatever) is as expected.
  • Medium tests
    • Test that only execute portions of the application
    • I.E. You might replace the real RPC handler with a mock object, fire up the server, open a web browser to a page with the grid, and assert the grid renders data correctly. A test like this would not ensure that the real RPC server is working correctly, it would only ensure the grid is talking to the RPC server as it should
  • Large tests
    • AKA End to end tests, system tests
    • These tests run your application in an environment as close to production as possible
    • I.E. You'd populate the database with data, fire up the HTTP server, fire up the RPC server, open a web browser to the page with the checkboxes and the grid, then send mouse or keyboard events through the browser and assert expectations of behavior.
• Which tests should you write?
  • Small tests are usually the fastest and easiest to write but they don't test any interaction
  • Large tests are the most comprehensive tests but they are slow and often require cumbersome setup / teardown.
  • Medium tests can still be fast and often simplify setup / teardown. They allow you to test isolated segments of your application. However, like small tests they might not test crucial interactions between sections of an application
• Every test is an answer to a specific question.
  • Can the grid be sorted by the Product column?
  • Does the checkbox filter data in the grid?
• Ask yourself: What is the smallest test I can write that answers the question.
• Use large tests sparingly.
• Don't rely entirely on small tests
• The best approach is a combination of small, medium, and large tests. To scale a large test suite as an app grows more complex, use medium sized tests.
• Downsides to using mock objects
  • Can sometimes make more maintenance work
  • Can give you false positives
  • If you find yourself needing 17 mock objects to perform one test then you should consider redesigning this area of the application
• Last year at GTAC 2007, a speaker mentioned that Gmail was running 50,000 Selenium tests per day. Someone asks a question mentioning this and asks if Gmail uses the small, medium, and large strategy. Markus answers that Gmail's tests have gotten out of control and that his team is refactoring them for the medium sized approach.

back