Strategies For Testing Ajax

Agenda

• For Ajax developers
  • How do I test this thing?!
  • Automated testing
  • Manual testing
    • (refresh, point, click, type)
• I will present 5 essential strategies
  • Concepts, not tools
  • Real examples

• This is a talk for any web developer who has worked on some kind of JavaScript enabled website and wondered, how the hell do I test this thing?!
• I've put together five essential strategies that can be used together to test any kind of Ajax website, large or small.
• These are concepts, not tools, but I want to show some real examples with tools written in Python or other languages.

Why create automated tests?

• Quickest way to verify that all features "work"
  • Avoid regressions
• Freedom to experiment
  • Builds confidence
  • Essential for refactoring
• Manual testing is error prone and time-consuming
  • People are lazy
  • Does not scale
  • Hard to test edge cases

• Why do we even spend the time and effort to create and maintain a suite of tests? Running a suite of tests is the quickest way to make sure everything in our application still works. If releasing a bug fix, you can run a suite of tests to make sure some small code change hasn't broken anything that already worked.

My Background

• I've built some Ajax sites using :
  • Pylons + Dojo
  • Pylons + ExtJS
  • Django + jQuery

• I have failed at testing Ajax. Many times.

Testing Ajax is hard

• Target: desktop Web browser
  • not meant to be automated
  • hard to simulate
• JavaScript!
  • Every web browser misbehaves (a little)
  • JavaScript frameworks mitigate but not entirely

• The type of website I'm talking about testing is one that gets deployed to a web browser. This is great for Ajax because most Web browsers support enough JavaScript to do cool stuff but it's not great for testing because a web browser is hard to automate.

Testing Ajax is easy

• Ajax encourages modular design
• Server sends data (i.e. JSON) to UI components
• Potentially less code, less complexity

• On the other hand, Ajax encourages modular design and this can make your site easier to test.

Example: aintjustsoul.net

• Pylons + Dojo
• Lets you listen to records on eBay
• Has dynamic browsing, filtering
  • Ajax used to fetch album thumbnails

• In action!

aintjustsoul.net

Server Sends JSON To UI

GET /auctions.json

{
    "images": [{
        "height": 60,
        "width": 80,
        "auction_id": 28465,
        "path": "http://.../images/59831.jpg"
    }, {
        ...
    }]
}

• The GET response sends back JSON which gets evaluated in the browser as JavaScript code and here an array of image data is used to render img tags on the page.

Strategy #1

1 . Test Data Handlers

2 . Test JavaScript

3 . Isolate UI For Testing

4 . Automate UI Tests

5 . Gridify your test suite

Python HTTP get test

def test_auction_images():
    response = app.get("/auctions.json")
    json = simplejson.loads(response.body)
    assertEqual(
        json['images'][0]['auction_id'],
        28465 )
    # etc...

• Here is a simple test in Python that runs against the aintjustsoul.net application in-process without a server or a web browser.
• It makes a request to auctions.json, loads the response, deserialies the JSON to Python, and makes assertions about the data content.

Python HTTP GET test

In action!

Python HTTP GET test

Strategy #2

1 . Test Data Handlers

2 . Test JavaScript

3 . Isolate UI For Testing

4 . Automate UI Tests

5 . Gridify your test suite

Check For Lint

http://javascriptlint.com/

$ jsl -process ./js/cool-ajax.js

cool-ajax.js(41): trailing comma is not legal
var conf = {debug: true,};
........................^

0 error(s), 1 warning(s)

• Before even running any actual unit tests you can do yourself a big favor by checking for "lint"
• This refers to checking for syntax errors, for undeclared variables, and other mishaps.
• The error you see here is telling you that you can't leave a trailing comma on an object literal
  • This code will work in Firefox which you may be using for development.
  • But it will not work in IE 6
• There are several other tools for checking lint; this one is the jsl command and you can run it by simply passing -process and a JavaScript file to check the lint of.

Run Unit Tests With Rhino

http://www.mozilla.org/rhino/

java -jar js.jar ./your_test_runner.js

• Upsides:
  • No browser, command line
  • Continuous integration

• Next I'll show some ways to actually run unit tests against your JavaScript code
• You can use an open source product from Mozilla called Rhino
• This is a pure Java implementation of JavaScript

Run Unit Tests With Rhino

http://www.mozilla.org/rhino/

java -jar js.jar ./your_test_runner.js

• Downsides:
  • Custom implementation of JavaScript
  • No DOM
    • could use John Resig's env.js

Run Unit Tests With Rhino

http://www.mozilla.org/rhino/

java -jar js.jar ./your_test_runner.js

• Alternatives:
  • Spidermonkey C bindings
  • ?

Dojo's Unit Tests in Rhino

In action!

Dojo's Unit Tests in Rhino

Run Unit Tests In A Browser

http://localhost/app/run-tests.html

• Upsides
  • You get what you pay for, it's a web browser
• Downsides
  • Hard to simulate user events
    • Selenium
    • jquery.simulate, YUI.test
    • Java Applet?!
  • Tricky to automate

Unit Tests For Fudge

• QUnit test runner (jQuery)
• Example: JavaScript library for testing with fake objects

• In action!

Unit Tests For Fudge

jQuery QUnit Example

<script src="qunit/testrunner.js"></script>
<script type="text/javascript">

module("Ajax navigation");

test("go() sets hash", function() {
    nav.go("#somewhere");
    equals(window.location.hash,
           "#somewhere");
});

Here's Where It Gets Tricky

• Up to this point I've been talking about testing the low-level components of a website.
• This is great but how do you start testing a full-stack web application?
• This is where it gets tricky

Strategy #3

1 . Test Data Handlers

2 . Test JavaScript

3 . Isolate UI For Testing

4 . Automate UI Tests

5 . Gridify your test suite

• Remember the part about me failing at testing Ajax? This is where the fail is.
• In all the problems I ran into I realized it's pretty impossible to test a full-stack web application in the browser.
• Instead, I've been isolating just the parts of the user interface that need testing.

Example Ajax application

• To understand what I mean, let's look at the full-stack

Isolate UI For Testing

Aintjustsoul UI, Isolated

In action!

Aintjustsoul UI, Isolated

• Here's a version of the same development site but without the database

Using a fake server

if config['ui.enable_test'] == True:
    map.connect('/auctions.json',
        controller='_ui_test_/browse',
        action='auctions')
else:
    map.connect('/auctions.json',
        controller='browse',
        action='auctions')

# ...

• To do this what I've been doing is just creating fake controllers in my server for testing.
• This is Pylons routing code that looks for a configuration flag then swaps out the mapping so that all requests for auctions.json are handled by a secret hidden controller _ui_test_/browse

UI Layer Doesn't Know

dojo.xhrGet({
    url: "/auctions.json",
    handleAs: "json",
    load: function(response) {
        images.load(response.images);
        // etc...
    }
});

• The front-end UI doesn't even know what just happened. It still thinks it's talking to a real app with a real database.

Using a fake server

Advantages :

• Instant prototype
• UI tests can run without a database
  • faster tests
• No blocking when running tests in parallel
  • no transactions, no locks

Using a fake server

Disadvantages :

• Adds a level of indirection
• Boilerplate code to setup fake controllers

Bypass Server Altogether

jQuery.extend({
    ajax: function( ajax ) {
        switch (ajax.url) {
            case '/auctions.json':
                ajax.success({images:[]});
                break;
            default:
                ajax.error("Unexpected URL");
        }
    }
});

• Here's another method I've been experimenting with
• Instead of creating a fake server you can bypass the server altogether by stubbing out the JavaScript functions in your app that make HTTP requests.
• In this example, the extend() method replaces the jQuery.ajax() function with some code that calls the success() callback as if some JSON data was fetched from the server.

Strategy #4

1 . Test Data Handlers

2 . Test JavaScript

3 . Isolate UI For Testing

4 . Automate UI Tests

5 . Gridify your test suite

• For the next strategy I'll show how you can automate a web browser to run tests against your isolated UI

Selenium Remote Control

def test_click_auction_image():
    selenium.open("/")
    selenium.click("auction-1")
    assertEqual(
        selenium.get_text("css=#details h1"),
        'Rare James Brown 45'
    )
    # etc ...

• I'm not going to go into the mechanics of Selenium Remote Control but in this example it allows you to write Python code to automate a web browser
• This test opens up a page and checks the values of some elements on the page and can be run from the command line.

Selenium Remote Control

In action!

Selenium Remote Control

Web Browser Automation Tools

• Selenium (JavaScript, Python, Java, Ruby, etc)
  • Selenium Remote Control
  • Selenium Grid
• Windmill (Python)
• Watir / FireWatir (Ruby)
• WebDriver (Java)
• Canoo Web Test (Java)

Strategy #5

1 . Test Data Handlers

2 . Test JavaScript

3 . Isolate UI For Testing

4 . Automate UI Tests

5 . Gridify your test suite

Gridify your tests

Gridify your tests

Gridify your tests

• Run Selenium tests in parallel for huge speedup
  • Selenium Grid
  • saucelabs.com (grid service)
• Considerations:
  • Optimize your test app for concurrent hits
    • No database
  • Break apart long running tests

Future?

• Test Swarm (pre-alpha)
  • http://testswarm.com/
  • http://bit.ly/about-testswarm
  • Crowd sourced continuous integration

Recap of Ajax testing strategies

1. Test Data Handlers
   • HTTP Request / Response
2. Test JavaScript
   • Check for lint
   • Using Rhino or a web browser
3. Isolate UI For Testing
   • Fake server or JavaScript stubs
4. Automate UI Tests
   • Selenium, etc
5. Gridify your test suite

Wrap-up

• Taming The Beast: How To Test an Ajax Application, Oct 2008
  • by Markus Clermont and John Thomas
  • (Gmail, etc)
• How are you testing Ajax?
  • http://bit.ly/testing-ajax (mailing list)
• Big thanks to Leapfrog Online for sponsoring PyCon
• Questions?