Thoughts

An In-Process, Headless Web Browser for Python?

posted in JavaScript, Python by kumar on Thursday Jun 12th, 2008 at 11:46a.m.

Yesterday, Atul Varma announced he had resurrected python-spidermonkey, John J. Lee's project to run JavaScript in Python. Woo! Spider Monkey is the JavaScript runtime that Mozilla (i.e. Firefox) uses internally. However, it's just an engine and doesn't deal with the DOM or anything about the typical web browser environment JavaScript files usually run in. Roughly, that environment defines the following JavaScript objects:

I've always been interested in python-spidermonkey for the prospect of fully testing JavaScript-dependent web applications in Python alone. This is no small feat and there are many caveats. The first caveat is that if the Application Under Test is running in something too far off from its native runtime then the value of your tests rapidly decreases. In other words, the best way to test a JavaScript-heavy web app is to use Selenium or Windmill to automate and introspect the app in its native runtime: the web browser. I believe Java has Rhino but since it's implemented in Java my guess is its behavior is pretty far off from that of Firefox. We all know and love the quirks of web browsers; this is what makes web development so fun :(

A lightweight version of Firefox — its JavaScript engine, its DOM, etc — could possibly make web testing a lot more efficient. By lightweight I mean no GUI, no visual rendering (although removing this may greatly hinder JavaScript, not sure). Why? At the company I'm at we build and test web applications. Our largest suite of Selenium tests that automate a real web browser takes about 2 hours to run and that's expensive. We are looking into a Selenium Grid solution but if there's a way to simply cut out all the browser overhead that we don't need, this would be a huge score.

And, oh yeah, I haven't even talked about how tests for web apps need to run in Internet Explorer if they are to be worth anything. However, we make some apps for internal company use and thus can mandate the use of Firefox. To solve the cross-browser problem one approach could be to implement all this logic as a driver that can be hooked into the Selenium interface we use, which is all in Python code thanks to Selenium Remote Control. This the exact approach that Webdriver for Java takes &mdash you can simply tell it to use a faster in-process web browser and it will run JavaScript but only via Rhino, which goes back to the non-native runtime problem. In fact, WebDriver will soon be an actual driver for Selenium.

Hmm ... thinking out loud ... maybe the solution is to just run XULRunner as a headless firefox, the way SMILE web scraper does. Maybe I'll try that.

Besides web testing, one cool thing about python-spidermonkey is it may allow for unit testing of JavaScript libs in Python. This means you could create an import hook that loaded *.js files into Python and do all your testing in Python, reaping the benefits of tools like nose. Or, maybe more realistically, automate the testing of custom Firefox extensions implemented in JavaScript.

Have you ever ran unit tests in JavaScript? It's clunky because you need to run them in a browser. That's how JSUnit works anyway, which is otherwise a passable testing environment. To automate such tests you have to use something like Selenium to run your test suite in a real web browser.

After the release of Google App Engine, the Rubyists ...

posted in Google App Engine, Python, Ruby by kumar on Monday May 5th, 2008 at 3:07p.m.

... learn Python? No. They try compiling Ruby to Python bytecode, naturally. One has to admire their pride, I guess. The funny part is this might actually work. As the article points out, Ruby's opcode structure was inspired by Python's (I had no idea it was so similar, but this makes sense). And a while back Fuzzyman performed what looks to me a successful experiment in implementing Ruby-style anonymous blocks in Python bytecode, which is probably the only Ruby logic that Python doesn't do. Although I'm probably forgetting something as I don't use Ruby much.

Making Erlang indentation-sensitive

posted in Erlang, Python, The Future by kumar on Monday Apr 28th, 2008 at 5:56p.m.

Ulf Wiger has been experimenting with making Erlang indentation-senstive. He followed up with a Part 2 after comments.

For most people who start working in the Erlang language, the most annoying new concept is all the various line-terminators you have to know. Damien Katz sums this up nicely. All these explicit terminators might make other people's code easier to read but writing code should be easy too (they are equally important). Mue says just get over it but as a Python user I agree that Erlang would be way better off if it was indentation-based so this was an interesting experiment.

As for Python, I'm still shocked to hear people say "Python would be nice ... if it wasn't for that whitespace thing." Who is perpetuating this? I have no idea. The next time you hear someone say this, pass along Python: Myths about Indentation!

The Python Make tool

posted in Python by kumar on Tuesday Apr 22nd, 2008 at 10:54a.m.

A while back there was a lot of talk about needing a Make tool for Python, one like Ruby Rake. There are a few packages now:

SCONS
- A software construction tool
zc.buildout
- System for managing development buildouts

... and more recently:

Vellum
- Vellum is Zed Shaw's project to create a nice build tool for Python, or really for anything you need to automate in a consistent fashion.
Paver
- Paver is a Python-based build/distribution/deployment scripting tool along the lines of Make or Rake

Personally, I liked Ian Bicking's suggestion to just use distutils but apparently his examples came off convoluted. I think this is because distutils is somewhat convoluted (to be fair, it was created a long time ago in software years).

At the time of that article I took Ian's advice and tried to do Make-like work in distutils commands. It was a pain. Mostly, I kept forgetting how everything needed to be set up. Since distutils silently ignores errors it doesn't tell you what's wrong, your command just doesn't work. But I still think the core concept is a good one. You probably already have a setup.py file anyway, so why not add some commands to it?

disthelper

So ... I made a helper to reduce the pain of creating setup.py commands: disthelper

It's the simplest possible approach: it just automates the creation of standard setup.py commands. You need setuptools and Paste to run disthelper but you don't need anything beyond the stdlib distutils module to run your custom command. I think more pain can be abstracted out of creating distutils commands. First off, I've been experimenting with an optparse-based wrapper around distutils.core.Command. This isn't ready for release but so far I am convinced it is possible to make a nicer Command class that's fully backwards compatible with distutils.core.Command. This could be taken even further and grow a decorator interface or something simpler that didn't require a class (for simple commands). If anyone wants to help dream up an interface, I've laid down the ground work to make this distutils-compatible and am willing to help implement it (my time permitting, of course). As an aside, I think such a wrapper would make a nice addition to the stdlib as long as backwards-compatibility is maintained.

Somewhat related is the buildutils project. Buildutils defines many custom tasks you typically want to perform on your project. At some point I may also make disthelper a repository for typical commands but right now I'd like to focus on making it a facilitator of creating setup.py commands.

Testing Google App Engine sites

posted in Google App Engine, Python, Testing by kumar on Thursday Apr 17th, 2008 at 3:07p.m.

The Google App Engine SDK sets up a fairly restrictive Python environment on your local machine to simulate their runtime restrictions. Roughly this consists of no access to sockets and thus barely working httplib, urllib, etc (replaced by urlfetch.fetch()), inability to use Python C modules, and no access to the file system. The SDK lets you run your app very easily using the dev_appserver.py script but I thought, how the heck would I test my app without the dev server?

It turns out this was ridiculously easy. You just run about 10 lines of code at the start of your test suite. Of course, it might change with an SDK upgrade but here's what worked in my tests/__init__.py in today's SDK (besides sys.path munging):

import os
from google.appengine.tools import dev_appserver
from google.appengine.tools.dev_appserver_main import *
option_dict = DEFAULT_ARGS.copy()
option_dict[ARG_CLEAR_DATASTORE] = True

def setup():
    ## path to app:
    root_path = os.path.join(os.path.dirname(__file__), '..')
    logging.basicConfig(
        level=option_dict[ARG_LOG_LEVEL],
        format='%(levelname)-8s %(asctime)s %(filename)s] %(message)s')
    config, matcher = dev_appserver.LoadAppConfig(root_path, {})
    ## commented out stuff that checked for SDK updates
    dev_appserver.SetupStubs(config.application, **option_dict)

The setup() is called by nose at the beginning of all tests but if you weren't using nose you could put it at the module level or anywhere else to be called once.

The really cool thing is running tests like this will automatically add indexes for your queries (just like the SDK dev server will) so if you had good code coverage your app would be ready to go live.

Next, you can test your URLs with something like WebTest like so:

from YOURAPP import application
from webtest import TestApp
def test_index():
    app = TestApp(application)
    response = app.get('/')

...where application is any WSGI app, like one defined in the Hello World tutorial:

from google.appengine.ext import webapp
class MainPage(webapp.RequestHandler):
  def get(self):
    self.response.out.write('Hello, webapp World!')

application = webapp.WSGIApplication(
                [('/', MainPage)], debug=True)

I haven't tried this for Django, but I'm pretty sure it would work as advertised, making your application object this:

application = django.core.handlers.wsgi.WSGIHandler()

And there you have it. The two modules used here aren't in the SDK but that's fine because you don't need to upload them to App Engine anyway. You can run easy_install nose WebTest and be ready to test. You were planning on testing your App Engine site, weren't you? :)

If you want to poke around in the test suite I made for pypi.appspot.com, the code is all in the Pypione project, specifically, the tests directory.

UPDATE

I have since realize the above method does not actually simulate all restrictions, it just inserts some stub modules. Specifically, it doesn't simulate restricting imports of a lot of builtin modules, nor does it remove file() and open().

Jason Pellerin is working on a nose plugin, noseGAE (mentioned below in comments), that aims to get all this simulation accurate. It is coming along nicely.

PyPi (Cheeseshop) on Google App Engine

posted in Google App Engine, Python, The Future by kumar on Tuesday Apr 15th, 2008 at 9:55a.m.

Like many of you, I've had my jaw on the floor since the release of Google App Engine. Although there are skeptics out there, a careful read of their terms will show you that it's for real — Google has released GOOGLE to the world and it's not for scary marketing purposes. In fact, I've been growing tired of paranoid Google haters; I'm hoping this will shut them up for a while.

Why is App Engine such a breakthrough? The concept of a hosted web application is nothing new but it has never been done this well. Mundane server maintenance? Gone. Infinite scalability? Check. 100% uptime? Let's face it, if Google went offline you'd probably be down in a nuclear bunker playing Parcheesi.

So ... how should we leverage this tool for the greater good of the community? I can't count the ways without getting dizzy. How about let's start with a mirror of PyPi, the Python Package Index?

PyPi on the App Engine

I barely spent two days on it, but here it is: http://pypi.appspot.com/. Test it out, play with it, try to break it.

As Python grows, especially due to App Engine, PyPi needs to scale too. Zope has put together a PyPi mirror but that's the only other one I know of (actually, I can't even find the link to it right now). Coincidentally, PyPi even went offline for a few min while I was writing this blog post.

Issues...

urlfetch can't get big packages
- fetching SQLAlchemy triggered the ResponseTooLarge error but its package was barely over 1MB. We might be able to create a download proxy to work around this.
May not be fully compatible with easy_install yet
Can't run scripts on App Engine
- Not too big a deal, you have to think of web pages as scripts and use a separate crontab to make timed requests. There are security issues with this, but for mirroring pypi it's not a problem
- Note that I haven't actually mirrored all of PyPi, I'm still testing it out.
It's not very pretty yet (see below)
The prototype works, now it needs some tests :)
- I'm curious to give WebTest a whirl on top of webapp.

You Can Help

I'm not dedicated to this project, I just thought it sounded like a good idea and would be a fun way to experiment with the App Engine. If anyone is interested in working on it just let me know --kumar.mcmillan@gmail.com. If there is enough interest I'll put it on Google Code. Possibly the most exciting feature of App Engine is the Datastore API (aka BigTable) and Ben Bangert agrees. It's a little hard for me to wrap my head around it but so far the Expando class—besides being the coolest name for a class—seems to work great for storing package data. If EGG-INO grows a new parameter, it just gets tacked on to the row dynamically.

This has also been a great way to dig up bugs, some of which have already been fixed.

Unicode In Python, Completely Demystified (slides available)

posted in Chicago, Pycon 2008, Python by kumar on Saturday Mar 15th, 2008 at 2:06p.m.

Big thanks to everyone who attended my talk at PyCon today, Unicode In Python, Completely Demystified. It was an ambitious title; I highly doubt I "demystified" everything, but I was happy with how it went. There were a ton of great questions — even some I couldn't answer, of course. If you have any other questions feel free to comment here and I'll try my best to answer.

For those who couldn't make the talk or those who just want to refer back to the talk and / or source code, I've posted the slides here: http://farmdev.com/talks/unicode/. The audio / video should be available soon so I'll re-post a message when that's available.

Also, for reference, here is the primer:

This talks aims to make every single last person in the audience understand exactly how to write Unicode-aware applications in Python 2. If necessary, we will move to a Birds of Feather gathering, to the bar, to your hotel room, I'll start hanging around your cube at work -- whatever it takes -- until you completely "get it." But it's really simple so bring an open mind, a notepad, and get ready to create bullet proof Python software that can read and write text in Arabic, Russian, Chinese, Klingon, et cetera. As a citizen of the Python community you have the responsibility of creating Unicode-aware applications!

Data mining in Python and beyond?

posted in Data Mining, Python by kumar on Wednesday Mar 12th, 2008 at 4:49p.m.

Dear Lazy Web,

I am a software engineer who hasn't done any college level math (gasp!). Recently, I've been having a lot of fun transforming data into more meaningful data. This, I believe, is more commonly known as data mining and I'd like to learn more about it.

Specifically, I've been looking at Internet search data where keywords are buried in some kind of template like {foo}::{bar}::{keywords}, placeholders replaced with actual content; there are many different, disparate template formats and no template ID to go by. So, I spent some quality time with my favorite programming language, Python, and identified as many patterns as I could in a sampling of 5 million candidate strings. After much tweakage, my pattern recognition became 94.54% accurate. This rate was more than good enough to the users of the data so I basked in sweet triumph and called it a day :)

In another project I used frequency analysis and deduction to turn eBay auction titles into more meaningful identifiers of items up for sale. This worked fairly well because my dataset was large and all the auctions were for a specific kind of item.

My question to you is where can I find out more about this kind of fun stuff? Can you recommend a good book on data mining? Any good blogs to read? Should I take some math classes before I get too deep into this? If so, which ones?

The Monty Hall Problem (win a goat or a car)

posted in Python, Whatever by kumar on Wednesday Feb 13th, 2008 at 9:51a.m.

There is a puzzle used in game shows known as The Monty Hall Problem. It's been around for a while but over lunch yesterday someone explained it to me for the first time and 3 out of 4 of us argued convincingly the same answer. And it was wrong. Here's the problem:

Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice?

The most logical answer to me was no it doesn't matter if you switch or not because you never knew what was behind the first door you chose anyway. It seemed to me that the problem was no different than having two choices, a goat or a car and randomly choosing one. But this is all wrong!

Since I'm not good at math I could only loosely follow the explanations for why I was wrong. So, naturally, I wrote some code to see it in action (bear with me, I spent all of 5 minutes on it):

import random
from decimal import Decimal
choices = ['goat','goat','car']
tries = 99000
switch_correct, stay_correct = 0, 0
for num in range(tries):
    doors = [c for c in choices]
    random.shuffle(doors)
    first = doors.pop(random.randint(0,len(doors)-1))
    for i,val in enumerate(doors):
        if val == 'goat':
            doors.pop(i)
    switched = doors[0]
    if switched == 'car':
        switch_correct += 1
    elif first == 'car':
        stay_correct += 1
print "stay: you win %s%% of the time" % (Decimal(stay_correct) / Decimal(tries) * 100)
print "switch: you win %s%% of the time" % (Decimal(switch_correct) / Decimal(tries) * 100)

I found the result astonishing:

stay: you win 33.5% of the time
switch: you win 66.5% of the time

The wikipedia link above explains why this is but it is still incredible to me, like a magic trick.

Building Flash/ActionScript sites entirely in code and using FireBug for debugging

posted in ActionScript, Flash, JavaScript, Python, Testing by kumar on Monday Jan 28th, 2008 at 1:09p.m.

Way back in the olden days I used to dabble in Flash for building dynamic websites. This was incredibly painful, as I recall, because I couldn't stand the Flash IDE. It was clunky and hard to navigate, there weren't enough key commands, and the code editor was a sad version of notepad at best. And why use Flash anyway? JavaScript will do most of what you need for a dynamic site nowadays and I believe things like Google AdSense read the DOM for content filtering, which would render a Flash site useless. For me, the answer was audio playback. I wanted to play some audio on a site and for this Flash seems like the only option.

Digging deep into my vault of web dabblery I remembered getting so fed up with the Flash IDE that I wrote an entire site in ActionScript alone. I was quite proud of this as it was an early foray into programming and made me realize how much I enjoyed writing software in code. By some stroke of luck, this ActionScript-only site written circa 2002 is still online! OK, the intro was made with some timeline animation (someone else did that) but everything else is pure code, I swear.

So there I was, planning out my soon-to-blow-your-mind Flash audio player, yet without Creative Suite 3, the latest Flash IDE. In fact, I didn't want it — $699.00, ouch! Not to mention: the pain of installing another massive, bloated application on my hard drive. A little Googling around later, I discovered Motion-Twin ActionScript 2 Compiler, or mtasc for short, written in OCaml. This was exactly what I was hoping for, an open source command line tool for compiling ActionScript code into SWF (Flash) movies without the need for any clunky IDE. And it works very nicely.

Unfortanately, ActionScript seems to have no error handling whatsoever (or I haven't figured out how to handle errors yet) so doing stupid stuff like calling a method that doesn't exist will not stop program execution as it should. Great. So I quickly became familiar with hooks that mtasc provides to the trace() command. Calling trace() writes to the Flash IDE's debug log. But since you aren't in a Flash IDE when testing a freshly compiled swf, mtasc allows you to define your own implementation of trace. Cool! So naturally, I created my own trace that writes to the FireBug log. FireBug, of course, is the ultimate webapp-debugging tool.

Here it is first added to the example app (from the mtasc tutorial):

class Tuto {

    static var app : Tuto;

    function Tuto() {
        // creates a 'tf' TextField size 800x600 at pos 0,0
        _root.createTextField("tf",0,0,0,800,600);
        // write some text into it
        _root.tf.text = "Hello world !";
        
        trace("debugging with trace rocks!");
    }

    // entry point
    static function main(mc) {
        // note that this seems to help mtasc find the trace method
        var f = new FireTrace();
        app = new Tuto();
    }
}

...saved to Tuto.as. And here is my trace implementation, saved to FireTrace.as:

class FireTrace {
    static function trace(msg, class_, file, line) {
        getURL('javascript:console.debug("[' + file + ':' + line + ' ' + class_ + '] ' + msg + '")');
    }
}

I compiled it like so:

mtasc -swf Tuto.swf -trace FireTrace.trace -main -header 800:600:20 Tuto.as

And used a simple HTML page to run it:

<html>
<head>
<script type="text/javascript">

if (!window.console || !console.firebug)
{
    var names = ["log", "debug", "info", "warn", "error", "assert", "dir", "dirxml",
    "group", "groupEnd", "time", "timeEnd", "count", "trace", "profile", "profileEnd"];

    window.console = {};
    for (var i = 0; i < names.length; ++i)
        window.console[names[i]] = function() {}
}

</script>
</head>
<body>
<object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" name="TopPlayer" id="TopPlayer"
     codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=5,0,0,0"
     width="800" height="600">
 <param name="movie" value="Tuto.swf"> 
 <param name="quality" value="high">
 <param name="loop" value="false">
 <param name="play" value="true">
 <!--<param name="bgcolor" value="#ffffff">-->
 <param name="swliveconnect" value="true">

 <embed src="Tuto.swf" quality="high" width="800" height="600"
 	type="application/x-shockwave-flash" name="TopPlayer" id="TopPlayer"
 	pluginspage="http://www.macromedia.com/shockwave/download/index.cgi?p1_prod_version=shockwaveflash"
 	swliveconnect="true">
 </embed>
</object>
</body>
</html>

...and here is what it prints to my FireBug console:

[Tuto.as:11 Tuto::Tuto] debugging with trace rocks!

Notice that I had to add this line of code to the Tuto class:

var f = new FireTrace();

I don't know if this is a bug in mtasc but without it the custom trace seems like it never gets loaded. Also notice that I used the FireBug Lite JavaScript snippet for gracefully degrading console methods when FireBug isn't running.

After my romantic reunion with writing ActionScript, what has changed? The first thing I've noticed is that ActionScript 3 now looks exactly Java but I have no idea why. Luckily, it remains compatible with ActionScript 2 so you can leave out all the type declarations and everything still seems to work. Thus, it is pretty much a clone of JavaScript with some extra craziness you probably don't need. I'm still trying to figure out a better way to handle errors; putting debug statements everywhere is pretty lame. If anyone has a suggestion let me know. At the least, I would like to get an error in the log when I stupidly try to call a non-existant method.

Now that I have a prototype, the next step is to write automated tests, of course! This will be a lot of fun as I love watching tests pass. At first glance, the AsUnit (ActionScript Unit Test Runner) seems like it will be useful. I have no doubt that GUI testing Flash is no easier than it is in other languages, and thus not so easy to automate ... but one always needs unit tests.

UPDATE

Ah, looks like there is an even better way to call JavaScript by way of the ExternalInterface class:

ExternalInterface.call("console.log", variable1, variable2, variableN);

I haven't tried it but this article suggests to limit your calls to String, Number, Array, Object, and Boolean datatypes.

Software is written by hand

posted in JavaScript, Python, Ruby, The Future, Whatever by kumar on Friday Dec 21st, 2007 at 3:36p.m.

...that's right, it's not molded or prefabbed, it's not made on a production line or in a lab. Are we insane??! Here is a hilarious probe into the darker side of this art we call programming.

Converting ReStructuredText to Wiki syntax

posted in Python by kumar on Friday Dec 21st, 2007 at 12:45p.m.

I just released a little package called wikir that attempts to convert reStructuredText to Wiki syntax.

Why? Mostly because I have a lot of projects hosted on Google Code and reStructuredText is my documentation weapon of choice. It's also widely supported in the Python world, The Cheeseshop, pydoctor, etc.

The 0.1 release has some very basic things working. Next I hope to add an entry_point that can list modules and their respective wiki files for the publish_wiki command and have that hook into the standard wiki directory of a Google Code repository.

Leapfrog Online is looking for some Django developers (Chicago area)

posted in Chicago, Django, JavaScript, Python, Testing by kumar on Thursday Dec 13th, 2007 at 4:45p.m.

Leapfrog Online is looking to hire several Python developers to work on a Django site. If you know Python but not Django, this is an excellent opportunity to learn. If you know Django but want to learn how to use Python in other contexts, you'll get to do that, too. You'll be working on a high traffic website that hooks into several web services to help customers find Broadband Internet connectivity based on geo location (just US and Canada at the moment). Surrounding that basic function are all kinds of front-end and back-end features, services and systems.

The Software Engineer position is outlined in detail here.

You can send your resume to kumar.mcmillan@gmail.com or send it through the site above. These are full-time positions but if you'd rather work with us as a contractor that may be possible.

What Do We Do?

Leapfrog Online does performance-based customer acquisition, which translates to "we don't make money unless our clients make money." Because of this our software has to work well and we need to collect lots of structured, sensical data so our analysts can build the right marketing strategies. In a more abstract sense, the interesting challenges we face are building high-availability websites, fault-tolerant web services, pushing and pulling at hundreds of gigs of data, and accounting for tight security all along the way. As for the atmosphere, we're still a small company but we're not a struggling startup.

We Care About Open Source

We use open source tools that are right for the job. Currently we use Python or Ruby for websites / web services (Django, Pylons, Paste, Ruby on Rails), Python for backend tools, PostgreSQL for databases, and Trac for our projects. We use rich web interface libraries like Ext JS and we even wrote a distributed content system in Erlang because it was a good fit.

We have contributed patches to most of the projects listed above and maintain our own projects like nose, a few nose plugins, fixture, wsgi_intercept, and sv. We give talks at conferences like Pycon (see #24, #85 and #127). Also, Jeff Cohen (one of our senior developers) runs a popular blog called Softies on Rails and teaches and writes books about Rails.

Scrum: You'll Like It

We started with Extreme Programming a few years ago and have moved towards Scrum and other Agile methods as our approach to software development. We are constantly refining our process, keeping what works, discarding what doesn't. The company is on board with Scrum all the way up to the principles and we are always working to improve how Scrum is integrated holistically (a training program is in the works).

We think you'll like Agile for development. We have several teams of no more than three developers who work in two week "sprints." The sprints are planned out by product owners, developers, and project managers with user stories estimated in "story points" so that the business gets what it needs in order of priority. A sprint is exactly what it sounds like -- you just work! At the end of each sprint the work is released and you attend a retrospective meeting to see what was good, bad, and ugly, and how much work you did. Nothing is perfect so, of course, there are emergencies and derailments here and there but for the most part Scrum keeps things moving at a productive pace. As a developer, I find this discipline empowering and highly motivational.

You Must Test It

We are nutty about automated testing (in case you didn't notice). All code must have automated regression tests so if you're not familiar with this way of writing software, you will learn! We have a fairly involved continuous integration process running in buildbot (though probably moving to Bamboo soon) that performs several builds of each app, one with stable 3rd party libs, the others with trunk versions of 3rd party libs. As well as getting immediate feedback when a bad change is checked in, this also helps us pinpoint bugs in our dependencies before they are released. Our QA department is also different than most in that it consists of developers who are writing functional and/or integration tests in code and adding these to the build process. They are essentially software engineers like the rest of us.

Your Time Is Valuable

No one has a sleeping bad under their desk here; we work until 5 or 6 (weekdays only) to achieve a "sustainable pace." Most of us have been through the "death march" routine at other companies so we know it doesn't work long term. Scrum helps us maintain this ethic.

No Pigeonholes

While we are currently looking for Python/Django programmers, we are always interested in meeting people who think in Ruby/Rails, PHP 5 and other open source web technologies, too. We're especially interested if you're feeling ecumenical and want to learn about and work with, say, both Python and Ruby. You might only work in one language most of the time, but we think it is important for developers to stretch themselves and understand what tools are best for the job.

Datejs - A JavaScript Date Library

posted in JavaScript by kumar on Monday Dec 3rd, 2007 at 5:32p.m.

Found out about this nifty date package for JavaScript today: Datejs. Its chainable interface makes the code read very naturally:

// What date is next thursday?
Date.today().next().thursday();
 
// Add 3 days to Today
Date.today().add(3).days();
 
// Is today Friday?
Date.today().is().friday();

It also does quite a bit of natural English parsing, like Date.parse(‘next thursday’), although most date libraries support that too. I think the chainable interface stands out here as something special. Let's contrast that with some perl code to calculate yesterday, shall we??

my @lt = localtime(time - 86400);
return ($lt[5]+1900) .'-' . ($lt[4]+1) . '-' . ($lt[3]);

(Disclaimer: I didn't actually write that, but it's in a legacy app I sometimes maintain. I'm sure there is a cleaner way in perl to write that code, I just wouldn't know it, nor do I care to!)

WSGI Intercept Has A New Home

posted in Python, Testing by kumar on Monday Dec 3rd, 2007 at 9:46a.m.

A while ago I started using Titus Brown's wsgi_intercept module to test some XML services I had been working on. It was a great way to set up a stub response so that the tests never hit our real services. We even created a nifty little decorator that would look for an environment variable, $USE_REAL_SERVICE, then send the stub response when False or hit a "staged" version of the service when True. This allowed us to transparently perform a slower but more comprehensive test at will (currently running once a day from buildbot).

Although I was able to whip this up pretty quick, I had trouble importing from wsgi_intercept since all its pieces were in disparate modules. No biggie, so I bundled them all together and submitted a patch back to Titus. Me and my big mouth! One thing led to another and I've since agreed to take over maintenance of the project. But seriously, I'm happy to announce that wsgi_intercept has a new home and is available for download from the cheeseshop.

This first release tries to remain true to the interface that twill can work with, although a few import paths probably need changing and I'm not sure how easy it is to use a global version of wsgi_intercept with out-of-the-box twill.

I'm not thrilled about the current interface to wsgi_intercept so the next release will probably deprecate a lot of methods in lieu of a cleaner interface ;) If anyone has ideas for improvement please let me know, preferably as a feature request ticket. Stay tuned. Oh, it also needs some love for python 2.5 so the first person to submit me an issue for all the broken pieces is automatically "awesome."

Thoughts

disthelper

UPDATE

PyPi on the App Engine

Issues...

You Can Help

UPDATE

What Do We Do?

We Care About Open Source

Scrum: You'll Like It

You Must Test It

Your Time Is Valuable

No Pigeonholes

Recent Projects

Wikir

fixture

nosetty