There was much rejoicing on Twitter this afternoon over a blog posting.
Apparently, IMVU is rolling out fifty deployments each and every day, and they’re doing so by the magic of Continuous Deployment. “The high level of our process is dead simple: Continuously integrate (commit early and often). On commit automatically run all tests. If the tests pass deploy to the cluster. If the deploy succeeds, repeat.“
Some more details:
“Our tests suite takes nine minutes to run (distributed across 30-40 machines). Our code pushes take another six minutes. Since these two steps are pipelined that means at peak we’re pushing a new revision of the code to the website every nine minutes. That’s 6 deploys an hour. Even at that pace we’re often batching multiple commits into a single test/push cycle. On average we deploy new code fifty times a day.“
“So what magic happens in our test suite that allows us to skip having a manual Quality Assurance step in our deploy process? The magic is in the scope, scale and thoroughness. It’s a thousand test files and counting. 4.4 machine hours of automated tests to be exact. Over an hour of these tests are instances of Internet Explorer automatically clicking through use cases and asserting on behaviour, thanks to Selenium. The rest of the time is spent running unit tests that poke at classes and functions and running functional tests that make web requests and assert on results.“
Now: to be fair to Tim, he may be making claims about the deployment process only. That’s not clear to me from his posting. I inferred that he was making claims about the overall product or service.
If so, those claims sounded very promising. I wanted to test them against experience, so I went to IMVU to see what a robust, well-tested-by-automation application looks like. I also invited James Bach over Skype, impromptu, to try testing the service at the same time, so that I’d have someone to chat to; IMVU is, essentially, a virtual world and chat room. What follows is a merging of our very rough notes, based on about 25 minutes of testing for me, 15 for James. If we had been testing in a more formal setting (that is, in-house for the company as employees or contractors), rather than for fun on a Thursday night, we would have been more detailed and specific about recording our actions. We would likely have used BB Test Assistant or some other form of screen-and-audio capture that we could annotate and narrate. Working together, we were able to confirm a general problem that I found to be the most annoying bug: music was played automatically, intermittently, and loudly, and the volume control and the mute button that appeared intended to control the music didn’t ever work.
I started the session by making a testing mistake: I went looking for bugs too early. I often make this mistake, and need to adjust my process. The problem, as Jon Bach pointed out to me several years ago, is that if you look for bugs too early, you may end up finding them, investigating them, and recording them. That takes time away from getting an overview of the product, building your model, and understanding the benefits that the product is trying to offer. I stepped into this trap right away.
It’s a particularly bad trap, in fact, when the zeal for bugs prevents you from being able to log in properly. That’s what happened. After a certain amount of rather sophmoric input constraint attacks (the email field accepted at least 1,000,000 characters of data; I didn’t attack further), every attempt to use the signup screen looked like this:
“Please correct the following errors:
Error IMVU was unable to process your request”
I couldn’t find a way to apologize and make the application start working, even though I fed it grovellingly valid data. To save troubleshooting time, I ended up setting up an account on a second machine. Here are the notes of what James and I did after that.
- End and Shift-End keys don’t work in the “Choose Your Name” window.
- pressing “cancel” during install results in “Download Failed” message.
- IMVU (not responding) when trying to log in
- clicked on the “New? click here to register” link and it froze. It turns out that it froze while attempting to load the webpage, but there was about a minute and a half delay or hang
- unable to paste text into chat box
- couldn’t stop the avatar from dancing (later found out that if he sat down, he’d stop)
- in one of the rooms, I choose to move to a spot on the couch. I suddenly found my avatar arranged on the couch with a female avatar, such that our legs passed through one another in a rather painful and physically impossible kind of way.
- music (techno/hip-hop, often in Japanese) started and stopped randomly, without warning.
- the music was loud. The volume control and mute button on the music screen didn’t work at all. The music played at top volume at random times.
- music appears subject to intermittent dropouts.
- when I max out the characters in a message in the “special someone” box, I can still add additional characters one at a time, despite being told I’m at the max
- when I tried to become a “special someone” with myself, the error message was “you are not buddies with that avatar!”
- you can add yourself as a buddy
- after adding yourself as a buddy, you can invite yourself for a chat. The message “inviting…” appears in the top of the room window and is still there 10 minutes later
- after adding myself as buddy, I could add myself as a “special someone” to myself
- in one of the rooms, there was a dropdown box labeled Media/Music. Double-clicking on one of the options, “Country 108″, gave a dialog box captioned Message, with “Loading error!Please try again!” (with the missing space after the first exclamation point) and a close button that said “Sure”.
- with the uncontrolled music on in the room already, clicking on a working radio channel superimposes extra music over the existing music. Two tracks at the same time; din.
- clicking on the music channel very quickly after clicking the first time results in a stream played at half-speed or double or n-tuple speed.
- when the camera is tilted low, chat bubbles (dis)appear outside of the readable bounds of the window
- sometimes you can’t get to your name in the room window, so you can’t get to your menu of options
- in the menu, there’s an option for pasting text into the chat window. The first time I did that, I accidentally pasted the HTML source for the home page that was hanging around in my paste buffer. That text went out immediately to the chat room. There’s no way to buffer and identify what you’re pasting, as there is in Skype or Windows Live Messenger or other chat rooms
- the text that goes into the chat window is different from the text that appears in the bubbles in the virtual world. If it’s sufficiently long, it throws away most of the text, and truncates at either end and you only get to see the middle.
- why is the chat room text duplicated?
- in all rooms, the rendering routines allowed limbs to penetrate furniture or other people’s limbs. Or vice versa. You can see an example:
- my avatar has a necklace that appears to enter his back through the left shoulder blade, and then reappears at the front of the body, roughly through the heart. I know piercing is fashionable these days, but that looked painful.
- the lyrical content sounded a little rough to ears that were brought up on the Beatles. That’s not a big deal to me personally, but there was no parental advisory. Which itself isn’t a big deal unless some parent sues.
- chat text display isn’t apparently configurable
- the product shows significant inconsistencies in presentation, look, and feel. Many features appear to open a browser outside of the IMVU application; the design esthetic of the Web-based service is very different from the application in which the action happens.
- A prominent part of the screen in the www.imvu.com/shop window appears to be a message center. It turns out to be an adlink to myluvcrush.ca, which is one of those sites that posts two annoying modal dialogs before it lets you leave. (This isn’t necessarily an IMVU thing; I don’t know the relationship between the companies.)
Most of the comments on the blog post mentioned above were positive. I’ll summarize: “D0000DE! That is sew kewl. U R l33t!” The enthusiasm suggested two things to me. The first is that nobody apparently had a look at the product that was being deployed the way James and I did—or if they did, they were far too polite to make comments or raise questions about it. The roars of approval were, to me, like the blurbs on the back of a book—you know the ones, in which you’re skeptical that the reviewers ever read the book. There used to be a feature in Spy Magazine, “Logrolling In Our Time”. Every month, they would find a smattering of book blurbs in this pattern:
“Best book I’ve read in years. Squane does it again!” Raoul Duke, on the back cover of J.D. Squane’s Frequent Manhood
“Brilliant, incisive. Duke’s a genius” J.D. Squane, on the flyleaf of Raoul Duke’s Temporarily Unsanitary
The second point derives from the first: that, among the technologists, there’s such a strong fetish for the technology—the automated deployment—that what is being deployed is incidental to the conversation. Yes, folks, you can deploy 50 times a day. If you don’t care about the quality of what you’re deploying, you can meet any other requirement, to paraphrase Jerry Weinberg. If you’re willing to settle for a system that looks like this and accept the risk of the Black Swan that manifests as a privacy or security or database-lclearing problem, you really don’t need testers.
Except… maybe I’m wrong about all this stuff. Quality is value to some person.
- Maybe I’m not the person to make the evaluation. In the screen shot above, note the conversation that I’m having with Guest_SexyRocker1. I’m asking her if she finds bugs in this application. She says No. I’ve found it impossible not to find bugs.
- Maybe these applications aren’t supposed to have elegance or polish; maybe audio controls aren’t supposed to work; maybe it’s cool, in a virtual world, to render furniture embedded in people.
- Maybe this is just a proof of concept. If you look really closely, you’ll notice that the IMVU logo is occasionally accompanied by “Beta”. Maybe the Perpetual Beta, a release model that Google seems to have perfected, is the new normal, allowing us to forgive all sins.
- Maybe, to use Bob Martin’s lingo, we don’t value craftsmanship over crap.
- Maybe we expect things to have problems that can be revealed by these terribly basic tests. And maybe, in the sense that James is talking about quality here, quality is dead.