All About the Code

I have the exact same challenge as Kenneth Kvalvik (article in Norwegian): A 6 year old boy who is starting to read but can’t find anything he likes to read. He hates home work assignments (and who can blame him, they’re boring). And as the son of the author of the article above, my son also loves Star Wars. He hasn’t seen any of the films but plays star wars games quite diligently on his Nintendo DS and plays with Star Wars lego a lot.

So why not give it a go?

The originator bought a star wars book and translated it and added images from the movies. I skipped that step and went straight to the movies. In part because after reading for only a couple of months (and having only covered about half the alphabet in detail) my son needs very simple text.

One tough question is: which order should I present the movies in? Original trilogy first or prequel first? I feel every fiber of my Star Wars fanboy body resisting, but I chose prequel trilogy first. Anything else would probably add to the confusion.

The Plan

My son also loves using the IPad, so I thought I’d take advantage of that and create an ebook app (or “app”) for that platform. I also needed to be able to add more pages quickly (if it was a hit). So I landed on using plain HTML5 and just creating a web page and use the “add to home screen” feature and make it look like an app.

I am using swipe.js which is very lightweight javascript library for just doing swipe gestures on the IPad. I checked out both Sencha Touch and JQTouch (which I have used before) but for this small project they were both overkill.

The layout of the app is a simple directory structure, one folder for a book, with one sub folder for a chapter with images and videos for that chapter. Each chapter is one html file, with divs that swipe.js is showing and hiding as you swipe. Each page has either an image or a video, and from one to three short sentences in uppercase (written in lowercase and text-transformed with css, so I can lowercase it again when he’s better at reading).

I went with using movie screen shots for some pages (and the parts where we need some censorship of gory details (not much gore in Star Wars but for a six year old it can be a bit scary)). I also added some small movie clips to some of the pages to keep his interest up and give him a little break in the reading. I researched som HTML5 video players, but found that they all, as per usual, sucked in their own unique way, and went with the safari default video player on the IPad. Which works wonderfully for my use.

Another takeaway is how much you can cut away from those movies (especially the prequel trilogy that, well, isn’t all that good) without losing the core of the history. The second movie with all that sticky love stuff between Anakin and Padmè will be cut even more. 6yo boys are not that into romance.

Results

So how did this turn out? I created two chapters of the first book before starting a beta test on my first (and only (non-paying)) customer. He read everything in one sitting and immediately asked for more chapters. And that question was repeated every day over the following weeks where I hadn’t had the time to write more chapters. Never had such a pushy user before!

Technically I have some duplication in the code, specifically the chapter menu. The layout isn’t all that great either, but I plan to fix that soon. The customer was so demanding that I was forced to take on some technical debt to get the product out into the market :-) I have also planned to add slideshow variant to make images clickable and go full screen.

Now we’re at the end of the first book, and he is asking when the second books becomes available. So I would say in terms of getting a reluctant 6 year old to read, it’s a huge success. He want’s to read it again too. He comes for the action sequences in the video clips, but there are no protests when I say we should read too.

(Since these movies are all copyrighted, and in these dire #SOPA times, I can’t publish the finished results anywhere, I’m afraid. The text, in Norwegian, is in the code over at Github, but the images and videos are not. And won’t be.)

(This was written just after the Smidig conference but got lost in daily life and something else (probably very important!). Just posted it to get it out of the way)

I attended the 5th Norwegian agile conference Smidig in November, and as any good agilist, I thought I’d do a little retrospective for the conference. Often the good ideas flurry at the conference, but when reality sets in when you return to work, the good ideas have a tendency to move to the back of the consciousness and not get tried. So, in order to prevent that here’s a list of impressions and more importantly a list of things to try out at work.

1. Pull based Meetings

This is an idea I had before the conference, but it solidified during it. Instead of running regularly scheduled demos, plannings and backlog grooming meetings, make them happen on-demand. Demo when you have something to demo, estimate and ready stories when backlog is getting slim. And perhaps use shorter, more frequent meetings instead of longer, infrequent meetings. Long meetings suck.

2. Code Review as Step on The Board

We have an informal, voluntary code review step, but use is infrequent and varies a lot. After a lunch talk with my former colleague Rune Sundling, who work in kanban heavy shop, I thought it worth a try to create a code review step before “done”. Wonder how the team will react to the suggestion, though :-)

3. Defining requirements quantitatively

I didn’t catch Tom Gilbs talk at smidig, but I know his ideas on this subject. And I am in a position to try this out now. A lot of work will have to go into finding the right quantitative measures for the requirements, but I like the idea and I think it fits our domain and prioritization process well.

Btw, the talk videos are already available on the smidig vimeo account (first videos appeared the same day the talks were held!)

Second day of Roots 2011 is over and here are my highlights:

The 7 Duties

Opening keynote of the day was Jurgen Appelo with a talk on the 7 duties of a software professional. Basically how and why you should work on improving yourself in every aspect: motivating yourself, direct yourself, connect yourself (to others), measure yourself and so on. A previously recorded version of this talk, with jokes about the Germans instead of the Swedes is on the internet. All in all a great talk and nice advice for moving forward instead of standing still in your life and career.

Graph Goodness

After that came Jim Webbers Neo4j hacking tutorial by going through (some of) the neo4j koans koans to get a feel for the neo4j API and how to program it. Didn't have time to cover a whole lot, but I will try to go through the rest of Koans back home. I have used Neo4j a bit earlier, but I want to get the feel of it in a larger context and use the API more thoroughly.

Secrets of Agile Architecture

Dan North was on once again after lunch with a talk on Secrets of Agile Architecture. He was all over the map talking about topics such as treating organisations as systems and how changing the architecture changes the organisation, managing impact instead of obsessing about likelihood of options and choices. Conway's law and Dan's corollary to it, Real Options, PDCA, the role of the architect as a minstrel in modern software development and much much more. Dan is never short on ideas. Some cognitive overload happened there and I need some time to digest all the ideas. Nice talk.

HAML And SASS

Anders Norås then again took over with a tuturial on using HAML and SASS for writing web applications. Great intro and kudos for creating usb sticks with virtual image with pre-installed software. I have used these two technologies a bit before but I joined anyway to pick up some new tricks. Nothing revolutionary there, but a nice reminder that I use it to seldom in daily life. Need to prioritize my rails projects now that the talk for Roots is out of the way.

I'm here at the Roots 2011 conference in Bergen and the program is packed with wonderful stuff. So I thought I'd venture into liveblogging/core dumping the interesting stuff I see. 

This is part II in a blog mini series on building a build pipeline with Jenkins. The other parts, part I and part III could be worth a read too.

This part is about a implementing built-in smoke tests in a rails application. This Is by no means new, but I added a little something at the end which I thought was quite nifty.

Class definition

I created a small class to hold the actual smoke tests (pardon my ruby, I'm learning):

class Smoketest
  attr_accessor :description, :status, :message

  def initialize(descr)
    self.description = descr
  end

  def run
    yield
  end

  def self.OK
    "OK"
  end

  def self.FAIL
    "FAIL"
  end

end

These are called from the SmoketestController. I chose to just put all (well all two) smoke tests in the controller, but they could easily be distributed and put wherever you want. The controller methods looks like this:

require 'smoketest'

class SmoketestController < ApplicationController

  def index
    @tests = Array.new

    mongodb_smoke_test
    test_test

    respond_to do |format|
      format.html
      format.xml { render :action => 'index.xml.haml',
                          :layout => false }
    end
  end


  def test_test
    test2 = Smoketest.new("Test")
    test2.run {
      test2.status = Smoketest.OK
    }

    @tests << test2
  end


  def mongodb_smoke_test
    test = Smoketest.new("MongoDB test connection")

    test.run {
      begin
        User.first
        test.status = Smoketest.OK
      rescue Exception => e
        test.status = Smoketest.FAIL
        test.message = e.message
      end
    }

    @tests << test
  end

end

The "test" one is just for making it more than one, and the mongoDB test tries to do a query and gives it and ok status if that works, and a fail with stack trace if somethings amiss. In a bigger application with more external systems, the number of smoke tests would go up. Running this, gives this nice web page for visual inspection:

(The layout is a bit wide since I wrapped in the standard layout for the app.)

The Junit output

I wanted the jenkins build job to track these smoke tests, which are being run as a build step after deployment. The easiest way to do this I figured was to make it spit out Junit XML format and let Jenkins chew on that. So a created an XML template like so:

%testsuite{:name => "smoketests", :failures => @tests.select { |x| x.status == "FAIL" }.count, :tests => @tests.count, :skipped => 0}
  - @tests.each do |test|
    %testcase{:name => test.description}
      - if test.status != "OK"
        %failure{:message => test.message}

And when fetching the url http://server/smoketest.xml you get the following output:

(It handles failures too).

The following small shell script is used to check the status:

#!/bin/bash

if [ -z "$1" ]
then
    echo "usage: smoketest.sh "
    exit 1
fi

status=`curl --silent --head $1 | head -1 | cut -f 2 -d' '`

if [ $status != "200" ]
then
    echo "status was other than 200: was $status"
    exit 1
fi

if [ ! -e "reports" ]
then
  mkdir "reports"
fi

#put the xml version of the page into a file
curl --silent $1 > reports/smoke.xml

The shell scripts stores the file which is then read by Jenkins. Works like a charm!

Stay tuned for the third article on build pipelines where I explore some future possibilities when using a build pipeline.

I have been reading the excellent book Continuous Delivery by Jez Humble and Dave Farley and been dying to try out some of it in practice. Since work these days is filled with big not-so-easy-to-change java legacy systems, a small greenfield ruby project was the way to go.

This article ran long, so it is split into three parts:

• part I (this part): Jenkins setup and project setup.
• part II: Ruby Smoketest Implementation
• part III: Further Build Steps

My aim was to deploy the Ruby on Rails project to heroku which is a nice and easy-to-use cloudy service for hosting ruby apps. Elegant and easy on both web interface and command-line tools.

The Project Setup

The project I just started is a Rails3 app using bundler and running in ruby 1.9.2 using rvm. This, it turns out, brings some special considerations to the mix. Just for kicks and because I start to dislike SQL databases more and more, especially for the cases where you don't really need it, I use MongoDB and Mongoid as the ORM. MongoDB is supported by Heroku so all's good there.

One gotcha it is easy to run into with Heroku and a rails app is that you should have your rails app be at the top level of the git repo. It is supported to run with a rails app in a sub directory, as I did, and you can configure it to work via the config.ru file, but bundler does not seem to run well on heroku when you do that. So keep the rails app on the top level if you can.

The projects uses rspec for unit tests and some rails-specific integration/functional tests, and cucumber for end-to-end acceptance style tests. More on that when it comes to the deployment pipeline and jenkins project setup.

Jenkins Setup

I belong to the school that dictates that the first thing you should do with a project is to deploy it and then setup CI for it and make that deploy for you. The rake plugin for Jenkins supports rvm so we can use rake tasks directly on the project configuration.

First of all, I needed cucumber and rspec to output junit format test results to make pretty graphs and record status in Jenkins. Cucumber has a handy --format junit, but rspec does not. Luckily the gem cireporter does the trick for us. But some customs rake tasks are in order. This is the file _lib/tasks/jenkins.rake:

namespace :jenkins do
  def cucumber_report_path
    "reports/features/"
  end

  def rspec_report_path
    "reports/rspec/"
  end

  Cucumber::Rake::Task.new({'cucumber'  => ['db:test:prepare']}) do |t|
    t.cucumber_opts = %{--profile default  --format junit --out #{cucumber_report_path}}
  end

  task :cukes_report_setup do
    rm_rf cucumber_report_path
    mkdir_p cucumber_report_path
  end

  task :spec_report_setup do
    rm_rf rspec_report_path
    mkdir_p rspec_report_path
  end


  task :ci => [:report_setup, 'jenkins:setup:rspec', 'rake:spec', 'cucumber']

  task :unit => [:spec_report_setup, "jenkins:setup:rspec", 'rake:spec:lib']

  task :functional => [:cukes_report_setup,
                       :spec_report_setup,
                       "jenkins:setup:rspec",
                       'rake:spec',
                       'cucumber', ]

  namespace :setup do
    task :pre_ci do
      ENV["CI_REPORTS"] = rspec_report_path
      gem 'ci_reporter'
      require 'ci/reporter/rake/rspec'
    end
    task :rspec => [:pre_ci, "ci:setup:rspec"]
  end

end

It does some setup of of report output directories, some setup of cireporter in the :setup task and makes sure rspec tasks does some setup first. The tasks called from jenkins are jenkins:unit for the first build step and jenkins:functional for the second step. The jenkins:ci task is an all in one if that's your bag. The main point is that the first step runs unit test, hence rake:spec:lib only, and the second step runs all tests and metrics. Ideally if the metricfu plugin in Jenkins had supported bundler well, I would like to run some metrics in the first step and fail the build if values where below or above a limit. But that does not seem to fly easily.

I used a small shell script to deploy to Heroku and the test server instead of using the git publisher plugin. This is to make the deployment build step fail the build. The git publisher plugin is run as a post build action and this won't fail the build. Likewise, the smoke test script needs to run as a build step since the recording of test results is done post build.

And here's where the ideals of Continuous Delivery (CD) kicks in. The book advocates doing the build in several steps and the central point is fast feedback. The first step is compile, fast-running unit tests, metrics and building of artifacts (if you're on a platform that does that). And if all that passes should you go on to step two using the same artifacts created in step one, possibly aided by a repository manager of some sort to take care of the files in between steps.

If the first step fails you get unit level and metric violations feedback. If the second fails, you get system level functionality feedback. If later steps involving e.g. performance testing and manual testing you get feedback on those. But you don't go to that step with a bad build from step one.

The build pipeline plugin basically gives you a view and a(nother) way of specifying upstream/downstream projects. The view looks like this after it is configured according to this article with the four build steps in place.

The build pipeline plugin is in version 1.0.0 and has some major drawbacks. The main one being that it doesn't stop the pipeline when one step fails. This is a bug and it has been reported on the issue tracker already. I would guess this is fixed in the next release. Another is that it does not seem to be easy to pass params from one build step to the next. There has been some mention of using the parameterized build plugin for that, but haven't had a chance to test it yet and I don't know how well the two plugins play well together.

The Build

So in my case, the first step runs unit tests (spec:lib) and the second runs all spec, cucumber features and metric_fu for metrics. The third step deploys to my local test server and runs an automated smoke test. See part II for details on the implementation of that. The output is in junit format, so Jenkins records it and outputs it along with the other tests. If that test brake, the build in step 3 brakes too. I would like to rollback to the previous version on build failure, but I'm not there yet. This article on using capistrano looks interesting in that regard

The fourth step deploys to heroku via a script and runs the same smoke test as the step before.

Benefits and Ending Thoughts

The biggest benefit of using a pipeline is more fine grained feedback on the build process. The compile step is fast and fails almost immediately. The separate steps for integration tests and deployment to test server and heroku gives more accurate feedback on those steps too. I mean, combining deployment to test and production in a single step would be nonsensical. And you can do the deployment to production (or test) a manual one, which the plugin supports. For instance only after manual testing in the test environment. Combined with Jenkins' matrix style security model you can grant access to certain users for running that build step. There are a lot of possibilities and I will be exploring more of them in part three of the series. Part two is implementation of the smoke tests in ruby.

I had an idea in a twitter discussion with @olecr about automating different aspects of the development process, coupled with and in-house discussion at work on maintaining database test data for integration tests, which can be a real pain. How about hooking into migrations to rewrite/update fixture data when migrating database schema?

In ruby one approach to test data, is to use fixtures, often written in yaml, to define test data sets, which gets inserted into the test database before running tests. But you still have to evolve the fixtures when the schema evolves. And the more often it changes, the more of a maintenance nightmare you got on your hands. Other approaches can be to have object factories that create data in the database, or SQL script that gen run (which is what yaml fixtures are, only in a more sane and not-so-verbose format). But the maintenance is there no matter what the approach.

And then the idea struck me: if you use fixtures for inserting test data into the database before running tests (or between tests) and migrations to modify the schema of the database as you develop the application, couldn't you create a clever hack that modifies the fixtures to match the migrations? All in one operation? And you can minimize the manual maintenance to the stuff that is really hard to automate and to sanity check to results.

Yaml is machine-readable format, so one approach could be to read the migrations, find the diff for the migration in question, read the yaml fixtures, modify them and write them out again. In ruby/rake this could be some custom rake tasks that squeezes in with the migrations or something to that effect.

I would think this could work for things like deleting columns from a table (delete that key in the yaml file), adding new columns (add columns in fixture with dummy value according to data type) and column and table renaming. Some user edits would be necessary nonetheless.

What do you think? Viable idea or just plain stupid?

The logical next step of rewriting the test code using these fixtures to match the new schema, well I leave that as an exercise to the reader... :-)

In the latest episode of The Agile Toolkit podcast in an interview with Dave Thomas about, well just about everything, they talk jokingly about the notion of a certification for keyboard shortcuts and I thought, wow, that is just a fantastic idea! This would be a really useful certification as opposed to the many completely or nearly completely useless certifications out there. None mentioned none forgotten.

Shortcuts makes you more productive all over the place. In your IDE, on the command line and all sorts of other tools. Not having to remove your hand from the keyboard in order to use the mouse saves you time and concentration. You start out by typing, move your one hand to use the mouse, possibly moving your eyes to see the mouse, then moving the mouse pointer to the menu item or button you want, clicking it and then moving your hand back to the keyboard. When the shortcut is second nature you type, enter shortcut and continue typing. No lateral movement of neither hand(s) nor eyes (except to see result of command, but that is true of not using the shortcut too).

The effect is more flow in your work. Most developer know when they are in the zone and want to stay there to keep the code flowing. Every little distraction should be avoided to maintain the state of flow. Agreed, using the mouse is not a big distraction from the flow, but everything helps, right?

If you get repetitive stress injuries from using the mouse, using the keyboard could help make them go away. Just make sure you get a good quality keyboard, adjust your desk and chair to the correct height and take frequent short brakes.

So How Should We Do This in Practice?

There is the trouble of all the different sets of keyboard shortcuts for different tools, of course. Mac users have an advantage here since the platform streamlines a bunch of shortcuts. Tools like IntelliJ IDEA and Resharper which lets you choose between different shortcut schemes (us Emacs users love the Emacs shortcut scheme, at least I do, but sadly it doesn't quite fit in IntelliJ on the Mac) and you can use the same one in several tools.

So I guess we need different certifications for different tools, that would make sense. You could then embroider your CV with the title "Certified Mac Keyboard Developer", Certified Microsoft 2010 Visual Studio Keyboard Core Foundation User" (the Microsoft marketing division can get a say in this, I don't care), or Certificed Emacs Hacker.

And the actual test? Give the candidate a list of tasks to to (normal stuff every developer needs like opening files, navigating project structure, running unit test, build/package/deploy, the most common refactoring shortcuts, snippets, code formatting, line navigation and editing and other useful stuff. And just simply disable the mouse or remove it altogether. If they're done within the time frame, they're certified! I see a million dollars coming my way with this scheme.

Command Line Users Have the Edge

Users accustomed to using the command line for all sorts of tasks are used to remembering commands and shortcuts to greater extent than "normal" windows users (I use the command line in windows too, but not the crummy built-in cmd.exe) and could get certified more easily. The history navigation in the unix shell, combined with aliases and piping is awesome for productivity.

Training For the Exam

There are some techniques for training for this kind of certification. First there are cheat sheets that should decorate your desktop or cubicle wall and http://www.cheat-sheets.org/ has a huge collection. Some java IDEs have plugins that will remind you of the fact that a shortcut exists if you select a menu item with the mouse. Some can even be configured to disable the menu item if you won't take the hint. Lastly just time yourself doing something you do every day and avoid the mouse altogether and see the effects for yourself.

Disabling the mouse for some period of time could also be tried. It hurts in the beginning, but you should quickly get over it. Jon Galloway also has some good suggestions in his blog post on mouseless computing.

I for one try to use the mouse as little a possible.

I am doing a C# project at the moment where we are using log4net to log to the Windows event log. From the windows service this is straight forward but it turns out when you are trying to do this from an application running in IIS, a WCF service in our case, this is surprisingly hard. So I thought I'd document this one for later reference.

Those Pesky Permissions

Just duplicating the config for a normal windows application doing perfectly good event logging isn't going to work as you will quickly discover. A good way to debug this is to set up a FileAppender to log to a temporary directory somewhere. Log lines ends up there when the config is read OK, but not in the event log. It's a permission thing. The user running IIS probably has got privileges to log to the event log, but not create the event source initially. Nor should it, since running IIS as a privileged user is security disaster waiting to happen. Don't do it.

The config used for logging in this example is this:

The Solution

Step one is to create the event source by hand as a normal user. You can do this via the following code in a C# app.

EventLog.CreateEventSource(source, "My Application")

Or do it from the command line using the eventcreate program

c:\» eventcreate /ID 1 /L APPLICATION /T INFORMATION /SO "My Application" /D "Dummy log message"

This gives you a "dummy log message" in the application log. And you should be good to go. But not so fast. Eventcreate requires an EventID between 1 and 1000. Default logging from log4net for some reason uses event id 0, which will give you an event log error message in the application log. It contains your log message but it looks like a mess. So how to persuade log4net to use event id 1, which we used when creating the event source?

There are two ways of doing this. One is to use the log4net extension log4net.Ext.EventID and configure it there. If that isn't your cup of tea, you can quite easily, although a bit hackish, do it directly in your code before any log messages are sent with this piece of code:

log4net.ThreadContext.Properties["EventID"] = 1;

For instance in a constructor or other suitable initialization code.