Pragmatic Automation

Rapid Feedback

James Duncan Davidson and I are building a Rails application for a client. We needed a way to synchronize the code in the Subversion repository with the code being run on the shared test server. That way, everyone on the team could benefit from rapid feedback.

Time for a little automation. So James cooked up a Subversion post-commit script which I’ve posted on my blog.

Tell Me Where It Hurts

: Minutes ago, you released the latest version of your buzzword-compliant hosted application to your most important customer. It’s no small accomplishment for the team and is a potentially huge deal for your company. Your face lights up with a big, toothy grin as a wave of high-fives goes around the team. When the excitement turns into a collective sigh of relief, the team huddles up to depart for a well-deserved celebration dinner. Just then, the sound of a ringing telephone cuts through the air, freezing everyone in their tracks. It’s the support phone, and that could only mean one thing: deployment trouble.
: (Full article: PDF)

Ah yes, if you haven’t been there then count yourself lucky. Getting an application properly seated in its production environment, time after time, requires careful planning and inspection. The slightest misconfiguration can tie up endless hours troubleshooting and quickly rack up cell phone minutes.

Read how to write and distribute diagnostic tests that let your application do it’s own first-level support in my recent article (pdf) for Better Software magazine.

Automation Saves Christmas

Last Christmas, Simon Chappell and his team leaned on software automation to keep up with the pace of warehouse automation. Santa only wishes his elves were this resourceful. Simon writes:

Last year, Lands’ End selected its strategic J2EE application server and the word came down from upon high that projects should port to it as quickly as possible. Because life has a sense of humour and I’m often the punchline, it was decided that our project should be the first one to make the transition.

Being a big and expensive application server (no names to protect the guilty), it lives on its own server and mere programmers are not allowed to touch it. This eliminated the easy deploy that we had previously enjoyed. But the spirit of pragmatism will not be deprived! Now our Ant script has a target for building the required EAR file and then we run a second script to use secure copy to move the file to a directory on a designated machine.

For our web application, we had special capacity needs above and beyond the capabilities of the internal toolset. While our public facing website is very well tested, it is done using an external service. This was not an option for our internal web application. Yet, the load it was required to handle was above anything that we could generate with even our whole team rapidly hitting it from our workstations. By using JUnitPerf, I was able to construct a test that simulated anything from 1 to 100 simultaneous users, using HttpUnit to interact with our web application and with each simulated user using unique data.

With the realistic traffic that we generated, we were able to identify and resolve several bottlenecks in our system’s performance. The day was saved. Our system took us through our Christmas peak season with no support calls. And with the tests being scripted, we can re-run them at anytime that we please: after changes to the internal infrastructure, or as we recently did to test that our system would run unaffected by our newest companion system in the J2EE cluster.

If you can’t automate everything, you automate what you can.

I was teaching at Lands’ End earlier this month and got to see Simon’s web application being put to the task during this year’s peak shopping season. At stations on the warehouse floor, amidst a dizzying network of conveyor belts loaded with holiday gifts, an army of workers use the web application to help package and ship orders. It seemed only fitting that delivering the software to these work stations be as automated as delivering the packages to your house. Seeing this well-oiled machine in action, I just kept repeating one word: Wow!

One Build File, Many Deployment Environments

Daniel Frey sent in this report describing how he uses Ant’s immutable properties and code generation to build a J2EE application for any given deployment environment at the push of a button:

I was looking for a way to manage automation for a web-based, J2EE system deployed into development, QA, pre-production, and production environments, each of which uses a WebLogic application server and an Oracle database. My biggest concern was the configuration data for the database and application server. A "push button" build would only be possible for one environment without some way to parameterize the configuration data. Ant has facilities to aid in this, so I expanded on what I read in Pragmatic Project Automation to help me achieve this cross-environment deployment automation.

Before describing how I created my "push button" build, let me briefly explain our system. It’s a J2EE-based application in a tiered architecture. Persistence for the proof of concept was achieved using a MySQL database. Hibernate is providing the O/R mapping capabilities and the Spring framework is wiring all of this up together. Hibernate provides Ant tasks for generating .java files and building the database structure from Hibernate XML files. I am also using DBUnit as a means of populating test data.

I created my project directory structure as defined in Pragmatic Project Automation with a couple of minor additions. First, I added a config directory to the root of my project directory structure. The config directory contains various configuration files— properties files and XML files—for the deployment environment. These configuration files contain place holders (tokens) for deployment information such as what database to use. The tokens are replaced with actual values when Ant copies the configuration files to the build/prod directory. Here is the config Ant target that performs the copy:

  <target name="config" depends="prepare">

    <copy todir="${build.prod.dir}">
      <fileset dir="${config.dir}">
        <include name="*.properties" />
        <include name="*.xml" />
      </fileset>
      <filterset begintoken="%" endtoken="%">
        <filter token="HIBERNATE.DIALECT" value="${hibernate.dialect}" />
        <filter token="HIBERNATE.DRIVER" value="${hibernate.connection.driver_class}" />
        <filter token="HIBERNATE.URL" value="${hibernate.connection.url}" />
        <filter token="HIBERNATE.USER" value="${hibernate.connection.username}" />
        <filter token="HIBERNATE.PASSWORD" value="${hibernate.connection.password}" />
      </filterset>
    </copy>

  </target>

This copies all of the configuration files in the config directory to the build/prod directory. As the files are copied, the filterset replaces all of the tokens observed in the configuration files with the appropriate values. All of the token values are supplied in one properties file that I specify when Ant runs the build file. For example, when I want to deploy into an environment that uses MySQL, I use a properties file that contains these key/value pairs:

  hibernate.dialect=net.sf.hibernate.dialect.MySQLDialect
  hibernate.connection.driver_class=com.mysql.jdbc.Driver
  hibernate.connection.url=jdbc:mysql://server/database
  hibernate.connection.username=user
  hibernate.connection.password=password

Using this file, any configuration file in the config directory that contains the token %HIBERNATE_USER%, for example, will be changed to reference the value user. I have a default properties file, but I can change that via the command line when Ant is run. For example, if I want to deploy to the QA environment, I use:

  $ ant -propertyfile qa.properties

I can also override specific properties on the command line. For example, to override the value of hibernate.connection.username, I’d type:

  $ ant -Dhibernate.connection.username=admin -propertyfile qa.properties

I also added a stage directory to my build directory to store the .java files generated by Hibernate. I needed a separate staging area for these files because I did not want them to be in the src directory. These are generated files and can be re-generated at any given time by executing the codegen target:

  <target name="codegen" depends="config">

    <taskdef name="hbm2java"
             classname="net.sf.hibernate.tool.hbm2java.Hbm2JavaTask"
             classpathref="project.classpath" />

    <hbm2java output="${build.stage.dir}">
      <fileset dir="${src.dir}">
        <include name="**/*.hbm.xml" />
      </fileset>
    </hbm2java>

    <copy todir="${build.prod.dir}">
      <fileset dir="${src.dir}">
        <include name="**/*.hbm.xml" />
      </fileset>
    </copy>

  </target>

The hbm2java task generates the .java files into the build/stage directory from the Hibernate .hbm.xml files in the src directory. The <copy> task just moves the .hbm.xml files to the build/prod directory so that they are available on the classpath for the Spring framework to find.

After all the configuration is complete and the .java files are generated, Ant then compiles everything:

  <target name="compile" depends="codegen">
    <javac srcdir="${build.stage.dir}" destdir="${build.prod.dir}">
      <classpath refid="project.classpath" />
    </javac>
    <javac srcdir="${src.dir}" destdir="${build.prod.dir}">
      <classpath refid="project.classpath" />
    </javac>
  </target>

However, I don’t run my tests right away. Hibernate must first generate and execute the DDL to the MySQL database in order to set up the database structure:

  <target name="schema" depends="compile">

    <taskdef name="schemaexport"
             classname="net.sf.hibernate.tool.hbm2ddl.SchemaExportTask"
             classpathref="project.classpath" />

    <schemaexport properties="${build.prod.dir}/hibernate.properties"
      quiet="no" text="no" drop="no" delimiter=";">
      <fileset dir="${build.prod.dir}">
        <include name="**/*.hbm.xml" />
      </fileset>
    </schemaexport>

  </target>

Then I use DBUnit to populate my test data:

  <target name="setupDB" depends="schema">

    <taskdef name="dbunit"
             classname="org.dbunit.ant.DbUnitTask">
      <classpath refid="project.classpath" />
    </taskdef>

    <dbunit driver="${hibernate.connection.driver_class}"
            url="${hibernate.connection.url}"
            userid="${hibernate.connection.username}"
            password="${hibernate.connection.password}">
      <operation type="CLEAN_INSERT" src="db/TestData.xml" />
    </dbunit>

  </target>

Finally, Ant runs the test target to compile and execute my tests. With these build targets chained together as I’ve described, I can execute the test target and everything is run at the push of a button.

This has proved to be very useful and successful every time I execute the build script. It also provides a number of advantages. First, I can execute this on each of my systems, changing only the properties file referenced on the command line. Second, I can easily change the target database because I am using Hibernate to provide O/R mapping capabilities. To accomplish this, all I have to do is change the value of the Hibernate properties in one properties file or on the command line when Ant is run. I have successfully done this between MySQL and HSQLDB.

YIGNI

Glenn Vanderburg shares this story from the field about why you should always assume that you’ll need a real programming language if you want to allow your software system to be automated through commands, and how doing so can make your system more robust:

In 1999 I was consulting for a company that builds telecom equipment. They were developing a new variety of … well, let’s just call it a "local area telecom network." It included multiple nodes for traffic switching, call control and setup, and administration. It was to be a full-fledged product, but (as so often happens) there was an artificially imposed deadline: they wanted to demonstrate the product at a trade show in just a few months. Those kinds of things can be death to a project, but this company was attacking it rationally—for the demo, they were quickly pulling together "version 1" using existing internal simulation, testing, and prototyping tools, while a different team was working on building "version 1.5" from scratch, and there was a reasonable amount of communication and feedback between the two teams.

I was working on the operations and management system for the version 1 team, and was asked to design and build a system that could start and configure the various separate processes on a node, hooking them together so that they would be ready to join the network and start doing work. The testing group had an existing tool that did a similar job, and although it wasn’t suitable for use even in the demo product, the team asked that I keep the command structure of that testing tool. I don’t remember the details, but that tool could respond to three or four very simple commands: "connect," "send," "wait," for example.

At that time I was a fan of the Tcl language, which was designed to be a neutral, extensible, embedded command language for applications. Tcl’s inventor, John Ousterhout, designed the language after noticing a pattern in application development: nearly every application needs a command language of some sort, but the developers are focused on the application itself, so they cut corners with the command language, thinking they can get by with less. The command languages end up having "insufficient power and clumsy syntax," in Ousterhout’s words … and then, contrary to the developers’ expectations, users find that they need to use that command language to do complex things.

I saw this pattern developing in our telecom system. The command language I was asked to develop was almost hopelessly simple, but the people asking for it were adamant that it was all they needed. And yet I noticed that the command syntax was completely compatible with Tcl syntax. Plus, an implementation of Tcl atop Java, called Jacl, was available for me to use. Because Jacl, like Tcl, was designed to be extended with application-specific commands, I made the case that it would be just as easy—probably easier, in fact—for me to implement the desired commands as Jacl extensions than to write a custom parser. Then they would have a real programming language at their disposal "in the unlikely event that they ever needed it." My clients reluctantly agreed to this plan, but only because it would not involve extra cost. They were sure they knew their needs.

Implementing the commands as Jacl extensions was easy, and the payoff was huge. During preparations for the first deployments (for the trade show demo and a couple of early customers) the team wrote hundreds of lines of Tcl scripts to automate the setup and configuration of the different pieces of software that made up the network. If that power hadn’t been available to them, they would’ve had to make requests to the development team for added features in the system, which would’ve been a huge bottleneck. It was generally acknowledged that the use of Tcl for the configuration language was one of several decisions that helped the deployment teams meet the project deadlines. It also made the system more robust: the configuration scripts were originally intended just for starting the system, but they turned out to be very useful for dynamic error detection and recovery as well. Tcl has fallen out of favor (for reasons both good and bad) and today I’d probably use Ruby or Python. But that’s an implementation detail.

I’m a big believer in the YAGNI principle: if you don’t have a real, demonstrated need for some architectural or design feature today, assume that "you ain’t gonna need it" and keep your design simple and focused on the needs of the day. But YAGNI is a principle, not a law, and there are some places where I’ve learned that it doesn’t apply. This is one: if you know you need a scripting, configuration, or extension language—something that allows your system to be automated—you can safely assume that you need a complete programming language in that role. YIGNI: you is gonna need it.

I find dynamic languages such as Ruby and Python to be ideal for this type of commanded automation, though they certainly aren’t limited to this scope. Glenn’s story also reminds me how convenient it is that most of my GUI-based Mac applications can be automated using AppleScript. I wish more applications had this type of commanded automation interface.