Blog JVM Stuff.

JUnit Rules and Spock

I recently had to implement a file upload for a Grails application. In this application we have quite a bunch of JUnit tests but we do want to utilize Spock for all the newly added tests.

As it is in the nature of file uploads, the Spock specification needs to create quite a few temporary files and folders.

One option I've seen quite a few times is to either use File#createTemp or use JDK 7's Files#createTempDirectory/Files#createTempFile methods. Both have the disadvantage of the intial setup and cleanup, which results in helper code that is added to the test and distracts from the real test code.

JUnit Rules

JUnit provides a way to intercept the test suite and test method execution by providing the concept of rules. A rule implementation can intercept test method execution and alter the behaviour of these tests, or add cleanup work as it was done in @Before, @After, @BeforeClass and @AfterClass or Spock's setup, cleanup, setupSpec and cleanupSpec methods. For a particular test case, an instance of the rule implementation must be available via a public instance field and it must be annotated with @Rule.

For example, the TestName rule implementation can be used to have access to the test case name in a test method at runtime.

public class NameRuleTest {
  @Rule
  public TestName name = new TestName();

  @Test
  public void testA() {
    assertEquals("testA", name.getMethodName());
  }

  @Test
  public void testB() {
    assertEquals("testB", name.getMethodName());
  }
}

Spock and JUnit Rules

As it turns out, Spock has support for applying JUnit rules in specifications. This be done by providing a Groovy property with the type of the rule implementation, annotated by @Rule. This was really good news for my file upload tests as this allowed me to use one of my favorite JUnit rules: the org.junit.rules.TemporaryFolder rule.

As its name implies, the TemporaryFolder rule gives a convenient way to create temporary folders and files in test methods. The rule concept is used to intercept before and after each test method execution to do all the setup work.

This makes testing my AttachmentService very slick:

@TestFor(AttachmentService)
@Mock([Attachment])
class AttachmentServiceSpec extends Specification {

    @Rule
    TemporaryFolder temporaryFolder // see Peter's comment below :-) = new TemporaryFolder()

    def "load the persistent file for a given attachment"() {
        setup:
          def tempFile = temporaryFolder.newFile('test.txt')
          def attachment = new Attachment(
                               uploadId: '123', 
                               originalFilename: 'test.txt', 
                               location: tempFile.toURL()).save()

        when: "an attachment with a URL reference is loaded"
          def file = service.loadFile(attachment)

        then: "the underyling File must be returned"
          file == tempFile
    }
}

As you can see in the code above, the TemporaryFolder can be used to create a new file with the newFile method. If we wanted to create a new folder, there is also a newFolder method available. We do not have to specify any temporary folder or do any cleanup work, this is all done by the rule implementation itself.

There is a good overview for the base rules provided in JUnit at Github.

Conclusion

Spock comes with support for JUnit rules. A rule can intercept test method execution, do setup or cleanup work and might even change the test results. The TemporaryFolder rule is a useful rule that allows to create temporary files and folders in test cases while keeping track of these files and cleaning them up after the test execution.

Vagrant - Configuring a Solr Box

Recently I started some experiments with Vagrant. Vagrant is a tool that lets you (pre-) configure development environments. This can be done based on VirtualBox, Hyper-V, VMWare or many more so-called providers. You see, another way to put it is that Vagrant is a tool that helps with configuring, building and running virtual machine instances.

The default provider is VirtualBox but the documentation actually recommends to use VMWare for more stable and performant environments.

Welcome to Vagrant

Vagrant can be installed via a manual download or your preferred package manager, such as brew on Mac OS.

We needed to write (I guess they're called) system tests, to test integration scenarios between an Apache Solr server instance and a library. As you can imagine this implies that an Apache Solr instance needs to be installed on every developer machine that intends to execute the entire test suite.

Now Vagrant comes into play. It's a command-line tool that is used to setup a virtual machine with all components pre-installed.

However, instead of configuring virtual machine images from scratch, Vagrant introduces the concept of boxes. Boxes are the package format that is used to bring up identical development environments. The easiest way to use a box is to choose one from the publicy available ones.

The "precise64" Box

For our purpose, we based our Vagrant box on the "precise64" box. It contains an Ubuntu 12.04 LTS (precise) installation with some tools pre-installed. The default box is specified in a file called Vagrantfile.

Running vagrant init will create a template Vagrantfile in the current directory.

Vagrantfile is a file written in a Ruby DSL and it contains the configuration of our custom box:

Vagrant.configure("2") do |config|
  # Defines the Vagrant box name, download URL, IP and hostname
  config.vm.define :vagrant do |vagrant|
    vagrant.vm.box = "precise64"
    vagrant.vm.box_url = "http://files.vagrantup.com/precise64.box"

    vagrant.vm.network :private_network, ip: "192.168.66.6"
    vagrant.vm.network "forwarded_port", guest: 8983, host: 8898

    vagrant.vm.hostname = "vagrant.dcl"
  end
end

The configuration specifies the pre-defined box precise64 as "parent" vm box. In addition, it specifies the URL under which this box can be downloaded.

More Vagrantfile

The Vagrantfile may consist of various sections. For a detailed overview of all available configuration options, please have a look at the Vagrant documentation.

Next up in our configuration file is the network configuration. We use a private network, this means we can access our guest from the host machine but the box won't be visible from the outside. It gets an IP address in the private address space. Plus, we define a forwarded port. In fact, this is the port under which Apache Solr listens in the default configuration. Once we access localhost:8983 on the host machine, the request will be forwarded to the Vagrant virtual machine instance port 8983.

Would we start with vagrant up we would have a running Ubuntu 12.04 LTS instance within seconds. Unfortunately, Ubuntu 12.04 doesn't come with Solr pre-installed, so there's some work left for us.

As the pre-defined "precise64" box doesn't fix our use case, we need to alter the environment a bit. We need to install Java and Apache Solr in our custom box. The process of adding/tweaking stuff in a box is called provisioning. The simplest way of implementing provisioning is to write shell scripts. An advanced way would be to use tools such as Chef or Puppet.

We decided to use plain shell scripts and added a shell-script called provision.sh to our Vagrantfile configuration:

Vagrant.configure("2") do |config|

  config.vm.provision :shell, :inline => "
    sh /vagrant/scripts/provision.sh;
  "

  # Defines the Vagrant box name, download URL, IP and hostname
  config.vm.define :vagrant do |vagrant|
    vagrant.vm.box = "precise64"
    vagrant.vm.box_url = "http://files.vagrantup.com/precise64.box"

    vagrant.vm.network :private_network, ip: "192.168.66.6"
    vagrant.vm.network "forwarded_port", guest: 8983, host: 8983

    vagrant.vm.hostname = "vagrant.dcl"
  end
end

The provision.sh shell-script defines all the commands to install Java and run the Solr instance:

#! /bin/bash

##### VARIABLES #####

# Throughout this script, some variables are used, these are defined first.
# These variables can be altered to fit your specific needs or preferences.

# Server name
HOSTNAME="vagrant.dcl"

# Locale
LOCALE_LANGUAGE="en_US" # can be altered to your prefered locale, see http://docs.moodle.org/dev/Table_of_locales
LOCALE_CODESET="en_US.UTF-8"

# Timezone
TIMEZONE="Europe/Paris" # can be altered to your specific timezone, see http://manpages.ubuntu.com/manpages/jaunty/man3/DateTime::TimeZone::Catalog.3pm.html

VM_ID_ADDRESS="192.168.66.6"

#----- end of configurable variables -----#


##### PROVISION CHECK ######

# The provision check is intended to not run the full provision script when a box has already been provisioned.
# At the end of this script, a file is created on the vagrant box, we'll check if it exists now.
echo "[vagrant provisioning] Checking if the box was already provisioned..."

if [ -e "/home/vagrant/.provision_check" ]
then
  # Skipping provisioning if the box is already provisioned
  echo "[vagrant provisioning] The box is already provisioned..."
  exit
fi


##### PROVISION SOLR #####
echo "[vagrant provisioning] Updating mirrors in sources.list"

# prepend "mirror" entries to sources.list to let apt-get use the most performant mirror
sudo sed -i -e '1ideb mirror://mirrors.ubuntu.com/mirrors.txt precise main restricted universe multiverse\ndeb mirror://mirrors.ubuntu.com/mirrors.txt precise-updates main restricted universe multiverse\ndeb mirror://mirrors.ubuntu.com/mirrors.txt precise-backports main restricted universe multiverse\ndeb mirror://mirrors.ubuntu.com/mirrors.txt precise-security main restricted universe multiverse\n' /etc/apt/sources.list
sudo apt-get update

echo "[vagrant provisioning] Installing Java..."
sudo apt-get -y install curl
sudo apt-get -y install python-software-properties # adds add-apt-repository

sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update

# automatic install of the Oracle JDK 7
echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections

sudo apt-get -y install oracle-java7-set-default

export JAVA_HOME="/usr/lib/jvm/java-7-oracle/jre"

echo "[vagrant provisioning] Installing Apache Solr..."

sudo apt-get -y install unzip

curl -O http://tweedo.com/mirror/apache/lucene/solr/4.7.1/solr-4.7.1.zip
unzip solr-4.7.1.zip

rm solr-4.7.1.zip
cd solr-4.7.1/example/

# Solr startup
java -jar start.jar > /tmp/solr-server-log.txt &

echo "[vagrant provisioning] Bootstrapping Apache Solr..."

sleep 1
while ! grep -m1 'Registered new searcher' < /tmp/solr-server-log.txt; do
    sleep 1
done

echo "[vagrant provisioning] Apache Solr started. Index test data ..."

# Index some stuff
cd exampledocs/
java -jar post.jar solr.xml monitor.xml

##### CONFIGURATION #####

# Hostname
echo "[vagrant provisioning] Setting hostname..."
sudo hostname $HOSTNAME

##### CLEAN UP #####

sudo dpkg --configure -a # when upgrade or install doesn't run well (e.g. loss of connection) this may resolve quite a few issues
apt-get autoremove -y # remove obsolete packages

##### PROVISION CHECK #####

# Create .provision_check for the script to check on during a next vargant up.
echo "[vagrant provisioning] Creating .provision_check file..."
touch .provision_check

The provisioning process starts with the definition of some variables and the check for the .provision_check file. This file will be touched once the provisioning process has gone through successfully. All files within the directory containing the Vagrantfile will be available in the guest at /vagrant. The current home directory all file operations will be operated in during provisioning is /home/vagrant. Vagrant allows to configure more of these so-called synced folders but for our purposes the defaults were perfectly fine.

After the file check, sources.list will be updated and the following lines will be added at the beginning of the file:

sudo sed -i -e '1ideb mirror://mirrors.ubuntu.com/mirrors.txt precise main restricted universe multiverse\ndeb mirror://mirrors.ubuntu.com/mirrors.txt precise-updates main restricted universe multiverse\ndeb mirror://mirrors.ubuntu.com/mirrors.txt precise-backports main restricted universe multiverse\ndeb mirror://mirrors.ubuntu.com/mirrors.txt precise-security main restricted universe multiverse\n' /etc/apt/sources.list
sudo apt-get update

The deb mirror:... entries are used to optimize the selected apt-get mirror. This will result in the selection of the fastest available mirror without any hard-coded local preliminaries.

The next lines are pretty straight-forward. The ppa:webupd8team/java repository is used to fetch Oracle JDK 7u51. As the JDK installer comes with a modal confirmation dialog, the following lines are needed:

echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections

This quietly confirms and installs the JDK automatically.

Next up is Apache Solr installation. The Solr 4.7.1 zip is downloaded and gets extracted in the home directory. Afterwards, the example configuration is run with java -jar start.jar. This starts a Jetty instance. The script uses the following code to monitor when bootstrapping is done:

sleep 1
while ! grep -m1 'Registered new searcher' < /tmp/solr-server-log.txt; do
    sleep 1
done

Once the message "Registered new searcher" appears we can safely assume the Solr instance is started.

Solr's post.jar can be used to add documents to the index. In this script we simply add the files from the exampledocs directory.

Conclusion

And that's it.

With this configuration the development environment can be started via vagrant up. Once the machine shall be stopped, we can use vagrant halt and vagrant resume to resume were we stopped.

What's really cool about it is that once we commit our Vagrantfile and provision.sh to our git repository every developer is free to check it out and run the vagrant command line tool with it. It results in exactly the same development virtual machine that can now be used to run our systems tests.

Grails - Java-Based Configuration

Spring 3.0 introduced an interesting alternative to XML file-based Spring container configuration. After Spring 2.5 already introduced the annotation-based configuration, it added a new way to configure the Spring bean container: via java-based configuration (the SpringConfig project was taken included in the Spring framework).

Java-Based Configuration

As its name implies, Java-based Spring container configuration is about using Java code instead of XML or annotations to configure Spring beans. This way reminds of Google Guice, a dependency injection framework solely using Java code to setup the dependency injection components. In Spring, the org.springframework.context.annotation package holds the following annotations that can be used as an alternative to XML files to annotate certain classes as Spring context configuration classes:

  • @Configuration

    The @Configuration class indicates that a class is defining multiple methods creating Spring-managed beans. The @Configuration class needs to be found by the annotation processing configuration. We will have a look how that can be configured in a Grails application.

  • @Bean

    The @Bean annotation is mainly applied on methods (although it can be applied on annotation types too). It indicates that a method returns a bean manager by the Spring application container. The default strategy for the bean name is to use the name of the bean generating method.

  • @ComponentScan

    The @ComponentScan annotation can be applied on @Configuation classes to configure component scanning for certain Java packages or classes.

  • @DependsOn

    The @DependsOn annotation can be used to indicate that a bean depends on the creation of other beans. It defines a happens-before relationship in terms of the container-managed order of bean creations.

  • @Import

    The @Import annotation can be used to import @Configuration classes into the annotated class. It resembles the <beans:import/> XML tag.

  • @ImportResource

    The @ImportResource annotation can be used to import bean definitions into the current @Configuration class.

  • @Lazy

    The @Lazy anntotation indicates a bean has to be created by the Spring container in a lazy manner.

  • @Profile

    The @Profile annotation can be used to annotate @Configuration classes for certain profiles. The current profile can be set by using a VM argument or programatically by calling the ConfigurableEnvironment.setActiveProfiles(String ...) method.

  • @PropertySource

    The @PropertySource adds property files to the current Spring environment.

  • @Scope

    The @Scope annotation indicates the bean scope the bean is created in.

Now let's have a look at how Java-based Spring configuration can be used in a Grails application.

Enabling Java-Based Configs

Let's assume we have a package my.app.config that holds all configuration classes. In a Grails application we need to add this packacke in Config.groovy to the grails.spring.bean.packages bean scanning path list.

grails.spring.bean.packages = ['my.app.config']

VoilĂ , now we can use the annotations described above inside the my.app.config package. The classes can either be implemented as Groovy or Java classes.

The Java-based Spring configuration can even be used in a mixed-mode: there can still be a resources.groovy or a resources.xml or more Spring configuration files in the grails-app/conf/spring directory, the Spring bean container will merge and resolve the bean definitions from the various sources.

Configuration Example

The Spring Reactor project comes with a pretty neat AsyncTaskExecutor implementation that uses the LMAX Disruptor library under the hoods. If we wanted to use the reactor.spring.core.task.WorkQueueAsyncTaskExecutor from Reactors Spring module, we can do so by defining a Java-based bean configuration.

In the case of the Reactor project, we're already provided with a pre-defined configuration annotation from the Reactor project that we have to use additionally on our @Configuration class: the @EnableReactor annotation.

The @EnableRector annotation uses the @Import annotation to import a default configuration into our @Configuration class:

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Import(ReactorBeanDefinitionRegistrar.class)
public @interface EnableReactor {
    // ...
}

The ReactorBeanDefinitionRegistrar uses another cool feature of the Java-based Spring configuration. It implements the o.s.context.annotation.ImportBeanDefinitionRegistrar interface to register beans in the current application context if not already defined by the user configuration.

public class ReactorBeanDefinitionRegistrar implements ImportBeanDefinitionRegistrar {

  private static final String DEFAULT_ENV_NAME = "reactorEnv";

  @Override
  public void registerBeanDefinitions(AnnotationMetadata meta, BeanDefinitionRegistry registry) {
    Map<String, Object> attrs = meta.getAnnotationAttributes(EnableReactor.class.getName());

    // Create a root Enivronment
    if (!registry.containsBeanDefinition(DEFAULT_ENV_NAME)) {
      ...
      registry.registerBeanDefinition(DEFAULT_ENV_NAME, envBeanDef.getBeanDefinition());
    }
  }
}

The ImportBeanDefinitionRegistrar implementation can be used along @Configuration classes as annotation parameters for the @Import annotation in our configuration class:

@Configuration
@EnableReactor
class ReactorConfiguration {

}

To add a WorkQueueAsyncTaskExecutor bean to this configuration we simply have to provide a @Bean annotated method:

@Configuration
@EnableReactor
class ReactorConfiguration {

  @Bean
  AsyncTaskExecutor taskExecutor(Environment env) {
    return new WorkQueueAsyncTaskExecutor(env)
      .setName("workQueueAsyncTaskExecutor")
      .setBacklog(2048)
      .setProducerType(ProducerType.MULTI)
      .setWaitStrategy(new SleepingWaitStrategy()) 
  }
}

Since the method is named taskExecutor our bean has the same name. And of course, we could use arbitrary Java code to initialize our bean. Once defined, we can refer to taskExecutor in every configuration artefact of our application, let it be resources.groovy, resources.xml or any other configuration files.

Please don't confuse the Environment parameter with the org.springframework.core.env.Environment interface, the parameter is of type reactor.core.Environment and is injected automatically, after being created by the @EnableRector annotation.

If we have a look at our code example above, we could decided to move all the builder method arguments to our app.properties file. We can use the @PropertySource configuration annotation to include this file in our current configuration:

@Configuration
@PropertySource('classpath:/my/app/config/app.properties')
@EnableReactor
class ReactorConfiguration {

  @Autowired
  Environment environment

  @Bean
  AsyncTaskExecutor taskExecutor(reactor.core.Environment env) {
    return new WorkQueueAsyncTaskExecutor(env)
      .setName(environment.getProperty("queue.name"))
      .setBacklog(environment.getProperty("queue.backlog", Integer.class))
      .setProducerType(ProducerType.MULTI)
      .setWaitStrategy(new SleepingWaitStrategy()) 
  }
}

As can be seen above, the queue.name and queue.backlog settings in the app.properties file can be accessed via the org.springframework.core.env.Environment bean. This bean needs to be autowired in our configuration to access the message resources. This shows another impressive feature of Spring's Java-based configuration: @Configuration classes are themselves treated as beans. This means we can use for example @Autowired to inject beans into our @Configuration class. The Spring documentation shows an even better example:

@Configuration
public class ServiceConfig {

  @Autowired
  private AccountRepository accountRepository;

  @Bean
  public TransferService transferService() {
    return new TransferServiceImpl(accountRepository);
  }
}

@Configuration
public class RepositoryConfig {

  @Autowired
  private DataSource dataSource;

  @Bean
  public AccountRepository accountRepository() {
    return new JdbcAccountRepository(dataSource);
  }
}

@Configuration
@Import({ServiceConfig.class, RepositoryConfig.class})
public class SystemTestConfig {

  @Bean
  public DataSource dataSource() {
    // return new DataSource
  }
}

public static void main(String[] args) {
  ApplicationContext ctx = new AnnotationConfigApplicationContext(SystemTestConfig.class);
  // everything wires up across configuration classes...
  TransferService transferService = ctx.getBean(TransferService.class);
  transferService.transfer(100.00, "A123", "C456");
}

We can even auto-wire @Configuration classes themselves:

@Configuration
public class ServiceConfig {

  @Autowired
  private RepositoryConfig repositoryConfig;

  @Bean
  public TransferService transferService() {
    // navigate through the config class to the @Bean method!
    return new TransferServiceImpl(repositoryConfig.accountRepository());
  }
}

These examples show that the Java-based configuration approach can provide us with much more modularity and reusability than using (the already mighty) Spring bean Groovy DSL which is, by the way, part of Spring 4. As every Grails application is a Spring application, we can use this approach to better structure application provided beans.

Conclusion

Spring 3 introduced another configuration approach besides the XML configuration files and annotation-based configuration: the so-called Java-based configuration. Spring provides a buch of annotations enabling to define beans in pure Java/Groovy code with even better modularity and reusability as can be found in Grails Spring DSL files. This article is an introduction to Java-based configuration and shows how to enable the mechanism in Grails applications.

Grails - Environments in BuildConfig

As of Grails 2.3 it is possible to use the Environment class everywhere in BuildConfig.groovy. This can be useful for environment-specific settings regarding all the configuration that can be done in BuildConfig.groovy including dependencies and plugins.

What's the deal?

Recently I came across an issue with the UI performance plugin (like mentioned by someone on the Grails user list) where we needed to exclude dependencies to be only used in the test environment. Before Grails 2.3.x it was only possible to use the Environment class inside the dependencies or plugins closure:

dependencies {

    // ...

    if (Environment.current != Environment.PRODUCTION)  {
        test "org.gebish:geb-junit4:0.9.2"
        test "org.seleniumhq.selenium:selenium-support:2.39.0"
        test "org.seleniumhq.selenium:selenium-firefox-driver:2.39.0"
        test "org.seleniumhq.selenium:selenium-chrome-driver:2.39.0"
    }
}

plugins {

    // ...

    test ":spock:0.7"
    test(":cucumber:0.10.0") {
        exclude "spock-core"
    }

    test ":code-coverage:1.2.7"
    test ":geb:0.9.2"
}

This little trick was possible because the closure was evaluated at a point of time where the Environment already had been initialized. Something along the lines of

grails.server.port.http = 8080

Environment.executeForEnvironment(Environment.DEVELOPMENT)  {
    grails.server.port.http = 8082  
}

wouldn't have been successful caused by the Environment class not being initialized at runtime.

This behaviour has been fixed in Grails 2.3, the BuildConfig DSL has even been extended to support the environments block feature. We can use it to specify environment-specific dependencies blocks or other BuildConfig configuration settings:

environments {
    test {
        dependencies {
            test "org.gebish:geb-junit4:0.9.2"
            test "org.seleniumhq.selenium:selenium-support:2.39.0"
            test "org.seleniumhq.selenium:selenium-firefox-driver:2.39.0"
            test "org.seleniumhq.selenium:selenium-chrome-driver:2.39.0"
        }
    }
}

Conclusion

Before 2.3.x the Environment could only be used restrictively in BuildConfig.groovy. With 2.3.x this issue has been fixed and support for the environments block has been added to the BuildConfig DSL.

Grails Quick Tip - Cleaning Up

Mr. Haki, once again, wrote a really cool blog post about grails clean and how it can be configured to delete all or only certain generated files in the working directory.

When reading the blog post a trick came to my mind that was once mentioned by Burt Beckwith in his excellent book Programming Grails.

Burt said he often replaces the standard configuration in BuildConfig.groovy

grails.project.class.dir = "target/classes"
grails.project.test.class.dir = "target/test-classes"
grails.project.test.reports.dir = "target/test-reports"

with

grails.project.work.dir = "target"

This has the advantage that all generated classes, including plugin-classes, are saved in a single directory. This is really cool as installed plugins would otherwise be saved and cached in the ~/.grails/ directory.

Grails Work Dir Target

Once things mess up and a clean state is needed, a simple rm -rf target gives us a completely zero state in terms of generated artefacts.