Blog JVM Stuff.

Java Codecache

When a Java program is run, it executes the code in a tiered manner. In the first tier, it uses the so-called client compiler (C1 compiler mode) in order to compile the code with instrumentation. The profiling data is used in the second tier (C2 compiler mode) for the server compiler, to compile that code in an optimised and high-performant way.

Tiered compilation is not enabled per default in Java 7 (however, it might be enabled by the application server that your are running), but is enabled in Java 8.

The JIT-compiler stores the compiled code in an area called code-cache. This area is flushed if its size exceeds a certain treshold. The memory size and certain other parameters like the flush treshold can be set via JVM parameters.

When the codecache fills up

In one of our projects, we were running into a seemingly weird problem that actually could not be solved by the programmers for quite some time. It would show in a way that the JEE web application running on Apache Tomcat would slow down after some days. The slowdown was caused by an increased load being visible in the monitoring tools. However, there was no hint at performance problems neither on the database layer (in MySQL slow query logs) or on the application layer. It was just that the application got slower and slower over time.

It was only with jconsole, a tool that comes bundled with the JDK,
an issue got visibile. In the memory section of this tool we saw an increased use of the "Code Cache" section. The application was running on Sun Oracle 1.7 update 79 and therefore had a maximum code cache size of 48 MB (the default for the 64-bit version). As it turned out after monitoring the code cache with jconsole, once the code cache was filled after a couple of days, the performance degration began.

It was this blog article in the Oracle knowledge base that pointed out something interesting: starting from 1.7 Update 4 until the end of the Java 7 branch, there are a couple of known bugs that have to do with filled up code caches. The most important problems in Java 7 when the code cache fills up are:

  • The compiler may not get restarted even after the CodeCache occupancy drops down to almost half after the emergency flushing.
  • The emergency flushing may cause high CPU usage by the compiler threads leading to overall performance degradation.

There are two suggestions on how to fix this issue for Java 7:

  • turn off the code-cache flushing entirely
  • increase the size of the code cache up to a point never being reached

We decided to go with the 2nd option. We increased the code cache and kept watching with jconsole. As it turned out, this led to a much better overall load of the system, even after two weeks, the issue previously seen did not happen anymore.

So the outcome is that, when running still with Java 7, you should have a look at the size of the projects code-cache in order to avoid these problems or to run into other bugs that have to do with flushing the code-cache.

Grails, GPars and Hibernate

The GPars library is a great way to add concurrency supporting constructs to your Groovy code. It equips Groovy projects with powerful concurrency concepts like parallel collections, map reduce operations, actors and dataflow variables.

Adding the GPars dependency

GPars can also be added to Grails applications (in our case, we have a Grails 2.2.5 application, so this article was not tested with any versions up or below). Simply add GPars as a dependency in the BuildConfig.groovy:


dependecies {
  compile 'org.codehaus.gpars:gpars:some_version_number' // we use 1.1.0 for our code
}

That is actually enough to have the GPars features enabled, once included, you can access the parallel collection methods in the targeted collection classes, like:


GParsPool.withPool {
  [1, 2, 3, 4].findParallel { it == 3 }
}

You can see in the above example that the GPars convention is to use the original method name with a Parallel appended. The GParsPool.withPool method has to be called in order to initialise the GPars thread pool and equip collection classes with the parallel methods. An overview of all the available methods can be found in GParsExecutorsPoolEnhancer.

GPars and Hibernate

Our requirement was to speed up a Quartz job iterating over all our customers - which are all Hibernate entities. As the customers could be easily grouped into different sets, the precondition for jork-join processing were met. But the question was how to enable Hibernate read-only processing in the eachParallel GPars method as we wanted to do something like:


differentCustomerGroups.eachParallel { CustomerGroup customerGroup ->
  // process all customers of the given customer group
  // IN THE CURRENT HIBERNATE SESSION
}

As every Hibernate session is bound to the current thread, it was necessary to create a new Session for the current thread created by GPars, attach it and close it once processing was done.

The key to enabling this (in Grails 2.2.5 at least), was to use the persistenceInterceptor bean that can be injected into any Grails artefact:


def persistenceInterceptor

It implements the PersistenceContextInterceptor interface which can be used to initialise and destroy the current persistence context. In the case of its Hibernate implementation, HibernatePersistenceContextInterceptor, the persistence context is the current Hibernate session. Thus, the persistence context interceptor bean can be used to initialise and destroy the session in our closure. The eachParallel uses the bean like that (this pattern can also be found in other places in Grails btw):


differentCustomerGroups.eachParallel { CustomerGroup customerGroup ->
  // init the persistence context
  persistenceInterceptor.init()

  try {
    int offset = 0
    def customers = Customer.executeQuery("select c from customer c where c.customerGroup = ?", customerGroup, [readOnly: true, max: 100, offset: offset])

    // loop over customers till all are processed ...

    // flush the context
    persistenceInterceptor.flush()
  } finally {
    // destroy the context and release resources
        persistenceInterceptor.destroy()
  }
}

This is effectively enough to create and destroy a new Hibernate session to be used by the GPars code. Note, that we also had to disable the automatic Hibernate session creation done by Quartz, specifying the def sessionRequired = false property in the job class:


class CustomerJob {
  def concurrent = false
  def sessionRequired = false // do not create a Hibernate session on job startup

  def execute() {
    // ...
  }
}

Another thing to note is maybe the readOnly option that was given to the executeQuery method. It disables snap-shotting of entities, which in our case was possible (since the customer instances were not modified themselves).

Conclusion

GPars is a library that adds concurrency constructs to your class and also provides concurrency concepts like actors, agents and dataflow variables. This article shows how to handle the Hibernate session in code that is concurrently executed by GPars. The persistence context interceptor is a class provided by Grails that enables setting up and destroying the persistence context in arbitrary places. Note that this article is based on Grails 2.2.5.

Grails: Reconnecting JDBC Connections

At our company we are utilising the well-known Quartz library and Grails plugin for our batch jobs in Grails applications. When going throgh the server log files of our production server I lately came acress this error:


org.apache.tomcat.jdbc.pool.ConnectionPool abandon
WARNING: Connection has been abandoned PooledConnection[com.mysql.jdbc.JDBC4Connection@650599cb]:java.lang.Exception
        at org.apache.tomcat.jdbc.pool.ConnectionPool.getThreadDump(ConnectionPool.java:967)
        at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:721)
        at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:579)
        at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:174)
        at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:111)
        at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
        at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:224)
        ...
        at grails.plugins.quartz.GrailsJobFactory$GrailsJob.execute(GrailsJobFactory.java:104)
        at org.quartz.Job$execute.call(Unknown Source)
        at grails.plugins.quartz.QuartzDisplayJob.execute(QuartzDisplayJob.groovy:29)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:207)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:560)

The beginning of the stack-trace showed that the exception was thrown by a Quartz worker thread - so how come the current DB connection was abandoned?

I investigated the connection pool settings and I found the following in the Tomcat connection pool configuration:
removeAbandoned="true"
removeAbandonedTimeout="3600"

The abandoned timeout was set to 3600 seconds aka as one hour. The batch job took over 1 hour, therefore the connection pool abandoned the connection which led to the error above.

A Session in a Quartz Job

As long as the property sessionRequired is not explicitly set to false in a Grails Quartz job class, the Quartz plugin will create a Hibernate session that is bound to the Quartz worker thread. This is done by the SessionBinderJobListener that comes with the Quartz plugin that uses the current persistence context interceptor of type HibernatePersistenceContextInterceptor, it is only enabled when Hibernate is in use as general data store.

For long-running quartz jobs the binding to a Hibernate session is bad, as this means that the Hibernate session binds a JDBC connection from the connection pool for the live-time of the session (starting once the session has to use the connection to the database).

When skipping through the documentation for org.hibernate.Session the disconnect(Connection) method appeared to me. As the documentation mentioned:

Disconnect the Session from the current JDBC connection. If the connection was obtained by Hibernate close it and return it to the connection pool; otherwise, return it to the application. This is used by applications which supply JDBC connections to Hibernate and which require long-sessions (or long-conversations)


And it adds:

Note that disconnect() called on a session where the connection was retrieved by Hibernate through its configured org.hibernate.connection.ConnectionProvider has no effect, provided ConnectionReleaseMode.ON_CLOSE is not in effect.


So this means that the DB connection used by the session has to be given by the application-code in order to use disconnection. In our application, this was the case, so we could go into that direction to solve this issue.

Refreshing a Session

So the goal was to rewrite the long-running batch jobs so that they disconnect and close the JDBC connection from time to time. This would release the connection and put it back to the pool, a connection abandonment as we have seen before is not possible anymore as long as the time span for the disconnect is of course smaller than an hour.

First thing we did in the Quartz job was to disable the automatic Hibernate session creation via the sessionRequired property:


class SomeJob {

  static triggers = {
      // ...
  }

  def concurrent = false
  def sessionRequired = false

  GrailsApplication grailsApplication
  SessionFactory sessionFactory
  DataSource dataSource

  // ...
}

In order to get a connection from the DB pool we injected the javax.sql.DataSource and rewrote the code so that it would use the created Hibernate session (instead of the Grails provided ones):


def session = sessionFactory.openSession(dataSource.connection)

All the HQL queries and Hibernate operations would be rewritten to be running over this particular session.

Keeping the session open for the entire life-time of the job implies flushing and clearing the session from time to time to avoid running in to performance or out-of-memory issues. As suggested in the Hibernate Documentation, the session is flushed after a certain nubmer of entities processed:


protected void refreshJDBCConnection(Session session) {
    session.flush()
    session.clear()

    // ...
  }

Exactly at this pace in the code, we added the code that would take the current connection, close it, and return it to the database connection pool:


protected void refreshJDBCConnection(Session session) {
    session.flush()
    session.clear()

    // we need to disconnect and get a new DB connection here
    def connection = session.disconnect()
    connection.close()
    
    session.reconnect(dataSource.connection)
  }

The code above will get the JDBC connection out of the session and uses the dataSource to obtain a new database connection and reconnect it with the existing session. With this we can keep the abandoned connection setting and ensure that the job is not aborted after the configured aboandonment time.

Conclusion

Connection pools allow to abandon database connections. After a certain time, the DB connection can be forced to be put back into the connection pool, no matter if it is still in use by the application or not. With Grails Quartz jobs there can be an issue with this feature in combination with long-running jobs that exceed the configured abandonment time. This article showed how to disconnect and reconnect a Hibernate session that can be kept open during the life-time of such long-running jobs.

Spock Quick-Tip: Grails Integration Tests

I'm currently working in project with a nearly six year old Grails application. Over time, we changed from having plain JUnit 4/GroovyTestCase unit and integration tests to Spock tests. Spock is such a great library to write both, unit and integration tests. Normally, we tend to write unit tests whenever possible. But there are cases (and code parts) where it is more practical to rely on a fully initialized application context and (Hibernate) data-store.

Integration Tests with Spock

The Spock Grails plugin comes with a distinct base class for Grails integration tests: IntegrationSpec.

IntegrationSpec initialises the application context, sets up an autowirer that autowires bean properties from the specification class and create a transactional boundary around every feature method in the specification.

All our Spock integration tests extend IntegrationSpec.

Mocking with Groovy's meta-classes

One thing I love about Spock is that it comes out-of-the-box with great support for mocking and stubbing. But there are times when you actually need to stub certain parts of the Grails artifact that is currently under test by your integration test.

We do this with the help of the Groovy Meta-Object protocol (MOP), that is, by altering the underlying meta-class. The next example shows how getCurrentUser is overwritten, as we do want to stub out the Spring Security part from the StatisticsService.


StatisticsServiceIntegrationTest extends IntegrationSpec {
    
    StatisticsService statisticsService

    void "count login to private area"() {

        setup:
            def user = new User(statistics: new UserStatistics())
            statisticsService.metaClass.getCurrentUser = { -> user }

        when:
            statisticsService.countLoginPA()

        then:
            user.statistics.logins == 1

    }    
}

Altering classes at runtime is a nice feature, but it can also become confusing when you don't know about the side-effects it may cause. For integration tests, changes to the meta-class won't be resetted, so once you do changes to a meta-class (we are working with per-instance meta-class changes, the same is even more true for global meta-class changes) those will be persistent through the entire test.

To solve that, we added a helper method that allows to revoke meta-class changes inbetween test runs:


public static void revokeMetaClassChanges(Class type, def instance = null)  {
    GroovySystem.metaClassRegistry.removeMetaClass(type)
    if (instance != null)  {
        instance.metaClass = null
    }
}

And applied it like this:


StatisticsServiceIntegrationTest extends IntegrationSpec {
    
    StatisticsService statisticsService

    void "count login to private area"() {

        setup:
            def user = new User(statistics: new UserStatistics())
            statisticsService.metaClass.getCurrentUser = { -> user }

        when:
            statisticsService.countLoginPA()

        then:
            user.statistics.logins == 1

        cleanup:
            revokeMetaClassChanges(StatisticsService, statisticsService)

    }    
}

This actually sets back the meta-class code and the service class is again un-altered when executing the next feature method.

Be warned.

Meta-class overriding can become tricky. One thing we came across multiple times is that you can't replace methods of super classes being called from super class methods. Here is a simplified example:


class A {
    def a(){
       a2()

    }
    def a2(){   
         println 'In class A'
    }
}

class B extends A{
    def b(){
        a()
    }
}

B b = new B();

b.metaClass.a2 = {
    println 'In class B'
}

b.b(); // still prints 'In class A'

If we wanted to stub the implementation of b inside our test code, this wouldn't work, as a and a2 are implemented in the same class A and therefore the method call won't be intercepted by a per-instance change to instance b. This now might seem obvious, but we had a hard time tracking this down.

If you start to experience weird issues of tests failing when you run the entire test suite, but being green when executed separately, it almost certainly has to do with meta-class rewriting that isn't undone between feature methods or even specifications. Just be aware of that.

@ConfineMetaClassChanges

Lately I became aware that our revokeMetaClassChanges is actually "part" of Spock with the @ConfineMetaClassChanges extension annotation.

The code behind it works a bit differently but the meaning is the same; it can be used on methods or classes to rollback meta-class changes declaratively:


@ConfineMetaClassChanges([StatisticsService])
StatisticsServiceIntegrationTest extends IntegrationSpec {
    
    StatisticsService statisticsService

    void "count login to private area"() {

        setup:
            def user = new User(statistics: new UserStatistics())
            statisticsService.metaClass.getCurrentUser = { -> user }

        when:
            statisticsService.countLoginPA()

        then:
            user.statistics.logins == 1

    }    
}

Speaking of Spock extensions. It's definitely worth to have a look at the chapter on Spock Extensions in the documentation. There is lots of great stuff already available (and coming in Spock 1.0).

Conclusion

Besides Spock's great mocking and stubbing capabilities, writing Grails integration tests also involves meta-class changes. This article shows how to rollback these changes to avoid side-effects and explained the usage of @ConfineMetaClassChanges a Spock extension annotation.

Grails - Tracking Principals

We use the Grails auto timestamp feature in nearly all of our domain classes. It basically allows the definition of two special domain class properties dateCreated and lastUpdated and automatically sets the creation and modification date whenever a domain object is inserted or updated.

In addition to dateCreated and lastUpdated we wanted to have a way to define two additional properties userCreated and userUpdated to save the principal who created, updated or deleted a domain class (deletion because we have audit log tables that track all table changes, so when an entry is deleted and the principal is set before, we can see who deleted an entry).

PersistenceEventListener

Grails provides the concept of GORM events, so we thought its implementation might be a good hint on how to implement our requirement for having userCreated and userUpdated. And indeed, we found DomainEventListener, a descendant class of AbstractPersistenceEventListener. It turns out that DomainEventListener is responsible for executing the GORM event hooks on domain object inserts, updates and deletes.

The event listener is registered at the application context as the PersistenceListener interface (which is implemented by AbstractPersistenceListener) extends from Spring's ApplicationListener and therefore actually uses the Spring event system.

In order to create a custom persistence listener, we just have to extend AbstractPersistenceEventListener and listen for the GORM events which are useful to us. Here is the implementation we ended up with:


@Log4j
class PrincipalPersistenceListener extends AbstractPersistenceEventListener {

    public static final String PROPERTY_PRINCIPAL_UPDATED = 'userUpdated'
    public static final String PROPERTY_PRINCIPAL_CREATED = 'userCreated'

    SpringSecurityService springSecurityService

    PrincipalPersistenceListener(Datastore datastore) {
        super(datastore)
    }

    @Override
    protected void onPersistenceEvent(AbstractPersistenceEvent event) {

        def entityObject = event.entityObject

        if (hasPrincipalProperty(entityObject)) {
            switch (event.eventType) {
                case EventType.PreInsert:
                    setPrincipalProperties(entityObject, true)
                    break

                case EventType.Validation:
                    setPrincipalProperties(entityObject, entityObject.id == null)
                    break

                case EventType.PreUpdate:
                    setPrincipalProperties(entityObject, false)
                    break

                case EventType.PreDelete:
                    setPrincipalProperties(entityObject, false)
                    break
            }
        }
    }

    protected boolean hasPrincipalProperty(def entityObject) {
        return entityObject.metaClass.hasProperty(entityObject, PROPERTY_PRINCIPAL_UPDATED) || entityObject.metaClass.hasProperty(entityObject, PROPERTY_PRINCIPAL_CREATED)
    }

    protected void setPrincipalProperties(def entityObject, boolean insert)  {
        def currentUser = springSecurityService.currentUser

        if (currentUser instanceof User) {
            def propertyUpdated = entityObject.metaClass.getMetaProperty(PROPERTY_PRINCIPAL_UPDATED)
            if (propertyUpdated != null)  {
                propertyUpdated.setProperty(entityObject, currentUser.uuid)
            }

            if (insert)  {
                def propertyCreated = entityObject.metaClass.getMetaProperty(PROPERTY_PRINCIPAL_CREATED)
                if (propertyCreated != null)  {
                    propertyCreated.setProperty(entityObject, currentUser.uuid)
                }
            }
        }
    }

    @Override
    boolean supportsEventType(Class eventType) {
        return eventType.isAssignableFrom(PreInsertEvent) ||
                eventType.isAssignableFrom(PreUpdateEvent) ||
                eventType.isAssignableFrom(PreDeleteEvent) ||
                eventType.isAssignableFrom(ValidationEvent)
    }
}

As you can see in the code above, the implementation intercepts the PreInsert, PreUpdate and PreDelete events. If any of these event types is thrown, the code checks the affected domain object for the existence of either the userCreated or userUpdated property. If available, it uses the springSecurityService to access the currently logged-in principal and uses its uuid property, as this is the unique identifier of our users in this application.

To register the PrincipalPersistenceListener and attach it to a Grails datastore, we need to add the following code to BootStrap.groovy:


def ctx = grailsApplication.mainContext
ctx.eventTriggeringInterceptor.datastores.each { key, datastore ->

    def listener = new PrincipalPersistenceListener(datastore)
    listener.springSecurityService = springSecurityService

    ctx.addApplicationListener(listener)
}

To make this work, the springSecurityService needs to be injected, the same is true for grailsApplication.

But that's all we have to do to support our new domain class properties userCreated and userUpdated. The last step is to add both properties to the domain class(es) we want to track.

Conclusion

Grails integrates with Spring's event mechanism and provides the AbstractPersistenceEventListener base class to listen to certain GORM events. Grails uses this mechanism internally for example for the GORM event hooks but it can of course be used by the application logic too. This article showed how to introduce support for userCreated and userUpdated which are similar to dateCreated and lastUpdated but store the principal how created, updated or deleted a domain object.