javakaffee

The last months I was looking for a free, hosted Jenkins/Hudson offering to build my open source project memcached-session-manager (a memcached based session failover and session replication solution for sticky and non-sticky sessions compatible with tomcat 6.x/7.x).
This weekend finally I stumbled over CloudBees, which offers PaaS (”RUN@cloud”, similar to GAE or Amazon Beanstalk) and also provides the development infrastructure including git repositories, maven repos, and Jenkins/Hudson for continous integration (”DEV@cloud”). And the best thing is: they also offer a free bundle including a Jenkins instance - that was exactly what I was looking for!

So what was necessary to get up and running?

1. Signup for CloudBees
2. After completing the registration configure the Jenkins job
3. Done.

After that, as I wanted to have the Jenkins for memcached-session-manager be publicly available I submitted a support request with the corresponding question. Today, the first day after the weekend, I received the confirmation that it was done (really fast, you CloudBees!). Was that easy? It was!

I must say I’m really impressed by this service. Easy, valuable, makes fun to use.

So, do you know other free Jenkins/Hudson hostings?

This is mainly a note to myself regarding the “correct” installation of java on fedora, as I’m searching the concrete commands on each new setup of a fedora box.

The goal is to use the alternatives system as much as possible as this provides great flexibility.

These commands should be executed (after installing the jdk--linux--rpm.bin, here it was jdk-6u23-linux-x64-rpm.bin):

# The standard alternatives...
alternatives --install /usr/bin/java java /usr/java/jdk1.6.0_23/jre/bin/java 20000
alternatives --install /usr/bin/javac javac /usr/java/jdk1.6.0_23/bin/javac 20000
alternatives --install /usr/bin/jar jar /usr/java/jdk1.6.0_23/bin/jar 20000

# And perhaps this is also useful
alternatives --install /usr/lib64/mozilla/plugins/libjavaplugin.so libjavaplugin.so.x86_64 /usr/java/jdk1.6.0_23/jre/lib/amd64/libnpjp2.so 20000

# This is the most important IMHO but kept secret on the web...
alternatives --install /usr/lib/jvm/java-1.6.0 java_sdk_1.6.0 /usr/java/jdk1.6.0_23 2000
echo 'export JAVA_HOME=/usr/lib/jvm/java-1.6.0' > /etc/profile.d/jdk.sh
echo 'export PATH=$JAVA_HOME/bin:$PATH' >> /etc/profile.d/jdk.sh

The next time I install java on some system probably I’ll create a simple script that takes the version as an argument (and perhaps the architecture) to reduce this to a single step.

Please link / retweet this post so that I find it the next time when I’m searching for these commands :-)

I finally managed to release memcached-session-manager 1.1. This release covers pluggable session serialization/deserialization.

The main motivation was to get an alternative serialization mechanism to java serialization that supports different/new versions of classes (forward/backward compatibility) - in the end a deployment of a new software/application release should be possible with session failover still available.

Available serialization solutions

Therefore I started to watch out for serialization solutions that provide forward/backward compatibility. A very useful resource I found was the thrift-protobuf-compare project. It has a very good performance comparison of different serialization libraries. I checked one serialization solution after the other and had to realize that I have requirements that limit the potential solutions severely: this is the serialization of dynamic structures and the need to be able to recreate correct types during deserialization.

Schemaless serialization, serialization of dynamic structures

The serialization of dynamic structures is required, as users shall not be forced to provide s.th. like a schema for their session objects. Most of the time, users even don’t know really, what gets stored in the http session and what not. Therefore it’s rather impossible to provide a structural definition of the session attributes, and the serialization library has to figure out what needs to get serialized, and also what and how it is deserialized. Unfortunately, most of the “cool” and especially fast serialization libraries like protobuf, thrift or avro rely on some kind of schema that defines the serialization/deserialization and therefore they cannot be used.

JSON cannot be used as correct types need to be deserialized

Because of the second restricting requirement, the need to recreate correct types during deserialization, fast json libraries like jackson drop out.

Serialization with XStream

The serialization solution I chose was xstream, it does all what’s needed and also has a very simple to use api - all that needs to be done for serialization is s.th. like new XStream().toXML( Object, OutputStream ).

To get my own numbers, I compared the number of requests/second for java serialization and xstream based serialization. For this I created a simple wicket webapplication with some pages that differ in the amount what’s stored in the session.
Unfortunately I experienced the same that was said by the thrift-protobuf-compare project: xstream is considerably slower than java serialization.

Serialization with Javolution

Therefore I looked for other possible solutions and decided to use javolution for xml binding and to write the required reflection stuff to determine what needs to be serialized/deserialized.

The interesting part of this was that I learned a little bit more about reflection and serialization: I realized that it’s not possible with standard java reflection to deserialize private classes and classes without a default constructor. For this one needs to fall back to vendor specific solutions (e.g. the sun jdk come with a ReflectionFactory that allows to get a newConstructorForSerialization).

Other things I got aware of was that during serialization cyclic dependencies need to be handled correctly, and that during deserialzation different classes sharing the same reference to an object need to get these shared object references again. The solution for both requirements is the same: during serialization one needs to track which objects are already serialized and perhaps one just writes the reference to an already serialized object. During deserialization these references can be resolved accordingly.

After I solved these things and others with javolution I had another serialization strategy for the memcached-session-manager and could again compare the performance. The result was mixed: on the one hand I was happy because the javolution based solution is faster than xstream, on the other hand it was still slower than java serialization.

Looking for fast serialization solutions

So I’m still looking for serialization solutions that can do what’s needed and are still faster than java serialization.

One thing I’m currently working on is a solution based on aalto, which will be very similar to the javolution based serialization. But hopefully it will be even faster, as according to thrift-protobuf-compare aalto seems to be faster than javolution.

Another promising candidate is kryo. There still needs to be s.th. done so that it can be used (e.g. support for forward/backward compatibility and support for cyclic graphs), but I’m confident that Nate is going to implement these things.

The third candidate is jackson. The first thing that has to be implemented is including type information in json. After that cyclic graphs are still a challenge, but I still hope that at some time jackson will provide everything required.

If you know other fast serialization solutions or if you think that one of the libraries compared in thrift-protobuf-compare is still an option please let me know!

Finally developing version 1.1 of memcached-session-manager was fun: object serialization and deserialization is an interesting field, both the several problems that need to be solved and also the available solutions and libraries.

Now I’m looking forward to find/implement even faster serialization strategies and to optimize the performance of the memcached-session-manager in other ways.

After I started the memcached-session-manager in spring, version 1.0 is finally released.

To get this out I just had to fix some checkstyle stuff and add/complete javadoc. There’s also a new wiki page that shows how to setup and configure tomcat so that the memcached-session-manager does its work as intended.

When I had started this project and had a first version to try out, there were several users that contacted me via email with questions, suggestions and issue reports. And there were some that use the memcached-session-manager in their products. I was really happy to get this feedback and interest, this was very motivating - thanx to all of you!

If some of the suggestions are not yet implemented please be forgiving, their still on my list.

Just some weeks ago, I also started a google group for the memcached-session-manager, so that discussions can now take place publicly and questions/answers are shared.

Finally, what makes me really happy now after the release? It’s the fact that now all those new features and tasks can be tackled :) Just an unordered brain dump of what will come:

• Make serialization strategy pluggable (right now java serialization is used), so that different requirements could be implemented, e.g. forward/backward compatibility in terms of class versions ( so that old sessions can be picked up by a new software version after a deployment/release)
• Make backup storage pluggable, so that another backend instead of memcached might be used, e.g. ehcache server
• Managing configured memcached-nodes via JMX

(An up-to-date list of features/improvements should always be available in the issue tracker)

If you have any ideas/requirements what might be useful please let me know!

I just started my first project on google code with the source code hosted in github: the memcached session manager.

– At first I must say - creating new projects on google code or github feels completely different compared to some years ago when project were hosted on sourceforge.net. Remembering this feels somehow like stoneage :) –

So, back to the memcached session manager…

What problem does the memcached-session-manager solve?

Imagine you have a web application with sticky sessions running on tomcat and want to have some kind of session failover. You want to have a scalable solution for that – just add more servers to handle an increasing number of sessions. This can be handled by sessions that are stored for backup in memcached nodes: If a tomcat dies all other tomcats will take over the work of the lazy/dead one and fetch the sessions from the appropriate memcached node(s) and serve this session from thereon.

How does it work?

The MemcachedSessionManager installed in a tomcat holds all sessions locally in the own jvm, just like the StandardManager does it as well.

Additionally, after a request was finished, the session (only if existing) is additionally sent to a memcached node for backup.

When the next request for this session has to be served, the session is locally available and can be used, after this second request is finished the session again goes to the memcached node.

Now imagine the tomcat dies.

The next request will be routed to another tomcat. This tomcat (in more detail the memcached session manager) is asked for a session he does not know. He will now lookup the session in the memcached node (based on an id that was appended to the sessionId when the session was created). He will fetch the session from memcached and store the session locally in its own jvm - he is responsible for that session from now on. After the tomcat served this request of course he also sends the session to the memcached node.

That’s all.

What more?

Also memcached node failover is implemented: if a memcached node is not available anymore the session will be moved to anoder node - also the sessionId will be modified and a new JSESSIONID cookie will be sent to the browser.
How does the sessionId look like?

The memcached session manager knows a list of memcached nodes (e.g. as localhost:11211 localhost:11212 that he references by indes. This index is appended to the sessionId, so the sessionId might be 602F7397FBE4D9932E59A9D0E52FE178.0.

What’s planned?

• Update list of memcached nodes via JMX
• Automated integration testing of all this (start memcached servers, start tomcats, make requests + create sessions, simulate tomcat failure etc.)

Perhaps this is useful for other people as well - enjoy it and give feedback! :)

I also just posted this together with questions to the spymemcached group:

I’m just thinking about using memcached as a session store. For handling node failures and node adds/removals the normal way is to store sessions in the database - so that memcached is used as no more than the caching layer - data is persistent and available even when no memcached node is there.

This is quite reasonable, however I want to think about an alternative: Would it be possible to store the session in two memcached nodes, so that the 2nd node is chosen if the primary node is not there (or the session is not found in this node)? One could store the session not only in the primary node but additionally in the “next” node. Together with consistent hashing both node failures and adding new nodes should be handled well.

Making an example

Let’s look at an example to see if this seems to work. Asume you have 3 nodes, with hashes 1, 2 and 3 (just for simplicity in this example).

Then we put a value that consistently hashes to “3″. In this case the data is stored to the nodes 3 and 1.
Another case would be a value that consistently hashes to 4: this data is stored to nodes 1 and 2.

Just trying to put this in a matrix, with node hashes on the x-axis and the key hash on the y-axis:

Then lets look at two cases: a node failure/removal, and adding a node.

Node failure

Let’s look at what happens when e.g. node 3 or node 1 fails.

1. Node 3 fails
If a key hashes to 3, it is read from node 3 (not available) and node 1 (success).
If a key hashes to 4, it is read from node 1 (success).

2. Node 1 fails
If a key hashes to 3, it is read from node 3 (success).
If a key hashes to 4, it is read from node 1 (failure) and node 2 (success).

Adding a node

So what happens when a new node 4 (hashing to 4) is added?

If a key hashes to 3, it is still read from node 3 (success).
If a key hashes to 4, it is read from node 4 (failure) and then from node 1 (success).

These example show, that simple node failures and adds are fine. Of course only 1 node at a time can fail or can be added. If more than one node would be added or removed, previously stored data would be unavailable. Perhaps it would really be better to store data in the database…?

Yesterday I added another example resource to the spring-annotations sample that shows how to use spring-aop with jersey.

The example contains a resource class SpringAopResource that is created by spring (in this case via autodetection/classpath scanning). There’s an example SecurityAdvice defined with a pointcut matching methods annotated with a custom @Secure annotation. The resource method of the SpringAopResource of course is annotated with this annotation, so that the SecurityAdvice will be applied to this resource method. (The SecurityAdvice in our case does plain stupid logging, this should be changed of course :))

So that spring-aop can come into play our resource class needs to be proxied. This raises the question about subresources. Normally they would be created by the resource class, e.g. some UsersResource.getUserResource(String userId) would return some new UserResource(userId). At this point the UserResource could not be proxied by spring - simply impossible. For this we introduced the ResourceContext (I already blogged about this in the context of the initial jersey-spring integration), so that a resource class can fetch an instance of a subresource class from the IoC container.
This is demonstrated by the subresource locator SpringAopResource.getSubResource() which returns an instance of a SpringAopSubResource fetched from the ResourceContext.

The code is shown below - just too little to believe that this should be all :)

The configuration in applicationContext.xml for aspectj-annotation support, and the configuration of the advice bean.

<beans xmlns="..." etc.>
    ...
    <aop:aspectj-autoproxy/>
    <bean id="securityAdvice" class="com.sun.jersey.samples.springannotations.resources.aop.SecurityAdvice" />
    ...
</beans>

The @Secure annotation and the SecurityAdvice:

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface Secure {

}

@Aspect
public class SecurityAdvice {
    
    private static final Logger LOGGER = Logger.getLogger( SecurityAdvice.class.getName() );
    
    @Before( "@annotation(com.sun.jersey.samples.springannotations.resources.aop.Secure)" )
    public void check( JoinPoint jp ) {
        /* this might get some authentication/authorization info from the request 
         * and throw a WebApplicationException with a response with status 401 (unauthorized)
         * if the request is not authorized.
         */
        final Signature signature = jp.getSignature();
        LOGGER.info( "Authorized execution of " + signature.getDeclaringTypeName() + "." + signature.getName() );
    }
    
}

The resource and subresource classes:

@Path("/spring-aop")
@Component
@Scope("singleton")
public class SpringAopResource {
    
    @Context
    private ResourceContext _resourceContext;

    @Autowired
    private Item _item;

    @GET
    @Secure
    @Produces("application/xml")
    public Item getItem() {
        return _item;
    }
    
    @Path( "subresource" )
    public SpringAopSubResource getSubResource() {
        return _resourceContext.getResource( SpringAopSubResource.class );
    }
    
}


@Component
@Scope("prototype")
public class SpringAopSubResource {

    @Autowired
    @Qualifier("1")
    private Item2 _item;

    @GET
    @Secure
    @Produces("application/xml")
    public Item2 getItem() {
        return _item;
    }
    
}

That’s all. The Item and Item2 classes are just normal beans injected by spring.

You can pull the full example from subversion or browse it online.

After I have blogged about the jersey-spring integration (jersey-spring integration mostly complete, jersey-spring and changes now available in the jersey trunk / namespace com.sun.ws.rest renamed to com.sun.jersey) and Paul posted an update on the jersey-spring integration in his blog, I finally added a sample that shows how to use jersey and spring (using annotations provided/supported by spring >= 2.5: @Resource and @Autowired).

One resource class in this sample app also shows how users of spring < 2.5 (e.g. spring 2.0) can use annotation based autowiring, this is achieved using the @Autowire annotation provided by jersey-spring (which I already described in this posting).

This sample is especially useful as it will be up to date in the future: it will reflect configuration, namespace or other changes made in jersey, jersey-spring or even spring, so that there’s not the problem that there will be information that is out of date ;-)

Puh, I just wrote a bunch of text that describes what I implemented related to wadl for jersey, so I just drop a line here. The generated WADL is really cool as it contains javadoc comments from resource classes, and even might include example xml snipplets - for some users of REST APIs xml is easier to read than xsd 8-)

The documentation is not yet complete but covers most of the features:

• The general description what’s provided and done
• What the user has to do to get cool WADL
• Which javadoc tags are supported and provided

There’s still documentation missing that shows how to use the maven-wadl-plugin to generate the WADL without a running jersey app. Also some documentation should be written that describes, how the user can define and process his own javadoc tags and turn them to WADL.

Still s.th. to write, but the biggest part should be done - puh :-)

In a previous posting I blogged about integration of spring into jersey (and IoC-containers in general), and showed, how the new Inject annotation can be used to inject dependencies from your custom ComponentProvider.

To support this Inject annotation, a feature was introduced into jersey, that allows you to define your own annotations and inject instances to fields annotated with this annotation (in your resource classes). For retrieving instances for the specific annotation you have to add an Injectable for your annotation to the WebApplication. The Injectable is then asked to resolve the value for some field annotated with the specific annotation.

The easiest thing is probably to give an example…

Register the Injectable for some annotation we now call MyAnnotation. This can be done e.g. in your custom ServletContainer, where you have access to the WebApplication:

import com.sun.jersey.api.core.ResourceConfig;
import com.sun.jersey.spi.container.WebApplication;
import com.sun.jersey.spi.container.servlet.ServletContainer;
import com.sun.jersey.spi.resource.Injectable;

public class MyServletContainer extends ServletContainer {

    @Override
    protected void initiate( ResourceConfig rc, WebApplication wa ) {
        super.initiate( rc, wa );
        
        wa.addInjectable(new Injectable<MyAnnotation, String>() {
            @Override
            public String getInjectableValue(Object o, Field f, MyAnnotation a) {
                return "foo"; // can be retrieved from somewhere
            }

            @Override
            public Class<MyAnnotation> getAnnotationClass() {
                return MyAnnotation.class;
            }
        });
    }
    
    @Target({FIELD, PARAMETER, CONSTRUCTOR })
    @Retention(RUNTIME)
    @Documented
    public static @interface MyAnnotation {

    }

}

In your resource class, you can then annotate fields with MyAnnotation to get the value from the Injectable:

@Path("/")
public class MyResource {
        
    @MyAnnotation String injectedValue;
        
    @GET
    public String get() {
        return injectedValue;
    }                

}

The GET method returns “foo” in this case, but of course you want to retrieve this instance of String from somewhere else.

With this feature you can e.g. create some DAO annotation, annotate the fields in your resource classes referencing DAOs and retrieve instances for these DAOs from your custom ComponentProvider.

How to configure a custom ComponentProvider or your custom ServletContainer see my posting concerning the spring-integration or Paul Sandoz’ blog.