Traveling, IT, open-source and life
April 25th, 2013

One week on Tenerife

We spent last week on vacations in Tenerife, enjoying the nice climate and all the hiking possibilities.

The highlight of this short stay was to climb on the Teide volcano, in two steps – first to the Altavista hut at 3200m, then to the top, the next morning at 5:45 AM, to catch the sunrise at 3700m.

Teide-top

At the top of the Teide at sunrise, with the shadow of the crater

One nice thing I noticed when coming back is that one week of hiking and spending the day outside really helped refuel my creative energy – I have not been as productive as I was this week in a long time.

April 9th, 2013

Dynamic multi-modular SBT project

As the Delving Culture-Hub project grows, I found myself adding module definitions by hand to the SBT more often than what I’d wish for, and for each adjustment of a common value, having to go through all modules and adjust them.

So after a while, I wondered whether it wouldn’t be possible to discover modules dynamically, rather than maintaining them by hand. It turns out that this is a bit tricky, but still possible.

The main limitation is that as by default, projects are discovered via reflection, we can’t dynamically provision the projects of a build from the same Build definition. Or in other words, we need two separate Build files. As pointed out by Mark Harrah in the comments, we can override the projectDefinitions method and use it in order to define a simple dynamic build.

Let’s take a simplified example:

import sbt._
import Keys._

object Build extends sbt.Build {

  def modules(base: File) = for (x <- (base / "modules").listFiles if x.isDirectory) yield
    Project(id = "project-%s".format(x.getName), base = x)

  def toRef(base: File) = modules(base).map{x => x: ProjectReference}

  override def projectDefinitions(baseDirectory: File) = modules(baseDirectory) ++ Seq(root(baseDirectory))

  def root(base: File) = Project(id = "root", base = file("."), aggregate = toRef(base))
}

The modules method takes care of looking at the contents of the modules directory, and turns each directory into a SBT project definition. We then override the projectDefinition method to replace it with those dynamic definitons.

In practice, things get a little more complicated, mainly when there are dependencies between projects. I found that it makes sense to declare modules that many others depend upon explicitly anyway, rather than trying to do the dependency game dynamically. I suppose that this would be possible to achieve, but may in practice become quickly somewhat complex, given the restrictions brought in by reflection.

Initially, I had also planned to declare custom settings of modules that require them via the build.sbt mechanism, i.e. having a custom build.sbt file at each module root and pass in settings there. However, this does not seem to fare to well in this case, because the resulting Build includes import definitions twice. If I get this part to work, I’ll write about it in another post.

March 26th, 2013

Multi-modular development with Play 2.1

Play 2.1 makes it possible to write completely modular applications, and on top of this, to develop each module separately. I’ve been waiting for this for some time now, and even implemented a custom router based on plugins while waiting for the framework to support modular routing. Now that the wait is over, let’s see how to take advantage of this feature.

The Play documentation describes how to set-up a multi-modular project via SBT sub-projects. What is important to understand here is that:

  • sub-projects are a feature of SBT, and next to having a central Build.scala file, it is possible to add configuration to individual sub-projects via build.sbt files (in those sub-project’s root directory)
  • the proposed folder structure (i.e. putting the modules in a modules/ directory) is optional (but a good convention nonetheless)
  • each sub-project is a play.Project, so routes and messages files in conf/ will be recognized

Let’s get started with creating a new project:

manu@manu-mac ~/workspace » play new multi-module-example                                                                                                                                                                                                255 ?
       _            _
 _ __ | | __ _ _  _| |
| '_ \| |/ _' | || |_|
|  __/|_|\____|\__ (_)
|_|            |__/

play! 2.1.0 (using Java 1.7.0_15 and Scala 2.10.0), http://www.playframework.org

The new application will be created in /Users/manu/workspace/multi-module-example

What is the application name? [multi-module-example]
>

Which template do you want to use for this new application?

  1             - Create a simple Scala application
  2             - Create a simple Java application

> 1
OK, application multi-module-example is created.

Have fun!

Now that we have created a parent project, let’s right away create a first module:

manu@manu-mac ~/workspace » cd multi-module-example
manu@manu-mac ~/workspace/multi-module-example » mkdir modules
manu@manu-mac ~/workspace/multi-module-example » cd modules
manu@manu-mac ~/workspace/multi-module-example/modules » play new module1
       _            _
 _ __ | | __ _ _  _| |
| '_ \| |/ _' | || |_|
|  __/|_|\____|\__ (_)
|_|            |__/

play! 2.1.0 (using Java 1.7.0_15 and Scala 2.10.0), http://www.playframework.org

The new application will be created in /Users/manu/workspace/multi-module-example/modules/module1

What is the application name? [module1]
>

Which template do you want to use for this new application?

  1             - Create a simple Scala application
  2             - Create a simple Java application

> 1
OK, application module1 is created.

Have fun!

Now that we have the basic structure in place, let’s adjust the main’s project Build.sbt. At this point, we just want to make sure that our sub-module gets built, tested and packaged together with the main project. That is why we need to add the aggregate instruction to the main project. The projet/Build.scala build file should look like this:

import sbt._
import Keys._
import play.Project._

object ApplicationBuild extends Build {

  val appName         = "multi-module-example"
  val appVersion      = "1.0-SNAPSHOT"

  val appDependencies = Seq(
    // Add your project dependencies here,
    jdbc,
    anorm
  )

  val module1 = play.Project("module1", path = file("modules/module1"))


  val main = play.Project(appName, appVersion, appDependencies).settings(
    // Add your own project settings here
  ).dependsOn(
    module1
  ).aggregate(
    module1
  )

}

Note that we do not any longer need the Build.scala of the module, i.e. the directory modules/module1/project can be deleted.

Now, what we need for our module to work properly, is to:

  • Set-up the routing correctly
  • Move the controller to the right package
  • Wire-up the routes of the sub-module into the parent’s router

First, let’s rename the modules/module1/conf/routes file to modules/module1/conf/module1.routes and adjust its content:

# Routes
# This file defines all application routes (Higher priority routes first)
# ~~~~

# Home page
GET     /module1                    controllers.module1.Application.index

# Map static resources from the /public folder to the /assets URL path
GET     /assets/*file               controllers.module1.Assets.at(path="/public", file)

Note how the generated Application controller is now moved to the module1 package (it needs to be adjusted in the modules/module1/app/controllers/Application.scala file as well), and the Assets controller is also part of the module1 package. Each sub-project needs its own Assets controller, if it should provide assets.

The Assets file looks like this:

package controllers.module1
object Assets extends controllers.AssetsBuilder

Now, let’s plug-in the new router in the main’s project router, in the conf/routes file:

# Routes
# This file defines all application routes (Higher priority routes first)
# ~~~~

# Home page
GET     /                           controllers.Application.index

->      /                           module1.Routes

# Map static resources from the /public folder to the /assets URL path
GET     /assets/*file               controllers.Assets.at(path="/public", file)

We can check that this works by accessing an undefined route:

all-routes

Now that everything is in place, there is just one last thing to do in order to make the development a pleasurable experience. Indeed, it is possible to develop each module separately, and treat it as a separate application, by switching to this project in the SBT console:

[multi-module-example] $ projects
[info] In file:/Users/manu/workspace/multi-module-example/
[info] 	   module1
[info] 	 * multi-module-example
[multi-module-example] $ project module1
[info] Set current project to module1 (in build file:/Users/manu/workspace/multi-module-example/)
[module1] $ run

--- (Running the application from SBT, auto-reloading is enabled) ---

[info] play - Listening for HTTP on /0.0.0.0:9000

(Server started, use Ctrl+D to stop and go back to the console...)

However, this won’t work right away:

no-router

As we now switched to the module project, effectively making it the currently active Play project, the default application configuration kicks in. By default, the router is defined in the conf/routes file.

Luckily, there is a way to change this in the application.conf:

# Router
# ~~~~~
# Define the Router object to use for this application.
# This router will be looked up first when the application is starting up,
# so make sure this is the entry point.
# Furthermore, it's assumed your route file is named properly.
# So for an application router like `my.application.Router`,
# you may need to define a router file `conf/my.application.routes`.
# Default to Routes in the root package (and conf/routes)
# application.router=my.application.Routes

Let’s adjust our module’s application.conf to do just this, and to also inherit from the parent’s configuration:

include "../../../conf/application.conf"

application.router=module1.Routes

And that’s pretty much it, we can now work independently on our new module:

module1-route

October 26th, 2012

Activating logging in test mode in Play 2.0.x

Because of a permgen leak related to third-party libraries, the logging in test mode is disabled in Play 2.0.x (there’s a fix coming up in Play 2.1). In some cases, seeing what happens while running a test can be useful – e.g. when debugging calls to various Akka actors that don’t quite behave as expected.

Luckily there’s a workaround, which is to specify a logger for logback by hand when starting play or SBT:

play -Dlogger.resource=test-logger.xml

does the trick.

November 18th, 2011

MongoDB-based Cache for Play 1.2.x and Play-Scala

In case anyone needs this: https://gist.github.com/1377952

You’ve got to bootstrap it in some bootstrap job like so:

 ScalaCacheAccessor.set(new MongoCacheImpl)

It’s only tested with the Press module so far (shared state is evil anyway ;) )

November 1st, 2011

Writing custom de/serializers for Jerkson

Since I wrote my previous post on how to use Play! Scala with Jerkson, I was asked a number of times how to make use of custom de/serialization for classes. This happens when e.g. the case class you want to de/serialize has a POJO or another complex type that you’d like to have more control over, such as e.g. the BSON ObjectId or Play’s Anorm PK.

Here’s a wrap-up on how you can easily create custom de/serializers and use them within your Play! Scala application. Let’s take the example of the BSON ObjectId, which you will most likely run into if you use MongoDB in the backend.

First, we need to create a Jackon serializer and de-serializer for the org.bson.types.ObjectId type:

import org.codehaus.jackson.map.annotate.JsonCachable
import org.bson.types.ObjectId
import org.codehaus.jackson.map.{DeserializationContext, JsonDeserializer, SerializerProvider, JsonSerializer}
import org.codehaus.jackson.{Version, JsonParser, JsonGenerator}

@JsonCachable
class ObjectIdSerializer extends JsonSerializer[ObjectId] {
  def serialize(id: ObjectId, json: JsonGenerator, provider: SerializerProvider) {
    json.writeString(id.toString)
  }
}

class ObjectIdDeserializer extends JsonDeserializer[ObjectId] {
  def deserialize(jp: JsonParser, context: DeserializationContext) = {
    if (!ObjectId.isValid(jp.getText)) throw context.mappingException("invalid ObjectId " + jp.getText)
    new ObjectId(jp.getText)
  }
}

This is pretty straightforward: when serializing, we just print out the string representation of the ObjectId via toString(), and when deserializing we check if we have a valid ObjectId and then create an instance given the text representation (or throw an exception if we don’t get a valid one).

Now all what is left to do is to register this de/serializer couple. For that purpose we can extend Jerkson’s Json trait that gives us access to the mapper and register a new Jackson SimpleModule:


import org.codehaus.jackson.map.module.SimpleModule

object CustomJson extends com.codahale.jerkson.Json {
  val module = new SimpleModule("CustomJson", Version.unknownVersion())
  module.addSerializer(classOf[ObjectId], new ObjectIdSerializer)
  module.addDeserializer(classOf[ObjectId], new ObjectIdDeserializer)
  mapper.registerModule(module)
}

And that’s all that is needed. If you are using a custom action in Play! in order to generate Json, make sure to update it so that it uses the CustomJson object we just created:


trait JsonAction { self: Controller =>
  def Json(data: AnyRef): Result = new Result() {
    def apply(request: Request, response: Response) {
      val encoding = getEncoding
      setContentTypeIfNotSet(response, "application/json; charset=" + encoding)
      response.out.write(CustomJson.generate(data).getBytes(encoding))
    }
  }
}

September 16th, 2011

Play! Scala and JSON

Update: If you need to have custom de/serialization of some types, check out this new post

I’ve been working with Play! Scala for a couple of months now, and the experience is really good – the combination of the Scala language and the sound design principles of Play! make web-development a real pleasure.

One of the aspects I have been fighting with for a while (and which had initially put me off writing my first bigger Play! application with Play! Scala) is the lack of simple support for JSON de/serialization. With simple I mean the possibility to turn a case class into a JSON string and back, and to additionally have access to a couple of more advanced features that are often required (e.g. implementing custom serialization of specific types).

From the recent development on the Play! Scala Github repository, it looks as tough sjson will be used as the integrated library with the framework.

I have tried out sjson and my experience with it wasn’t all that good, I outlined the issues I had with it (mainly that it doesn’t offer support for custom de/serialization of Java beans) in this post to the Play! mailing-list.

The alternative I found bears the sexy name of Jerkson and is developed by Coda Hale. It is a scala wrapper for Jackson which supports all kind of features you may want from a JSON library.

The latest release now makes the integration of Jerkson very easy, as it addresses two issues that happened in the Play! context: it finds the custom classloader Play! uses, and allows to deserialize case classes with more than one constructor (this can be problematic in some cases, and Play! adds an empty constructor to case classes, necessary for e.g. the integration with SnakeYML).

Since the releases are available via Maven, it is possible to easily get Jerkson into a Play! Scala project by adding the following dependency to dependencies.yml:

- com.codahale -> jerkson_2.8.1 0.4.2-SNAPSHOT:
    exclude:
        - org.scala-lang -> scala-library
repositories:
    - codahale:
        type: iBiblio
        root: "http://repo.codahale.com/"
        contains:
            - com.codahale.*

And… that’s it. You can now use Jerkson like this (don’t forget to run a play dependencies):

import com.codahale.jerkson.Json._
case class Foo(bar: String, baz: Int)
val serialized: String = generate(Foo("bar", 42))
val deserialized: Foo = parse[Foo](serialized)

Additionally, you can use a trait to mixin with your controllers in order to make JSON serialization look more integrated with other actions:

trait JsonAction { self: Controller =>
  def Json(data: AnyRef): Result = new Result() {
    def apply(request: Request, response: Response) {
      val encoding = getEncoding
      setContentTypeIfNotSet(response, &amp;quot;application/json; charset=&amp;quot; + encoding)
      response.out.write(Json.generate(data).getBytes(encoding))
    }
  }
}
March 23rd, 2011

Selenium IDE plugin for the Play! framework

Two months ago we started using the Play! framework for our work at Oxiras (as I am writing these lines, the Oxiras website is still totally empty, this should change soon however).

As we started making UI tests using the excellent integration with the Selenium test runner, we quickly noticed that also having an integration with the Selenium IDE plugin would make our lifes much easier. So I looked a bit around, found that creating a Selenium IDE formatter was not all too hard and here we go with what I can call my first Firefox plugin.

So far, there’s no update site defined as the GitHub pages are not available via HTTPS, so this will have to wait until I set up SSL on some machine this is now set-up on the oxiras domain, which means updates will automatically be detected.

If you’re using the plugin, feel free to contact me for feedback / comments / bug reports!

Edit

February 11th, 2011

Information Retrieval

This morning I spent some time looking for information on how to deploy a legacy webservice on Apache Karaf.

This post isn’t about Karaf and OSGi and webservices though, but a reflection on how to efficiently find information that can lead towards a specific technical problem on the Internet. For quite some time now I feel that this task has become much more complicated in the past 2-3 years than it used to.

We are now in 2011 and at least in my perception, Google isn’t anymore a simple, efficient nor even effective way of finding information regarding specific problems. Which is an euphemism for “Google really sucks these days”. Type in a query, click on a link, search again for something rather different and you’ll see one of the results you clicked earlier appear in the results of your new query (which is related to the previous query by maybe only one keyword, two at most. What’s worse is that in many cases, the page that Google lists as result don’t even contain one of the keywords entered in the query, but have something to do with sites I browsed in the past. Seriously Google, why should I care about things I searched for in the past in a new query?

And yet, especially when it comes to software development, finding sound information about a problem is critical. Way too often I find myself rushing to Google and searching for a solution, then finding some random piece of information and trying to get something out of it – i.e. starting to work on a solution on a shaky ground, even though I perfectly know I’m on shaky grounds (intuition is very helpful there).

So there’s this one issue of finding high-quality information. There’s another, much bigger problem though: understanding the problem to look for the right solution. Oftentimes do I rush into looking for a solution when I haven’t yet understood the problem (or think I do, but really am just headed in the plain wrong direction).

At this point, I shall point to The Five Orders of Ignorance by Phillip G. Armour. If you haven’t read it and are in the business of writing software, do so. Now.

For the sake of the completeness of this post, I’ll list the Five Orders anyhow. They go like this:

  • 0th Order – Lack of Ignorance: “I have 0OI when I provably know something”
  • 1st Order – Lack of Knowledge: “I have 1OI when I don’t know something”
  • 2nd Order – Lack of Awareness: “I have 2OI when I don’t know that I don’t know something”
  • 3rd Order – Lack of a Suitably Efficient Process: “I have 3OI when I don’t have a suitably efficient way to find out that I don’t know that I don’t know something”
  • 4th Order – Meta Ignorance: “I have 4OI when I don’t know about the Five Orders of Ignorance”

So how does that relate to the information search issue from before? Well, as I pointed out, oftentimes a problem when looking for information is the lack of clear understanding of the problem I try to solve. So this is – as far as I understood the 5 Orders – a 2OI: I lack the awareness that I do not understand (know) the problem for which I am looking for a solution.

So how to get from 2OI to 1OI? Of course this is situation dependent, but I would say that when you have some domain knowledge things are much easier. In software development, there’s one type of problems which I call “configuration problems” (although they could probably also be called “integration problems”) and that go like this:

  • how do I deploy…
  • how can I configure…
  • I need to integrate…

and involve one or more technologies of the many, many technologies that are available in the open-source space.

Whenever I get to one of these questions (which appear to be 1OI-type-of-questions), I try to step back and ask myself: does that even make sense? Isn’t there a deeper problem I actually want / need to solve? I say that I “try to” step back because in some cases I either forget and rush into finding a solution to the wrong problem, or I know or feel that this is not the right question, but have a personal itch to get things to work anyway (“I really want Spring DM 2 to work on Karaf! It just has to work!”).

Now let’s go back to the initial issue described in this post (Google sucks): how to find answers to technical questions on the Internet these days?

When I think of not using Google, I am now trying the following:

  • if the problem relates to a specific technology, I try to find a mailing-list and its archive and look there first. It may just be a POMO (Plain Old Mailing-List) but I found that oftentimes, you can get good answers just by looking at the archives (and by archives I do not mean Googling for the archives but browsing through them or searching only the archive site with Google’s “site” keyword)
  • search and ask on Stackoverflow
  • subscribe to a technology mailing-list and send the question there, to the developers and experienced users. I used to avoid this step for some time, wanting to solve things all by myself…but really it is not worth the trouble.
  • check out the damn code. There is a lot in the code, much more than you’d think. And looking into the internals can sometimes be much more efficient than searching for a sample or documentation – let’s be honest, not too many projects have always up-to-date documentation and samples.

Allright, that’s it. Let’s hope Google fixes their search engine, or offers a paying high-quality service for search (I would pay for that).

December 15th, 2010

Deploying a GWT application as OSGi module with Apache Karaf and Maven

In a project I’m currently working on we use the OSGi platform in order to achieve higher modularity (the project is about building a framework for massive data ingestion and processing for a high-availability service, and the processing is dealt with via workflows that are composed of a number of plugins that we don’t develop, hence the need for modularity).

We use the Apache Karaf framework, which offers neat extension points and means to customize it. One of the nice features is the WebConsole that can also be extended, which would provide the means to add a GUI for the users. However, this happens through providing a Servlet implementation, which isn’t really handy these days when wanting to develop a bit more complicated GUIs.

Since I’ve got some experience with Google Web Toolkit I thought of using this instead for coding the GUI (and eventually integrate it with the WebConsole later on, not sure at which level of depth yet). And the first thing necessary in order to do this is to get the GWT application to run on the Karaf platform.

Here’s how I managed to get it work:

Step 1: create a vanilla GWT application with the gwt-maven-plugin

I used the gwt-maven-plugin archetype in order to create a new GWT module. At the time at which I am writing this post, I had some issues with the unit tests that are generated by default, so I just trashed them for the moment.

Step 2: make the GWT application ready for deployment via Pax Exam Runner

Karaf can leverage Pax Runner in order to deploy OSGified WARs. For that, you need to have a MANIFEST.MF generated and attached to the WAR maven generates. There’s two things required for this.

First, configure the Apache Felix maven-bundle-plugin to generate the manifest for you (you could do it by hand, but why bother):

            <plugin>
                <groupId>org.apache.felix</groupId>
                <artifactId>maven-bundle-plugin</artifactId>
                <version>2.1.0</version>
                <extensions>true</extensions>
                <executions>
                    <execution>
                        <id>bundle-manifest</id>
                        <phase>process-classes</phase>
                        <goals>
                            <goal>manifest</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <supportedProjectTypes>
                        <supportedProjectType>jar</supportedProjectType>
                        <supportedProjectType>bundle</supportedProjectType>
                        <supportedProjectType>war</supportedProjectType>
                    </supportedProjectTypes>
                    <instructions>
                        <Bundle-ClassPath>.,WEB-INF/classes</Bundle-ClassPath>
                        <Embed-Directory>WEB-INF/lib</Embed-Directory>
                        <Embed-Dependency>*;scope=compile|runtime</Embed-Dependency>
                        <Embed-Transitive>true</Embed-Transitive>
                        <Import-Package>*;resolution:=optional</Import-Package>
                    </instructions>
                </configuration>
            </plugin>

This will generate a manifest which takes into account the required dependencies that come with the WAR, so that it can be deployed by Pax Runner. Note that this isn’t just a configuration for GWT itself, but in theory for any kind of WAR.

Update: I figured out how to customize the webapp name, by adding the following in the manifest instructions:

                        <!-- OSGi spec config option -->
                        <Web-ContextPath>gui</Web-ContextPath>
                        <!-- Pax Runner knows this option -->
                        <Webapp-Context>gui</Webapp-Context>

Pax Runner does not understand the OSGi spec <Web-ContextPath> instruction, so it needs to be given the <Webapp-Context> instruction.

Next, you need to tell the maven-war-plugin to add the generated manifest file to the WAR:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                ...
                <configuration>
                    ...
                    <archive>
                        <!-- add the generated manifest to the war -->
                        <manifestFile>${project.build.outputDirectory}/META-INF/MANIFEST.MF</manifestFile>
                    </archive>
                </configuration>
                ...
            </plugin>

Step 3: Activate war deployment support on Karaf

Fire up the Karaf console and type:

features:install war

This will activate everything needed in order to deploy WARs on the platform.

Step 4: Deploy the WAR

Now you just need to deploy the WAR on Karaf. You can do this in several ways:

  • by copying the WAR file to the deploy/ directory of the Karaf platform
  • by defining a feature descriptor and adding the war artifact to it

Since we already had a feature descriptor (because we have many bundles that need to be deployed, I just added the artifact to the feature descriptor XML like this:

<?xml version="1.0" encoding="UTF-8"?>
<features name="uim-features">
    <feature name="uim-core" version="1.0.0.SNAPSHOT">
    ...
               <bundle>mvn:eu.europeana/europeana-uim-gui-gwt/1.0.0-SNAPSHOT/war</bundle>
    ...
    </feature>
</features>

Note that you need to explicitly reference the “war” classifier, or this won’t work. Now, re-install the feature in order for the changes to take effect.

Step 5: Enjoy

You should see something like this (amongst others) in the log:

14:42:50,487 | INFO  | l Console Thread | WebXmlObserver
| nder.war.internal.WebXmlObserver  128
| 581 - org.ops4j.pax.web.pax-web-extender-war - 0.7.3
| Using [gui] as web application context name

That’s it, you can access your GWT module at http://localhost:8181/<web app name>

This work is licensed under GPL - 2009 | Powered by Wordpress using the theme aav1