It turns out I’ve been at Tradeshift for quite a while now – it’s been almost two years since we started development, and we’ve achieved more than we could have dreamt of when we started. Working at Tradeshift has been great fun, and there’s no joy bigger than being able to come up with a new feature, design it, develop it, and deploy it into production in a matter of days. To be fair, most features do take longer to mature, but the basic premise is that given some quality criteria, we can push new features out at any time.
Many startups do this, but it’s a welcome change from normal consulting, as I did at Trifork, and it’s even better because we have a personal investment in Tradeshift (not in the form of money, but in the form of deep belief in the greater cause). However, one of the main differences from working at Trifork is the visibility of what we do. Trifork has the QCon conferences, where all employees go and get new input, talk to trend-setting developers, and generally get out a lot. We, on the other hand, do most of our work ourselves, and when looking at Tradeshift from the outside, it seems somewhat boring – who’d want to build an invoicing product?
Well, it turns out to be much more fun (and challenging) than you’d expect, and this is an attempt to raise awareness of some of the quite cool things we’re doing – both so that we ourselves appreciate it even more, and of course also to attract new talent.
Of course, we won’t be able to go through the entire technology stack in one article, so it’ll probably be some kind of series. We’re not that much into long-term planning at Tradeshift, so I won’t promise anything on timing
The first topic will be our use of Groovy – triggered by the Tradeshift Æbleskiver & Glögg event, where I apparently am scheduled to talk a little about just that.
First a quick intro to the basic Tradeshift platform: we’re primarily talking a standard Java stack, powered by Spring and with a transactional database underneath (Postgresql 9.0). The Java-based backend exposes a number of REST resources, which we use from our Grails-based frontend. The backend also exposes a more restricted API, which is exposed to external use in our public API.
Besides Grails, which will be another topic, we use Groovy primarily as a DSL language. We don’t have any “real” code implemented in Groovy on the backend, but we use it extensively to patch the void between configuration and code.
Begin a global platform, we have a lot of configuration around how addresses are formatted, how company identifiers behave, how different types of document look and behave, and so on. Much of this is static configuration, in the sense that there’s no behavior to it. However, we quickly found out that sometimes we do want to add small snippets of behavior. We also found out that encoding the configuration and behavior as Java classes didn’t work out very well.
Instead of encoding as classes, we actually just wanted different instances of one class, for example a class representing a company identifier. We’d then like to be able to ask that instance if a given value is valid, or for the description of the identifier. So, we’ve come up with a small DSL, which basically can map a Groovy config script to any Java class.
It goes something like this. Assume we have a class called Identifier:
There’s no real magic here… altough there is a slight surprise in there, because we use a Groovy closure. In other words, an identifier can have a format specified as a regular expression, and in addition to that, it can also have a custom validation block. Other than that there’s no magic from a Java point of view. Users of the class have no idea about the internals.The Groovy part of it looks like this:
Notice how this structure maps onto the Java class – each Groovy property matches a property in Java. The block key, ‘ID1′ in this example, becomes an argument to the constructor. Now, the validation block itself isn’t very interesting, but the point here is that you can now inject any code. This structure basically makes it possible to create instances of the Identifier type simply by adding them to the Groovy script.
Of course the instances have to come from somewhere in Java space. When we persist data, we use the instance key, and when loading data from the database, we use that key to look up the instance. The instance is loaded by what we call an ObjectBuilder (it can’t get any more generic than that). The ObjectBuilder basically takes a Groovy config file and returns a list of objects. The ObjectBuilder is where the real magic happens, and the full contents can be found here: https://gist.github.com/1440345
At the base, ObjectBuilder simply uses the Groovy ConfigSlurper to read the config file (from classpath, filesystem, or database). Reading using ConfigSlurper will basically return a map of objects, somewhat untyped. To fit it into an object, a good amount of reflection is needed.
Using ObjectBuilder is a matter of calling readConfig, which takes a Groovy config string and a type. The type is the class which the Groovy config will be mapped into.
Now, the ObjectBuilder has a number of different features, and I won’t go into all of them, just mention a couple.
Spring support
We base everything on Spring. The ObjectBuilder is aware of this, although it doesn’t require Spring to work. If the ObjectBuilder is created as a Spring bean itself, any instance created by the ObjectBuilder can contain autowired fields – they won’t be full Spring beans, but they can have dependencies injected. This is done by ApplicationContext.getBeanFactory().autowireBean(bean);
Also, code blocks of any kind can specify a Spring bean as a parameter. That parameter will be curried automatically, so that any callers only have to provide actual data into the block. Currying is another pretty nice feature of functional programming, and basically means that one function is wrapped in another function, which then sends some arguments by default. Think of it as this in Java terms:
The first argument can be curried, and will give a function like this:
Arbitrary object structures
The ObjectBuilder uses reflection to find class properties, either private fields or getters/setters. The property types can be of any complex type, since the ObjectBuilder goes through the structure recursively. Some constructs are not supported since we haven’t had a need for it yet, but basically anything goes, except for final fields.
Line number generation
When reading in a Groovy config string, the ConfigSlurper actually compiles the config string into a Groovy class. Unfortunately, the Groovy compiler discards the source code after compilation, so you can’t go from a compiled closure to the source code. This is not a problem under normal circumstances, but we have features where we want to display just the closure code in UI (for some pretty advanced editing). To do that, we need to know where in the config string the block is located. We could parse the file ourselves, but Groovy isn’t exactly to parse. So, instead of doing that we hook into the Groovy compiler using an AST visitor. Using an AST visitor, you can be notified for every operation that’s being compiled. We then look for equals expressions (for example validation = { -> }) with a variable expression on the left and a closure on the right. At this point, we’re still compiling, so we can access the sourcecode location. We save this in a map, which can then be retrieved later on.
Sandboxing
For what is probably very advanced use, we actually want to allow normal users of the platform to write Groovy blocks, which we run as part of the document processing. For example, some of our customers have quite specific validation needs, which don’t really align with anything else we’re doing. In those cases, we simply want to evaluate a validation block written by the customer.
Of course, we cannot just allow anything. Actually, we don’t want to allow much at all, but restricting Groovy is quite hard, and at the very least you normally have to enable a SecurityManager and do all kinds of fiddling with that.
We’ve chosen another route, at least tentatively. The main issue with Groovy is that static invocations are resolved at compile time. That means that we cannot to the usual Metaclass juggling to reroute requests, as methodMissing/propertyMissing won’t be hit for something like System.exit(0); – that will just be executed directly.
So, to fix that we hook into the compiler again. As with line numbers, we add an AST visitor and look at what’s being compiled. We then just throw exceptions if an illegal node is found.
In our case, we just disallow any class reference, both static and through getClass(). This handles most of it, and we then also disallow access to some of the default Groovy methods such as execute, invokeMethod, metaClass, and so on.
As you might have noticed, the readConfig method has a @Cacheable annotation on it – basically the output should always be stateless, and can then be cached indefinitely, so load performance isn’t really an issue under normal circumstances.
To avoid direct access to the ObjectBuilder all over the place, we normally wrap it in a factory class for the given type, for example an IdentifierFactory, which on startup reads the config, saves all the instances in a map, and then allows access through getters, for example getIdentifier(String key).
And that just about sums up our primary use of Groovy in the backend. It’s not much, but it has helped us enormously, and we use it extensively for just about any kind of domain configuration.
