ResourceTransformFilter – DataUris, LessCSS, Javascript and CSS Compression Made Easy!

ResourceTransformFilter is a swiss-army-knife for HTML/CSS/Javascript transformation in Java web applications.

It’s a new addition to the visural-common Apache 2.0 licensed project.

Apply this filter in your Java webapp to get -

  • Automatic CSS DataURI image inlining for supported browsers
  • Automatic CSS MHTML image inlining for supported IE browsers
  • LessCSS compilation for .less files
  • Automatic YUI compression of .css & .js resources
  • Optional: DataURI inlining of <img> tags in HTML responses

So as you can see, the primary purpose of this filter is web app resource optimization.

DataURIs and MHTML?

by http://www.flickr.com/photos/andresrueda/One of the most effective ways of improving website performance is to reduce the number of HTTP requests required for the page to fully load.

Typically you’ll have a bunch of small icons used for buttons, panels, links etc. in your CSS, as well as small background graphics here and there. These all add us to a significant number of HTTP requests, each one for only a couple of kilobytes of data.

“CSS Sprites” have often been used as a way of overcoming this issue, however two newer techniques (data URIs and MHTML)  are starting to become a more common way to address the problem.

The idea of data URIs and MHTML are the same – inline the image data into the actual CSS (or HTML) as base64-encoded text.

Data URIs are the preferred way of doing this, however they are not supported in older browsers, specifically IE6 and IE7. Also, IE7 on Windows Vista has problems, and I’ve talked about this in more detail recently. MHTML can be used reliably in IE6/7 on Windows XP, and so the filter will apply it for those clients.

Applying the Filter

I’ll cut right to the chase – here’s how you can apply this to your Java web-application in two simple steps:

  1. Add visural-common to your projects set of libraries, or adding it as a dependency (Maven). You will also need to add the YUI Jar to your project to use CSS / JS compression.
  2. Add com.visural.common.web.transform.ResourceTransformFilter to your your web.xml as a standard servlet filter, matched to the URLs of your resource file(s)

For example you might add the following XML your web.xml:

    <filter>
        <filter-name>ResourceTransformFilter</filter-name>
        <filter-class>com.visural.common.web.transform.ResourceTransformFilter</filter-class>
    </filter>
...
    <filter-mapping>
        <filter-name>ResourceTransformFilter</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>

And you’re done.

Default Configuation

The default configuration works as follows -

  • Supported features will be detected by using visural-common’s built in WebClient detector (based on User-Agent)
  • Resource URLs ending in .less will be compiled through the LessCSS compiler
  • Resources ending in .less or .css will have DataURIs or MHTML applied, but only if the client supports them, otherwise the original image references will be left untouched.
    By default, only images < 50kb in size will be inlined, any larger images will be left as url(..) references. This is configurable.
  • Resources ending in .less or .css will be compressed through the Yahoo YUI CSS Compressor
  • Resources ending in .js will be compressed through the Yahoo YUI Javascript Compressor

Transformed responses will be automatically cached, so any additional overhead to generate these responses will be a one-time only cost. Note that any response headers you set will also be cached, and the filter does not affect the detection and dynamic behaviour based on the client browser/OS.

Custom Configuation

You can customise the behaviour of the filter by extending the filter’s class com.visural.common.web.transform.ResourceTransformFilter and overriding the protected “over-ride points” to change the classes behaviour.

Use the source as the starting point.

Optional .html DataURI Conversion

The default configuration does not do auto-conversion of HTML <img> tags to DataURI’s.

The reason for this is, that it is not always possible to reliably detect HTML responses, without a performance cost, and additionally, the CPU, memory and bandwidth overhead to doing this can be undersirable.

Unlike .CSS files, HTML responses are typically dynamic and can’t be cached. Additionally, images in the response may be from external servers, and may be larger than the types of image typically referenced from CSS.

For this reason, you will need to evaluate your circumstances to determine if enabling HTML inlining is a good idea in your application.

If you do want to enable it, you could do so like in the following example -

import com.visural.common.web.client.WebClient;
import java.util.Arrays;
import java.util.List;
import javax.servlet.http.HttpServletRequest;

public class MyResourceTransformFilter extends com.visural.common.web.transform.ResourceTransformFilter {
    @Override
    protected List getTransforms(HttpServletRequest req, WebClient client) {
        String url = req.getRequestURI();
        String last = url.substring(url.lastIndexOf('/'));
        if (last.contains(".")) {
            return super.getTransforms(req, client);
        } else if (client.supportsDataUris()) {
            return Arrays.asList(Transform.HTML_DATAURI);
        } else {
            return null;
        }
    }
}

There are several other override points, included in the filter, so consult the source or drop me an email if you’d like to customise further.

Summary

So with this one simple addition we get framework-agnostic image inlining, compilation and compression which responds intelligently to the user’s choice of browser.

This has been live on onmydoorstep.com.au for a week or two is working out well thus far.

Great, I don’t use servlet filters!

There is also a command-line version of the DataURI and MHTML converters which you can apply to non-JVM based projects as part of a build-script or manual process.

If you’ve downloaded the commandline version you can run it by:

  java -cp visural-common.jar com.visural.common.web.css.CSSDataUri [input-file.css] [output-file.css]

This will convert the [input-file.css] and inline the image data into the output. Note that the relative URLs are assumed to be resolvable relative the location of the CSS on disk. Mostly this will work just fine, and it will even attempt to resolve http:// urls.

The one thing that won’t work with the command line version is root-relative urls, e.g. url(“/folder/myimage.png”), since there’s no way to know what the root is, in the context of the disk. I could fix this is a future version by adding another command line option, so if you need it, let me know.

Similarly, you can run the MHTML converter from the command-line as so:

  java -cp visural-common.jar com.visural.common.web.css.CSSMHTML [input-file.css] [output-file.css] [http://serverurl/for/css]

For this one, you need to supply the additional server URL for the CSS file. This is because MHTML file references need the full URL to the defining CSS file (relative URLs don’t work).

This is one of the reasons that I chose to implement the solution as a filter.

Anyhow, given the simplicity of this filter, there’s pretty much no reason not to use this in your Java web app and improve those YSlow / PageSpeed results and rating without very little effort :)

Posted in Design, Java, Software Engineering | Tagged automation, css, Java, javascript, open-source, tips, visural-common, web | 7 Comments

Thoughts on the WordPress & Thesis GPL hoo-haa

The blogosphere has been railing back and forth the past week about the assertion made by Matt Mullenweg that themes are covered by the GPL, as is WordPress.

This post is a pretty good jumping off point if you need to get up to speed.

It appears in the context of the for-pay Thesis theme which is the primary issue in the discussion, there are a few points worth pointing out -

  • The issue of whether it’s a derivative work is moot since it has been clearly noted that it contains code from the original WordPress base theme, ergo even by the legal definition (as compared with the GPL’s), it is a derivative work.
  • The GPL only covers code that is executed. This means that the Thesis developer is obligated to only distribute the .php source under the GPL, while the other theme assets may be proprietary.

So with that in mind, the real question is – does the GPL apply to a non derivative work (in the legal, copyright sense) that is executed in the same process as the core GPL WordPress code?

Yes it does.

At least, the GPL says it does – it’s yet to be tested in a meaningful way. So whether that’s a legally enforceable statement, remains to be seen, but the license is pretty clear on that matter.

My feelings about this issue in general are -

  • The key issue here is that WordPress is mostly an end-user product, and a very popular one. This means there are a lot of people in the WordPress community with no understanding of GPL or licensing in general. The GPL enforces Freedom, and that’s with a capital ‘F’ – but the community around WordPress isn’t necessarily unanimously aware of this goal.

    Actually, that’s precisely why Stallman wrote the GPL as such. To ensure that his vision of community (and its efforts) were maintained despite any community that latched on to it down the road.

  • People have a choice when they choose GPL licensed code.

    When you pick something that’s GPL’ed, you accept that other people have put a lot of effort into building it. You reciprocate that effort by providing any effort that you choose to add, to others. It’s a one good turn obliges another type philosophy.

    You may disagree with it – it goes too far!

    Well that’s no problem – just don’t use or modify GPL’ed software. There are plenty of other more permissive licenses.

I actually think this debate is a good thing for the WordPress’s community. A bunch of people who previously didn’t care about licensing and software freedom have now been pulled kicking and screaming into the debate.

And that can only be a good thing, if only a small step.

(For my general position on open-source licensing, have a read of my open-source page)

Posted in IT, Software Engineering | Tagged open-source, soap-box | Leave a comment

Detecting Which Browser In Java Servlet/Filter

The newest version of my open source tools project, visural-common – 0.4, includes a new feature the com.visural.web.client.WebClient class.

The purpose of this class is simple -

User-Agent string goes in -> Browser, version and platform comes out.

The “User-Agent” string is a HTTP header property that is sent as part of the HTTP request. All common web browsers (and other HTTP clients, such as search engine crawlers) send predictable user agent strings which allow detection of the browser and operating system being used.

This makes it very easy to add conditional code to your server-side logic (in servlet filters or otherwise), such that you can customise the web experience for each platform and browser.

The usage of the class is very simple -

import com.visural.web.client.*;

class MyFilter implements Filter {
    ...
    public void doFilter(ServletRequest sr, ServletResponse sr1, FilterChain fc)
            throws IOException, ServletException
    {
        HttpServletRequest req = (HttpServletRequest) sr;
        WebClient client = WebClient.detect(req);
        if (client.getUserAgent().equals(UserAgent.IE)) {
            // do something for IE only
        } else if (client.getUserAgent().equals(UserAgent.FIREFOX) &&
                 client.getPlatform().equals(Platform.LINUX)) {
            // etc.
        }
    }
    ...
}

Currently it supports the detection of the following browsers, including version number -

    IE,
    FIREFOX,
    CHROME,
    OPERA,
    SAFARI,
    GOOGLEBOT,
    YAHOO_SLURP,
    MSNBOT

And will detect the following operating systems -

    MACOSX,
    WIN95,
    WIN98,
    WINNT,
    WIN2K,
    WINXP,
    WINVISTA,
    WIN7,
    LINUX,
    IOS, (iPhone, iPad, iPod Touch)
    ANDROID

There are certainly other solutions for this, but I wanted a simple, elegant implementation, that is optimised for the typical sort of logic that you want this for on the server side.

You can download version 0.4 of visural-common at google code and get started right away. It is licensed under the Apache 2.0 License.

Posted in Java, Software Engineering | Tagged Java, open-source, visural-common | Leave a comment

Facelift time and on to WordPress 3

Well I finally upgraded to the new WordPress.

To be honest there’s not really a great deal I was interested in, although the new Twenty-Ten theme is nice.

I took the opportunity to the give the site a bit of a facelift, since it was starting to look a bit cluttered. The new theme actually allows for more content, and doesn’t feel so cramped.

I did consider for a moment, writing my own blog engine and moving the site on to Google App Engine, but then I thought of the 100 other things I’d rather be building, and sat down until the thought went away.

WordPress is pretty great for what it is. There are times when I would like to get under the covers and have a bit more control (without learning PHP), but for the most part it does everything I need and is very user friendly.

Maybe when I get that “spare time” I hear about, I’ll reconsider, but for now, I heart WP.

Posted in General | Tagged off-topic | Leave a comment