A requirement came up on a recent project to automatically crop displayed profile images of people to just the “face” area for a thumbnail.
This seems like a job for a face detection algorithm. Searching for appropriate open-source Java implementations didn’t yield too many results, however I was successful with JJIL – Jon’s Java Imaging Library, which is open sourced under the LGPL licence.
JJIL is targeted at Java ME / Android platforms and doesn’t have much documentation or a particularly intuitive API (not complaining, as clearly some stellar work has gone into it, kudos to Jon Webb, it’s creator).
In the end I muddled through, detecting faces in an image in a standard Java project, but given it took me a while to get everything working, I thought I’d write a quick guide to help out others that are trying to achieve similar results.
Getting the Right JARs
First things first, I had trouble getting the published JAR files to work happily together. There seems to be some sort of version mismatch issue between the core and J2SE versions.
So I built my own copy – you can download it here – JJIL-visural-build-20110112.zip
This is a build of the current trunk JJIL code, and the Java SE additions (jjil-j2se).
This build is guaranteed to work with the code examples below.
Basic Process
I’m going to try to explain the basic process of detecting the faces in terms of input and output data.
The key file provided by JJIL for easy face (and other body part) detection is Gray8DetectHaarMultiScale.java
This operation is applied to an 8-bit greyscale input image, in combination with a pre-defined Haar Cascade profile. The profile determines which areas of the image are “detected”. So you would want (for example) a profile to detect the frontal face features. JJIL provides several profiles out of the box.
The output image is a mask of the area of image where faces are detected (white) and the areas where no face was detected (black). This isn’t tremendously useful, as we’d usually rather just have the rectangular areas in coordinate form – I’ll address this later, after walking through the process.
1. Read an image from disk (.JPG, etc.)
2. Convert it into a jjil.core.Image
3. Generally we’ll have an RGB image (colored image) and so need to convert it to 8-bit greyscale, which is what the Gray8DetectHaarMultiScale class requires.
4. Create a new instance of Gray8DetectHaarMultiScale with the Haar profile for detecting faces (or other body part if that’s what you’re looking for).
5. Apply Gray8DetectHaarMultiScale to our 8-bit grey image.
6. Retrieve result from Gray8DetectHaarMultiScale.
The figure below shows, the source image, overlayed with the resulting mask, from step #6
So as you can see, the masks correctly identify the faces the two people in the image.
The Code
Here’s a small Java class that demonstrates how to read an image and apply the process described above.
Note: all the code in the article can be downloaded as a full project at the end of the article.
package jjilexample;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.InputStream;
import javax.imageio.ImageIO;
import jjil.algorithm.Gray8Rgb;
import jjil.algorithm.RgbAvgGray;
import jjil.core.Image;
import jjil.core.RgbImage;
import jjil.j2se.RgbImageJ2se;
import jjil.algorithm.Gray8DetectHaarMultiScale;
public class Main {
public static void findFaces(BufferedImage bi, int minScale, int maxScale, File output) {
try {
// step #2 - convert BufferedImage to JJIL Image
RgbImage im = RgbImageJ2se.toRgbImage(bi);
// step #3 - convert image to greyscale 8-bits
RgbAvgGray toGray = new RgbAvgGray();
toGray.push(im);
// step #4 - initialise face detector with correct Haar profile
InputStream is = Main.class.getResourceAsStream("/jjilexample/haar/HCSB.txt");
Gray8DetectHaarMultiScale detectHaar = new Gray8DetectHaarMultiScale(is, minScale, maxScale);
// step #5 - apply face detector to grayscale image
detectHaar.push(toGray.getFront());
// step #6 - retrieve resulting face detection mask
Image i = detectHaar.getFront();
// finally convert back to RGB image to write out to .jpg file
Gray8Rgb g2rgb = new Gray8Rgb();
g2rgb.push(i);
RgbImageJ2se conv = new RgbImageJ2se();
conv.toFile((RgbImage)g2rgb.getFront(), output.getCanonicalPath());
} catch (Throwable e) {
throw new IllegalStateException(e);
}
}
public static void main(String[] args) throws Exception {
// step #1 - read source image
BufferedImage bi = ImageIO.read(Main.class.getResourceAsStream("test.jpg"));
// onto following steps...
findFaces(bi, 1, 40, new File("c:/Temp/result.jpg")); // change as needed
}
}
Getting Face Rectangles Instead
It would be more useful in many cases, to get a collection of Rectangles, in coordinate form, instead of an image mask.
There is a version of DetectHaarMultiScale in the JJIL project SVN, which implements a “getRectangles” method to retrieve this data. Unfortunately the source in incompatible with the rest of the library in SVN, so it may be WIP or an abandoned version of the code.
To get around this, I created my own version of Gray8DetectHaarMultiScale, which you can download here – Gray8DetectHaarMultiScale
Here are the important changes below -
public void push(Image image) throws jjil.core.Error {
pushAndReturn(image);
}
public List pushAndReturn(Image image) throws jjil.core.Error
{
List result = new ArrayList();
....
if (hcc.eval(imSub)) {
// Found something.
nxLastFound = imSub.getXOffset();
nyLastFound = imSub.getYOffset();
// assign Byte.MAX_VALUE to the feature area so we don't
// search it again
result.add(new Rect(nxLastFound*nScale, nyLastFound*nScale,
this.hcc.getWidth()*nScale,
this.hcc.getHeight()*nScale));
Gray8Rect gr = new Gray8Rect(nxLastFound,
nyLastFound,
this.hcc.getWidth(),
this.hcc.getHeight(),
Byte.MAX_VALUE);
gr.push(imMask);
imMask = (Gray8Image) gr.getFront();
}
....
return result;
}
So now we can call “pushAndReturn(…)” instead of just push() to apply the process to our image, and get a List back of the detected faces. Perfect!
Using this here is a version of the example code above which prints out the rectangles where faces were detected -
package jjilexample;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.InputStream;
import java.util.List;
import javax.imageio.ImageIO;
import jjil.algorithm.Gray8Rgb;
import jjil.algorithm.RgbAvgGray;
import jjil.core.Image;
import jjil.core.Rect;
import jjil.core.RgbImage;
import jjil.j2se.RgbImageJ2se;
public class Main {
public static void findFaces(BufferedImage bi, int minScale, int maxScale, File output) {
try {
InputStream is = Main.class.getResourceAsStream("/jjilexample/haar/HCSB.txt");
Gray8DetectHaarMultiScale detectHaar = new Gray8DetectHaarMultiScale(is, minScale, maxScale);
RgbImage im = RgbImageJ2se.toRgbImage(bi);
RgbAvgGray toGray = new RgbAvgGray();
toGray.push(im);
List results = detectHaar.pushAndReturn(toGray.getFront());
System.out.println("Found "+results.size()+" faces");
Image i = detectHaar.getFront();
Gray8Rgb g2rgb = new Gray8Rgb();
g2rgb.push(i);
RgbImageJ2se conv = new RgbImageJ2se();
conv.toFile((RgbImage)g2rgb.getFront(), output.getCanonicalPath());
} catch (Throwable e) {
throw new IllegalStateException(e);
}
}
public static void main(String[] args) throws Exception {
BufferedImage bi = ImageIO.read(Main.class.getResourceAsStream("test.jpg"));
findFaces(bi, 1, 40, new File("c:/Temp/result.jpg")); // change as needed
}
}
If you run this, you will note that it actually detects 3 faces in the image. This is common, as the way the Haar algorithm works, is by resizing the image to different scales and running a fixed size matrix over the image. It is possible for the same face to be detected at different scales and so you end up with rectangles within rectangles. It is a pretty trivial matter to remove these “extra” rectangles though by just checking if they are fully contained by another and ignoring them accordingly.
Download the Project
To save you some time, here’s the full example as a Netbeans project that you can download and run, play with, etc. Have fun!
JJILExample
Photo used in this article courtesy of Gill Penny.
