Getting Emotional with Affectiva, F#, and Emgu

I’ve been playing with Affectiva emotion, demographics, and face detection SDK, found it excellent, however, their sample gallery lacks a sample in F#! So here we are to correct that.

I just wanted a simple F# script that would let me take all kinds of the SDK options for a ride. The script itself is 130 lines. Out of that about 30 lines is just a boilerplate to load all the relevant libraries, setup the environment, etc.

Finally, here I am goofing off in front of my webcam.

Setup

Not much in terms of setup. So, yes, regular things for downloading/installing EmguCV, OpenCV, and installing Affectiva SDK.

Then all this needs to be reflected in the script:

open System

Environment.CurrentDirectory <- @"C:\Program Files\Affectiva\Affdex SDK\bin\release"
#r "../packages/EmguCV.3.1.0.1/lib/net30/Emgu.CV.UI.dll"
#r "../packages/EmguCV.3.1.0.1/lib/net30/Emgu.CV.UI.GL.dll"
#r "../packages/EmguCV.3.1.0.1/lib/net30/Emgu.CV.World.dll"
#r "../packages/OpenTK.1.1.2225.0/lib/net20/OpenTK.dll"
#r "System.Drawing.dll"
#r "System.Windows.Forms.dll"
#r @"C:\Program Files\Affectiva\Affdex SDK\bin\release\Affdex.dll"

open Affdex
open Emgu.CV
open Emgu.CV.CvEnum
open System.IO
open System.Collections.Generic
open Emgu.CV.UI
open Emgu.CV.Structure
open System.Drawing
open System.Linq
open System.Threading
open System.Diagnostics

let classifierPath = @"C:\Program Files\Affectiva\Affdex SDK\data"
let resources = Path.Combine(__SOURCE_DIRECTORY__, "Resources")

Just loading libraries, no big deal. Except we need to make sure Affdex.dll finds its dependencies, hence setting the current path at the beginning.

Initializing the Detector

let detector = new CameraDetector()
try
    detector.setClassifierPath(classifierPath)

    detector.setDetectAllEmotions(true);
    detector.setDetectAllExpressions(false);
    detector.setDetectAllEmojis(true);
    detector.setDetectGender(true);
    detector.setDetectGlasses(true);
    detector.setDetectEngagement(true);
    detector.setDetectValence(true);
    detector.setDetectAttention(true);
    detector.setDetectAge(true);

    detector.setImageListener(imageListener)
    detector.setProcessStatusListener(processStatusListener)

    detector.start();

    sw.Start()
    while not finished do
        Thread.Sleep(1000)
    sw.Stop()
finally
    detector.Dispose()

Here setDetectGlasses is my favorite. Check it out in the video.

I’m using CameraDetector to capture video from the webcam, if I needed to capture a file video I’d use VideoDetector. Setting properties is easy, albeit slightly confusing at first – all these subtle differences between valence and attention… It makes sense when you get used to it. My favorite is setDetectAllEmojis. The SDK comes with quite a few emojis that can be used to reflect what’s going on in the video.

The VideoDetector is set up in a similar way, except you also need to issue detector.``process``() to start running, camera detector does it automatically.

I would also like to use use instead of let to instantiate the disposable detector, but cannot do it in the script, so true to an instinct for plugging memory leaks before they spring, I wrapped it in the try..finally – not at all necessary in a script, and I don’t do it for EmguCV elements anyway. This is not a production code practice.

Fun Part: Processing Results

As processed frames start coming in, we hook up to the detector image listener (detector.setImageListener()) which will feed us images and all kinds of fun stats as they come in. Also, setProcessStatusListener will tell us when things are done or errors occur.

let imageListener = {
new ImageListener with
    member this.onImageCapture (frame : Affdex.Frame) = ()

    member this.onImageResults(faces : Dictionary<int, Face>, frame : Affdex.Frame) =
        let img = new Image<Rgb, byte>(frame.getWidth(), frame.getHeight());
        img.Bytes <- frame.getBGRByteArray()

        let faces = faces |> Seq.map (fun kvp -> kvp.Key, kvp.Value) |> Seq.toArray

        // draw tracking points
        faces.ToList().ForEach(fun (idx, face) ->
            let points = face.FeaturePoints |> Array.map featurePointToPoint
            let tl, br = Point(points.Min(fun p -> p.X), points.Min(fun p -> p.Y)), Point(points.Max(fun p -> p.X), points.Max(fun p -> p.Y))

            let rect = Rectangle(tl, Size(Point(br.X - tl.X, br.Y - tl.Y)))
            CvInvoke.Rectangle(img, rect, Bgr(Color.Green).MCvScalar, 2)

            // tracking points
            points.AsParallel().ForAll(fun p ->
                CvInvoke.Circle(img, p, 2, Bgr(Color.Red).MCvScalar, -1)
            )

            // age
            let age = string face.Appearance.Age
            CvInvoke.PutText(img, age, Point(rect.Right + 5, rect.Top), FontFace.HersheyComplex, 0.5, Bgr(Color.BlueViolet).MCvScalar, 1)

            // gender & appearance
            let gender = int face.Appearance.Gender

            // glasses
            let glasses = int face.Appearance.Glasses

            let appearanceFile = makeFileName gender glasses
            loadIntoImage img appearanceFile (Point(rect.Right + 5, rect.Top + 15)) iconSize

            // emoji
            if face.Emojis.dominantEmoji <> Affdex.Emoji.Unknown then
                let emofile = Path.ChangeExtension(Path.Combine(resources, (int >> string) face.Emojis.dominantEmoji), ".png")
                loadIntoImage img emofile (Point(rect.Left, rect.Top - 50)) iconSize
        )

        viewer.Image <- img.Mat
}

let processStatusListener = {
new ProcessStatusListener with
    member this.onProcessingException ex = ()
    member this.onProcessingFinished () = finished <- true
}

Nothing all that tricky about this code. F# object expression comes in handy for quickly creating an object that implements an interface. onImageResults is the key function here. It processes everything and sends it to the EmguCV handy viewer, which is launched at the start of script execution and runs asynchronously (I like how it doesn’t force me to modify its UI elements on the same thread that created it. This is totally cheating and smells buggy, but it’s so convenient for scripting!)

// Create our simplistic UI
let viewer = new ImageViewer()
let sd =
    async {
        return (viewer.ShowDialog()) |> ignore
        }
Async.Start(sd)

In the first couple of lines we transform the captured frame to EmguCV-understandable format. I am using Image rather than the recommended Mat class, because I want to splat emojis over the existing frames and as amazing as it is, the only way to do it that I know of in EmguCV is this counter-intuitive use of ROI. If anyone knows a better way of copying one image on top of another (should be easy, right?) please let me know.

The next few lines draw the statistics on the image: tracking points, emojis, and demographic data. Emojis are stored in files located in the resources path (see above, in my case I just copied them locally) with file names matching the SDK emoji codes. A simple function transforms these codes into file names. Finally, the modified frame is sent to the EmguCV viewer. That’s it!

let featurePointToPoint (fp : FeaturePoint) = Point(int fp.X, int fp.Y)
let mutable finished = false
let makeFileName i j = Path.ChangeExtension(Path.Combine(resources, String.Format("{0}{1}", i, j)), ".png")

Image Copy

The following two functions do the magic of copying emojis on top of the image:

let copyImage (src : Image<Bgr, byte>) (dest : Image<Rgb, byte>) (topLeft : Point) =
    let prevRoi = dest.ROI
    dest.ROI <- Rectangle(topLeft, src.Size)
    CvInvoke.cvCopy(src.Ptr, dest.Ptr, IntPtr.Zero)
    dest.ROI <- prevRoi

let loadIntoImage (img : Image<Rgb, byte>) (file : string) (topLeft : Point) (size : Size) =
        let src = new Image<Bgr, byte>(size)
        CvInvoke.Resize(new Image<Bgr, byte>(file), src, size)
        copyImage src img topLeft

copyImage first sets the ROI of the destination, then issues a legacy cvCopy call. It operates on pointer structures which is so ugly! There really should be a better way.