Camera Futura uses ML.NET to automate photo culling and organizing

Camera Futura is a company based in Geneva, Switzerland. The purpose of the company is to develop and commercialize breakthrough software for enthusiast and professional photographers.

The main product currently in production is Futura Photo, a desktop application that can automate photo culling (the process of selecting the best images from a photo shoot) and can organize photos before starting post-processing. Futura Photo acts as a 'smart and personalized assistant.'

Business problem

Photographers spend a lot of time shooting images, and many enjoy enhancing the photos through post-processing. However, several steps are needed after the shoot and before post-processing. These steps, which are culling photos and organizing photos for post-processing, are not as exciting as other parts of the photography process and can be very time-consuming.

Culling images simply means discarding low quality images. 'Low quality' means that the photo was not shot as intended due to technical, not artistic, reasons.

Organizing images involves moving images to the right folder. This can mean a few different things, such as choosing between RAW or JPG images for post-processing, stitching together images to create panoramas, and stacking images for macro- or astrophotography. When talking about time lapses, organizing means finding the correct photos (possibly hundreds of images for each time lapse) and uploading them to software that will create the time lapse video.

There is also an important task that combines culling and organizing: collecting series of similar images and choosing the best image from the group. Photographers tend to shoot many images of the same subject (from action sports to events to just images of their kids playing). Choosing the best few images, while extremely important, can also be quite frustrating when the photographer must go through dozens of such groups for each shoot.

Futura Photo, which is mainly focused on helping enthusiast photographers who shoot around 10,000 to 25,000 images per year, is providing a unique experience to help solve the frustrations with image culling and organizing. While the automation of photo culling and organizing is not a new concept, the problem becomes particularly complex when you go beyond the basics, like trying to find totally blurred images or grouping together extremely similar images.

There are several reasons why this automation is not straightforward; there are many questions that a photographer must answer before starting photo post-processing, such as:

• What does a sharp image really mean?
• What are the reasons (good or bad) why a photographer shoots many similar images?
• When a photographer shoots RAW + JPG, when is the RAW file needed instead of the JPG file?
• Why should an image with a human face having eyes closed be discarded, or should it be post processed as something done on purpose?
• When a photographer shoots similar images, are they stack members for focus stacking (macro), HDR (landscapes), multi-exposure purpose (creative), or just a burst of images (sports)?

The list of these types of questions is quite lengthy, but answering them is something that Camera Futura wants to do with machine learning. A lot of added value comes from a 'Smart Assistant' that is customized for the photographer's needs and that can replace some steps of the workflow done manually by the photographer, but this is not possible without the help of machine learning. So, Futura Photo decided to use image classification to help photographers with the photo culling and organization process, including selecting the best image from a group.

Why ML.NET?

There are several reasons why Camera Futura chose ML.NET to power the machine learning features in their Futura Photo software.

First, Camera Futura uses .NET to build their applications, and using ML.NET has allowed the company to stay in the .NET ecosystem without having to code in another language or integrate with non-.NET frameworks. This has eliminated overhead in terms of learning a new language as well as complexity and potential performance hits from the integration of non-.NET tech into a .NET application.

Additionally, ML.NET Model Builder in Visual Studio made it extremely easy to run image classification scenarios, both for prototyping the initial models as well as training the final models that are currently being used in production. ML.NET itself also offered everything the company needed in terms of model availability, customization, and overall performance.

Impact of ML.NET

While ML.NET enabled Camera Futura to build, train, and deploy a production-level machine learning model quickly, the framework has also opened up new opportunities for the company. For a start-up, being able to do everything in .NET is a great advantage and allows the company to develop features that would be impossible to develop if it required the company to use several different technologies. For Camera Futura, ML.NET is both an enabler and a productivity tool.

Without ML.NET, delivering the first model production-ready would have needed several more months."

Solution architecture

Futura Photo is currently a WPF desktop application on .NET Framework 4.6.1 (with plans to migrate to .NET Core by the end of 2021).

How it works

Using Futura Photo is straightforward: photographers configure rules, upload images, check the results, and apply them to move photos to the right folder.

Users navigate through three main windows. The first window manages and configures the rules to be applied, displays the main gallery of images, shows progress during the analysis, and displays the results for each photo, as shown in the following figure:

When the analysis is done, a second window can be opened to check the results by rule:

A third window can be opened as needed for a full screen photo viewer to dive deeper into the results of each photo:

Data and training

Currently, Futura Photo uses company images and data from Camera Futura for training. The size of the dataset varies per model, but the company typically trains locally with 10,000-50,000 images per model After training and evaluating their models, they deploy the models along with the WPF application.

There are currently four models in production, three of which were built using ML.NET Model Builder's image classification scenario in Visual Studio. The last model, which is a clustering model, was built with the ML.NET API.

The image classification scenarios are used for deciding whether an image has or doesn't have a certain feature. This feature can be anything that cannot formally be described through a mathematical model.

For instance, one of the models can look at the white balance feature to help determine if an image has the right white balance or not, and why (too green, too yellow, etc.).

In the following example, the white balance has been well defined by the camera:

Futura Photo consumes the ML.NET model that calculates the white balance (as seen in the diagram above) by using code similar to the following code snippet:

// Method using ML to define whether the camera's AWB
// (Automatic White Balance) estimation is accurate
private Photo CalculationAWB(Photo photo)
{
    Photo _photo = photo;

    try
    {
        // Create single instance of sample data from first line of
        // dataset for model input
        ModelInput inputData = new ModelInput()
        {
            // FileNameImageLight: path for ad-hoc thumbnail of the
            // Photo source object (JPG or RAW file)
            ImageSource = _photo.FileNameImageLight
        };

        // WB_ML_PC: % of likelihood of AWB being either 'OK'
        // (well calculated), cold, tint green, tint purple, or warm
        // AWB's Accuracy Result ("OK"): predictionResult.Score[1]
        // Bias: predictionResult.Score[0] for Cold, [2] for Tint Green,
        // [3] for Tint Purple, and [4] for warm
        for (int i = 0; i < 5; i++){
            _photo.WB_ML_PC[i] = predictionResult.Score[i]) * 100;
        }
        catch (Exception ex)
        {
            log.Error(ex, "Throughout ML_IsAWB_OK");
        }

        return _photo;
    }
}

Another model can look at the sharpness of an image. In theory, there are a lot of mathematical methods that are able to define whether an image is sharp, but most, if not all methods, fail to detect the mix of motion blurriness and out of focus blurriness. Even more important, these methods rely on thresholding, which is fine when you compare similar images but not accurate enough to decide whether one singular image is sharp enough. These methods fail to detect a given level of sharpness due to many corner cases that impact the value to be compared to the threshold. Using image classification models trained with ML.NET, Futura Photo can decide with extremely low false positive and negative rates if an image is sharp enough.

The last model is a clustering model which uses unsupervised machine learning to group images into a 'Pass' group (images that have requested features) or into a 'Fail' group (meaning the image does not have the requested features).

The Future of Camera Futura

Camera Futura's vision is to have a cloud-based solution which includes loading images via a mobile app, running the software on Azure, and finalizing the photo culling in the desktop app. This means that all the photos will be ready for post-processing as soon as the photographer is back at their desktop computer after the photoshoot.

The company is currently developing several other models, including for image classification, object detection, and regression scenarios.

While end-users of Futura Photo currently only consume the models for photo culling or for organizing their photo shoot before post-processing, Camera Futura would also like to experiment with allowing customers to train models locally with their own data for even better models.