Microsoft Defender uses ML.NET to stop malware

Microsoft Defender Advanced Threat Protection (ATP) is Microsoft's unified security platform for intelligent preventative protection, post-breach detection, automated investigation, and response; it protects endpoints from cyber threats, detects advanced attacks and data breaches, automates security incidents, and improves security posture using a combination of the power of the cloud, behavior analytics, and machine learning.

Business problem

Microsoft Defender ATP processes trillions of signals every day and finds around 5 billion new threats every single month. These threats range from PDFs that try to phish user credentials and document files containing weaponized macros to password protected zip files containing polymorphic malware executables.

Being able to predict and stop these threats at first sight is critical for client safety and security. However, humans can only look at and hold so much information in their minds at once. To manually go through each of these attributes is a time-consuming task and is not quick enough to defend from incoming threats. With the scale of new threats seen each month, a manual process done by a human would never be scalable, making machine learning not just nice to have, but necessary for protecting users. Additionally, someone may look at a piece of malware and find a few attributes that make it malicious, but in reality, the malware may have hundreds of thousands of other attributes that indicate a threat that a human could not take the time to go figure out.

Machines, on the other hand, have a much larger capacity and a much faster response time; they can instantaneously look at all (possibly hundreds of thousands) of a potential threat's attributes and pick out all of the attributes that label the threat as malware. Then machines can use the attributes it found to discover new malware that a human may have not predicted using only a small number of attributes.

Thus, Microsoft Defender ATP decided to utilize machine learning and ML.NET (technically it uses a derivative of ML.NET called TLC, which has been the internal machine learning framework used at Microsoft for over 10 years), to improve real-time protection against malware so that they could more easily and accurately predict if signals are malicious and block incoming threats to keep their users' machines safe.

Impact of ML.NET

Microsoft Defender ATP uses classification algorithms to flag and surface threats, including previously unseen threats, that would otherwise remain unnoticed among the billions of normal events and the inability of first-generation sensors to react to unfamiliar and subtle stimuli. Microsoft Defender ATP's models optimize the use of the vast amounts of data and computational resources available to Microsoft Defender ATP. Additionally, based on Microsoft Defender ATP's analysis of actual alerts, the machine learning technologies being used are at least 20% more precise than manually crafted heuristics. ML.NET's supervised machine learning algorithms stop 35% of threats at the pre-breach level, which protects against malware from malicious URLs, email attachments, and other emerging threats.

Solution architecture

Data processing and feature selection

For Microsoft Defender ATP, it's critical to have a great set of labeled training data that includes clean data and malware in the training dataset so that it can mimic real-life scenarios and demonstrate what a real customer sees every day. On an average training cycle, a model in Microsoft Defender ATP may consume ~100 million rows of data with 190 thousand features each.

Feature selection is very important when training models that detect malware. There are two types of features that the researchers and machines look for: static file properties and behavioral components. Static file properties include things like if a file is signed or not, who signed the file, and various fuzzy hashes. Behavioral attributes include things like if the file is related to another file, whether this file injected was from another file, what IPs the file is connected to, and what changes the file made to the system. A training dataset can include thousands to millions of features.

Model training

Microsoft Defender ATP has a variety of models to serve multiple purposes. For instance, they have models that focus on first seen PE threats, macro threats, and script-based attacks. There are also models that are focused on the underlying data; for example, some models train exclusively on fuzzy hashes of files. On top of these, they have another layer of ensemble models that take signals from these individual classifiers to check once again if the system activity is malicious.

Microsoft Defender ATP trains these models daily on the latest data. Once the model is trained, it is saved, and the engineering pipeline loads it to the defender cloud infrastructure where it is queried by the clients.

Diversity [of models] is really key to having a tamper-resilient machine learning system."

Using ML.NET's algorithms, Microsoft Defender ATP has been able to produce a multitude of machine learning models to more effectively protect its users from potential threats, keeping half a billion people's computers safe from malware.