Brenmor uses ML.NET to improve patient experience & care quality

Brenmor Technologies partners with medical groups and health plan providers to create patient satisfaction solutions that improve patient care. They work closely with their clients to customize patient experience surveys that are a statistically reliable and cost-effective way for them to identify areas of strengths and weaknesses, then set improvement initiatives. These surveys often contain at least one free-form text response; during analysis, these responses are examined and funneled into a defined set of categories so that the clients can improve the visit experience and care quality by acting directly on patient feedback.

Business problem

Brenmor's solution provides deep insights and analytics that rely heavily on survey data from patients. The ability to analyze free-form patient text responses into multiple categories and into positive or negative sentiment is crucial to medical groups, health plans, and providers (that is, doctors) since these audiences are required to understand and create actionable plans based on weighted patient feedback.

By categorizing patient responses, different client organizations receive related responses to review and define quality initiatives. For example, comments that are categorized as Provider or Staff are given to the professionals who handle the one-on-one patient experience, while categories such as Appointments and Telephone are routed to the front office personnel.

Today, medical professionals require near real-time feedback on the quality of service provided by medical groups, offices, and providers, yet manual categorization is both slow and prone to inconsistencies. Thus, Brenmor decided to replace their monthly manual categorization process with a multiclass classification machine learning solution to quickly and accurately classify complex, free-form, patient text responses.

Why ML.NET?

The selection of ML.NET was an attractive solution given the existing investment Brenmor has with .NET solutions. In addition, a robust set of ML.NET code samples provided by Microsoft gave Brenmor a clear path to a working implementation. The ability to quickly and easily generate a machine learning model with AutoML and then leverage that model within their existing solutions allowed Brenmor to provide a to-market solution in a matter of weeks.

Since we began with ML.NET, Microsoft has rapidly evolved the ML.NET product. This has resulted in a significant increase in the speed and accuracy of categorizing complex, free-form, patient text responses. Our clients are thrilled."

Impact of ML.NET and AutoML

The ability to scale the classification of free-form patient text responses has been an enormous benefit for Brenmor. Previously, categorization was done once a month and took approximately 16 human hours. With ML.NET, this classification is done in real-time and provides immediate feedback to Brenmor's analytics clients, who can now spend less time analyzing and more time implementing solutions to the problems identified by the ML.NET model.

Another huge benefit that Brenmor has seen with using ML.NET is improved accuracy. Patient text responses often contain multiple topics. Before ML.NET, accuracy was determined manually by different individuals; this was error-prone, and multiclass classification with confidence levels was not possible. ML.NET's multiclass classification support has allowed Brenmor to increase classification accuracy from 50% to approximately 76% and provides the company's clients with confidence levels for the detection of multiple categories.

Additionally, by using AutoML with ML.NET, an understanding of the low-level nuances of machine learning algorithms and the extensive set of algorithm hyperparameters was not required. Developers at Brenmor no longer had to spend hours testing different algorithms and tweaking hyperparameters looking for the best model. This time-saving was of immense benefit to Brenmor as it allowed for an increased focus on improving the initial training data.

Because of the intelligence that AutoML provides, Microsoft has essentially provided a shrink-wrapped Data Scientist to users of ML.NET. This is an immense value to the .NET community!"

Solution architecture

Creating an application with ML.NET

The technical stack for Brenmor's survey categorization application is comprised of ML.NET, ASP.NET Core, Entity Framework Core, and SQL Server.

A great deal of up-front time was allocated to creating a clean and succinct set of training data, mostly based off raw survey responses. Now that the application executes in a production environment, when there are low confidence prediction results, these comments are reviewed and become additional training data . This workflow is a "rinse and repeat" process that injects additional intelligence and accuracy into the machine learning model. Updated machine learning models are versioned in source code control and are automatically deployed as part of regularly scheduled deployments in Azure DevOps.

Data processing

The initial training data, which consisted of around 3,000 rows of free-form comments, came from raw patient survey responses. All the data used by Brenmor was first HIPPA cleansed and then separated into succinct training data to more accurately weight comments with multiple categories. Once an initial, accurate set of training data existed and classification accuracy was established, the manual process decreased significantly.

Data transformations and machine learning algorithms

Brenmor used AutoML with ML.NET to automatically generate the data transformations and machine learning algorithms that resulted in the highest-quality model. Data transformations used include Map Value to Key and Featurize Text, and algorithms used include the One Versus All algorithm for multiclass classification (categorization of comments) and the L-BFGS Logistic Regression Binary algorithm for binary classification (positive vs. negative comments).

AutoML with ML.NET also generated the set of algorithm options below to further fine-tune the model for optimal model accuracy:

var trainer = mlContext.MulticlassClassification.Trainers.OneVersusAll(mlContext.BinaryClassification.Trainers.LbfgsLogisticRegression(new LbfgsLogisticRegressionBinaryTrainer.Options(){
    L2Regularization = 0.7512906f,
    L1Regularization = 0.4493316f,
    OptmizationTolerance = 0.0001f,
    HistorySize = 50,
    MaximumNumberOfIterations = 348607332,
    InitialWeightsDiameter = 0.1322905f,
    DenseOptimizer = false,
    LabelColumnName = "Category",
    FeatureColumnName = "Features",
    }),
    labelColumnName: "Category")
.Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel", "PredictedLabel"));

Today, patient satisfaction is a growing need for medical groups and health plans, and hearing directly from patients is critical. Accepting survey inputs and analyzing and providing classification of free-form text responses is complex and requires a high-degree of accuracy. By leveraging ML.NET and AutoML, Brenmor has been able to quickly and accurately provide their clients with additional insights and analytics related to the patient experience.