.NET for Apache Spark Tutorial | Get started in 10 minutes

Intro

Purpose

Setup .NET for Apache Spark on your machine and build your first application.

Prerequisites

Linux or Windows operating system.

Time to Complete

10 minutes

Scenario

Use Apache Spark to count the number of times each word appears across a collection sentences.

Install .NET

To start building .NET apps you just need to download and install the .NET SDK (Software Development Kit).

Download .NET SDK (64-bit)
32-bit download

Linux Distribution

Not running x64? You'll need to download the .NET Core binaries instead.

Enable .NET channel

In order to install .NET Core from Red Hat on RHEL, you first need to register using the Red Hat Subscription Manager. If this has not been done on your system, or if you are unsure, see the Red Hat Getting Started Guide.

Install the .NET SDK

After registering with the Subscription Manager and enabling the .NET Core channel, you are ready to install and enable the .NET SDK.

In your terminal, run the following commands:

Terminal

yum install rh-dotnet22 -y
scl enable rh-dotnet22 bash

Register Microsoft key and feed

Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.

Open a terminal and run the following commands:

Terminal

wget -q https://packages.microsoft.com/config/ubuntu/19.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb

Install the .NET SDK

Update the products available for installation, then install the .NET SDK.

In your terminal, run the following commands:

Terminal

sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-2.2

If you receive an error message similar to Unable to locate package dotnet-sdk-2.2, run the following commands.

Terminal

sudo dpkg --purge packages-microsoft-prod && sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update
sudo apt-get install dotnet-sdk-2.2

If that doesn't work, you can run a manual install with the following commands.

Terminal

sudo apt-get install -y gpg
wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
sudo mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/ubuntu/19.04/prod.list
sudo mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
sudo chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
sudo chown root:root /etc/apt/sources.list.d/microsoft-prod.list
sudo apt-get install -y apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-2.2

Register Microsoft key and feed

Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.

Open a terminal and run the following commands:

Terminal

wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb

Install the .NET SDK

Update the products available for installation, then install the .NET SDK.

In your terminal, run the following commands:

Terminal

sudo add-apt-repository universe
sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-2.2

If you receive an error message similar to Unable to locate package dotnet-sdk-2.2, run the following commands.

Terminal

sudo dpkg --purge packages-microsoft-prod && sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update
sudo apt-get install dotnet-sdk-2.2

If that doesn't work, you can run a manual install with the following commands.

Terminal

sudo apt-get install -y gpg
wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
sudo mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/ubuntu/18.04/prod.list
sudo mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
sudo chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
sudo chown root:root /etc/apt/sources.list.d/microsoft-prod.list
sudo apt-get install -y apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-2.2

Register Microsoft key and feed

Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.

Open a terminal and run the following commands:

Terminal

wget -q https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb

Install the .NET SDK

Update the products available for installation, then install the .NET SDK.

In your terminal, run the following commands:

Terminal

sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-2.2

If you later perform an in-place upgrade, you may get an error message similar to Unable to locate package dotnet-sdk-2.2 when trying to install/update .NET.

See Ubuntu 19.04 install instructions to learn how to fix this issue.

Register Microsoft key and feed

Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.

Open a terminal and run the following commands:

Terminal

wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
sudo mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/debian/9/prod.list
sudo mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
sudo chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
sudo chown root:root /etc/apt/sources.list.d/microsoft-prod.list

Install the .NET SDK

Update the products available for installation, then install the .NET SDK.

In your terminal, run the following commands:

Terminal

sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-2.2

Add the dotnet product feed

Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.

Open a terminal and run the following commands:

Terminal

sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
sudo wget -q -O /etc/yum.repos.d/microsoft-prod.repo https://packages.microsoft.com/config/fedora/27/prod.repo

Install the .NET SDK

Update the products available for installation, then install the .NET SDK.

In your terminal, run the following commands:

Terminal

sudo dnf update
sudo dnf install dotnet-sdk-2.2

Add the dotnet product feed

Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.

Open a terminal and run the following commands:

Terminal

sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
sudo wget -q -O /etc/yum.repos.d/microsoft-prod.repo https://packages.microsoft.com/config/fedora/27/prod.repo

Install the .NET SDK

Update the products available for installation, then install the .NET SDK.

In your terminal, run the following commands:

Terminal

sudo dnf update
sudo dnf install dotnet-sdk-2.2

Add the dotnet product feed

Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.

Open a terminal and run the following commands:

Terminal

sudo rpm -Uvh https://packages.microsoft.com/config/rhel/7/packages-microsoft-prod.rpm

Install the .NET SDK

Update the products available for installation, then install the .NET SDK.

In your terminal, run the following commands:

Terminal

sudo yum update
sudo yum install dotnet-sdk-2.2

Add the dotnet product feed

Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.

Open a terminal and run the following commands:

Terminal

sudo zypper install libicu
sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
wget -q https://packages.microsoft.com/config/opensuse/42.2/prod.repo
sudo mv prod.repo /etc/zypp/repos.d/microsoft-prod.repo
sudo chown root:root /etc/zypp/repos.d/microsoft-prod.repo

Install the .NET SDK

Update the products available for installation, then install the .NET SDK.

In your terminal, run the following commands:

Terminal

sudo zypper update
sudo zypper install dotnet-sdk-2.2

Add the dotnet product feed

Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.

Open a terminal and run the following commands:

Terminal

sudo rpm -Uvh https://packages.microsoft.com/config/sles/12/packages-microsoft-prod.rpm

Install the .NET SDK

Update the products available for installation, then install the .NET SDK.

In your terminal, run the following commands:

Terminal

sudo zypper update
sudo zypper install dotnet-sdk-2.2

Check everything installed correctly

Once you've installed, open a new command prompt and run the following command:

Command prompt

dotnet

If the command runs, printing out information about how to use dotnet, you're good to go.

If you receive a 'dotnet' is not recognized as an internal or external command error, make sure you opened a new command prompt. If you can't resolve the issue, use the I ran into an issue button to get help fixing the problem.

Install Prerequisites

Install Java

Apache Spark requires the Java SE Development Kit (JDK) 8.1.

Visit Java SE Development Kit 8 Downloads.
Locate the section titled Java SE Development Kit 8u211.
Review the License agreement, and accept if you agree.
Click the download for either Windows x64 or Windows x86, depending on your operating system.
Oracle requires an account to download the JDK. Complete the account registration process if prompted.
Once the download completes, run the installer using the default settings.

Install 7-zip

Apache Spark is downloaded as a compressed .tgz file. You'll need 7-zip to extract the file.

If you already have an alternative extraction program installed, you can use that instead.

Visit 7-Zip downloads.
In the first table on the page, select either the 32-bit x86 or 64-bit x64 download, depending on your operating system.
When the download completes, run the installer.

Install Apache Spark

Download and install Apache Spark:

Download Apache Spark 2.4.1
Extract the nested .tar file:
- Locate the spark-2.4.1-bin-hadoop2.7.tgz file that you downloaded.
- Right click on the file and select 7-Zip -> Extract here.
- spark-2.4.1-bin-hadoop2.7.tar is created alongside the .tgz file you downloaded.
Extract Apache Spark files:
- Right click on spark-2.4.1-bin-hadoop2.7.tar and select 7-Zip -> Extract files....
- Enter C:\bin in the Extract to field.
- Uncheck the checkbox below the Extract to field.
- click OK.
- The Apache Spark files are extracted to C:\bin\spark-2.4.1-bin-hadoop2.7\

Set environment variables

Run the following commands to set several environment variables used to locate Apache Spark:

Command prompt

setx PATH "C:\bin\spark-2.4.1-bin-hadoop2.7\bin;%PATH%"
setx HADOOP_HOME C:\bin\spark-2.4.1-bin-hadoop2.7\
setx SPARK_HOME C:\bin\spark-2.4.1-bin-hadoop2.7\

Check everything installed correctly

Once you've installed, open a new command prompt and run the following command:

Command prompt

spark-submit --version

If the command runs, printing our version information, you're good to go.

If you receive a 'spark-submit' is not recognized as an internal or external command error, make sure you opened a new command prompt. If you can't resolve the issue, use the I ran into an issue button to get help fixing the problem.

Install .NET for Apache Spark

Download and install .NET for Apache Spark:

Download .NET for Apache Spark
Locate the Microsoft.Spark.Worker.netcoreapp2.1.win-x64-0.4.0.zip file that you downloaded.
Right click and select 7-Zip -> Extract files....
Enter C:\bin in the Extract to field.
Uncheck the checkbox below the Extract to field.
click OK.

Install WinUtils

.NET for Apache Spark requires WinUtils to be installed alongside Apache Spark.

Download winutils.exe
Once winutils.exe downloads, copy it into C:\bin\spark-2.4.1-bin-hadoop2.7\bin.

Set DOTNET_WORKER_DIR

Run the following command to set the DOTNET_WORKER_DIR Environment Variable. This is used by .NET apps to locate .NET for Apache Spark.

Command prompt

setx DOTNET_WORKER_DIR "C:\bin\Microsoft.Spark.Worker-0.4.0"

Create your app

In your command prompt, run the following commands:

Command prompt

dotnet new console -o mySparkApp
cd mySparkApp

The dotnet command creates a new application of type console for you. The -o parameter creates a directory named mySparkApp where your app is stored, and populates it with the required files. The cd mySparkApp command puts you into the newly created app directory.

Install NuGet package

To use .NET for Apache Spark in an app, you need to install the Microsoft.Spark package.

In your command prompt, run the following command:

Command prompt

dotnet add package Microsoft.Spark --version 0.4.0

Add data file

Your app will be processing a file containing lines of text.

Create an input.txt file in your mySparkApp directory, containing the following text.

input.txt

Hello World
This .NET app uses .NET for Apache Spark
This .NET app counts words with Apache Spark

Code your app

Open Program.cs in any text editor and replace all of the code with the following:

Program.cs

using Microsoft.Spark.Sql;

namespace MySparkApp
{
    class Program
    {
        static void Main(string[] args)
        {
            // Create a Spark session
            var spark = SparkSession
                .Builder()
                .AppName("word_count_sample")
                .GetOrCreate();

            // Create initial DataFrame
            DataFrame dataFrame = spark.Read().Text("input.txt");

            // Count words
            var words = dataFrame
                .Select(Functions.Split(Functions.Col("value"), " ").Alias("words"))
                .Select(Functions.Explode(Functions.Col("words"))
                .Alias("word"))
                .GroupBy("word")
                .Count()
                .OrderBy(Functions.Col("count").Desc());

            // Show results
            words.Show();
        }
    }
}

Run your app

Run the following command to build you application:

Command prompt

dotnet build

Run the folliwng command to submit you application to run on Apache Spark:

Command prompt

spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin\Debug\netcoreapp2.2\microsoft-spark-2.4.x-0.4.0.jar dotnet bin\Debug\netcoreapp2.2\mySparkApp.dll

When your app runs, you will see the data written to the console.

Command prompt

... logging ...

+------+-----+
|  word|count|
+------+-----+
|  .NET|    3|
|Apache|    2|
|   app|    2|
|  This|    2|
| Spark|    2|
| World|    1|
|counts|    1|
|   for|    1|
| words|    1|
|  with|    1|
| Hello|    1|
|  uses|    1|
+------+-----+

... logging ...

Congratulations, you've created your first .NET for Apache Spark application!

.NET for Apache Spark Tutorial - Get started in 10 minutes

Intro

Purpose

Prerequisites

Time to Complete

Scenario

Install .NET

Enable .NET channel

Install the .NET SDK

Register Microsoft key and feed

Install the .NET SDK

Register Microsoft key and feed

Install the .NET SDK

Register Microsoft key and feed

Install the .NET SDK

Register Microsoft key and feed

Install the .NET SDK

Add the dotnet product feed

Install the .NET SDK

Add the dotnet product feed

Install the .NET SDK

Add the dotnet product feed

Install the .NET SDK

Add the dotnet product feed

Install the .NET SDK

Add the dotnet product feed

Install the .NET SDK

Check everything installed correctly

Install Prerequisites

Install Java

Install 7-zip

Install Java

Ensure you are using Java 8

Check everything installed correctly

Install Apache Spark

Set environment variables

Check everything installed correctly

Download Apache Spark

Extract Apache Spark

Run spark-shell

Install .NET for Apache Spark

Install WinUtils

Set DOTNET_WORKER_DIR

Download .NET for Apache Spark

Extract Apache Spark

Set DOTNET_WORKER_DIR

Create your app

Install NuGet package

Add data file

Code your app

Run your app