This site uses cookies for analytics, personalized content and ads. By continuing to browse this site, you agree to this use. Learn more
Setup .NET for Apache Spark on your machine and build your first application.
Linux or Windows operating system.
10 minutes
Use Apache Spark to count the number of times each word appears across a collection sentences.
To start building .NET apps you just need to download and install the .NET SDK (Software Development Kit).
Not running x64? You'll need to download the .NET Core binaries instead.
In order to install .NET Core from Red Hat on RHEL, you first need to register using the Red Hat Subscription Manager. If this has not been done on your system, or if you are unsure, see the Red Hat Product Documentation for .NET Core.
After registering with the Subscription Manager, you are ready to install and enable the .NET SDK.
In your terminal, run the following commands:
yum install rh-dotnet30 -y
scl enable rh-dotnet30 bashBefore installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.
Open a terminal and run the following commands:
wget -q https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.debUpdate the products available for installation, then install the .NET SDK.
In your terminal, run the following commands:
sudo apt-get update
sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-3.0If you receive an error message similar to Unable to locate package dotnet-sdk-3.0, run the following commands.
sudo dpkg --purge packages-microsoft-prod && sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update
sudo apt-get install dotnet-sdk-3.0If that doesn't work, you can run a manual install with the following commands.
sudo apt-get install -y gpg
wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
sudo mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/ubuntu/16.04/prod.list
sudo mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
sudo chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
sudo chown root:root /etc/apt/sources.list.d/microsoft-prod.list
sudo apt-get install -y apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-3.0Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.
Open a terminal and run the following commands:
wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.debUpdate the products available for installation, then install the .NET SDK.
In your terminal, run the following commands:
sudo add-apt-repository universe
sudo apt-get update
sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-3.0If you receive an error message similar to Unable to locate package dotnet-sdk-3.0, run the following commands.
sudo dpkg --purge packages-microsoft-prod && sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update
sudo apt-get install dotnet-sdk-3.0If that doesn't work, you can run a manual install with the following commands.
sudo apt-get install -y gpg
wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
sudo mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/ubuntu/18.04/prod.list
sudo mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
sudo chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
sudo chown root:root /etc/apt/sources.list.d/microsoft-prod.list
sudo apt-get install -y apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-3.0Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.
Open a terminal and run the following commands:
wget -q https://packages.microsoft.com/config/ubuntu/19.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.debUpdate the products available for installation, then install the .NET SDK.
In your terminal, run the following commands:
sudo apt-get update
sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-3.0If you receive an error message similar to Unable to locate package dotnet-sdk-3.0, run the following commands.
sudo dpkg --purge packages-microsoft-prod && sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update
sudo apt-get install dotnet-sdk-3.0If that doesn't work, you can run a manual install with the following commands.
sudo apt-get install -y gpg
wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
sudo mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/ubuntu/19.04/prod.list
sudo mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
sudo chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
sudo chown root:root /etc/apt/sources.list.d/microsoft-prod.list
sudo apt-get install -y apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-3.0Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.
Open a terminal and run the following commands:
wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
sudo mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/debian/9/prod.list
sudo mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
sudo chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
sudo chown root:root /etc/apt/sources.list.d/microsoft-prod.listUpdate the products available for installation, then install the .NET SDK.
In your terminal, run the following commands:
sudo apt-get update
sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-3.0Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.
Open a terminal and run the following commands:
wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
sudo mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/debian/10/prod.list
sudo mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
sudo chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
sudo chown root:root /etc/apt/sources.list.d/microsoft-prod.listUpdate the products available for installation, then install the .NET SDK.
In your terminal, run the following commands:
sudo apt-get update
sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-3.0Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.
Open a terminal and run the following commands:
sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
sudo wget -q -O /etc/yum.repos.d/microsoft-prod.repo https://packages.microsoft.com/config/fedora/29/prod.repoUpdate the products available for installation, then install the .NET SDK.
In your terminal, run the following commands:
sudo dnf install dotnet-sdk-3.0Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.
Open a terminal and run the following commands:
sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
sudo wget -q -O /etc/yum.repos.d/microsoft-prod.repo https://packages.microsoft.com/config/fedora/30/prod.repoUpdate the products available for installation, then install the .NET SDK.
In your terminal, run the following commands:
sudo dnf install dotnet-sdk-3.0Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.
Open a terminal and run the following commands:
sudo rpm -Uvh https://packages.microsoft.com/config/centos/7/packages-microsoft-prod.rpmUpdate the products available for installation, then install the .NET SDK.
In your terminal, run the following commands:
sudo yum install dotnet-sdk-3.0Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.
Open a terminal and run the following commands:
sudo zypper install libicu
sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
wget -q https://packages.microsoft.com/config/opensuse/42.3/prod.repo
sudo mv prod.repo /etc/zypp/repos.d/microsoft-prod.repo
sudo chown root:root /etc/zypp/repos.d/microsoft-prod.repoUpdate the products available for installation, then install the .NET SDK.
In your terminal, run the following commands:
sudo zypper install dotnet-sdk-3.0Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.
Open a terminal and run the following commands:
sudo zypper install libicu
sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
wget -q https://packages.microsoft.com/config/opensuse/15/prod.repo
sudo mv prod.repo /etc/zypp/repos.d/microsoft-prod.repo
sudo chown root:root /etc/zypp/repos.d/microsoft-prod.repoUpdate the products available for installation, then install the .NET SDK.
In your terminal, run the following commands:
sudo zypper install dotnet-sdk-3.0Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.
Open a terminal and run the following commands:
sudo rpm -Uvh https://packages.microsoft.com/config/sles/12/packages-microsoft-prod.rpmUpdate the products available for installation, then install the .NET SDK.
In your terminal, run the following commands:
sudo zypper install dotnet-sdk-3.0Before installing .NET, you'll need to register the Microsoft key, register the product repository, and install required dependencies. This only needs to be done once per machine.
Open a terminal and run the following commands:
sudo rpm -Uvh https://packages.microsoft.com/config/sles/15/packages-microsoft-prod.rpmUpdate the products available for installation, then install the .NET SDK.
In your terminal, run the following commands:
sudo zypper install dotnet-sdk-3.0Once you've installed, open a new command prompt and run the following command:
Once you've installed, open a new terminal and run the following command:
dotnetIf the command runs, printing out information about how to use dotnet, you're good to go.
If you receive a 'dotnet' is not recognized as an internal or external command error, make sure you opened a new command prompt. If you can't resolve the issue, use the I ran into an issue button to get help fixing the problem.
Apache Spark requires the Java SE Development Kit (JDK) 8.1.
Apache Spark is downloaded as a compressed .tgz file. You'll need 7-zip to extract the file.
If you already have an alternative extraction program installed, you can use that instead.
Apache Spark requires the Java Development Kit (JDK).
Open a terminal and run the following command:
sudo apt install openjdk-8-jdk openjdk-8-jreApache Spark requires Java 8. Run the following command and select Java 8 when prompted:
sudo update-alternatives --config javaOnce the previous command completes, run the following command:
java -versionIf the command runs, printing our version information, you're good to go.
If the command fails, and you can't resolve the issue, use the I ran into an issue button to get help fixing the problem.
Download and install Apache Spark:
Run the following commands to set several environment variables used to locate Apache Spark:
setx HADOOP_HOME C:\bin\spark-2.4.1-bin-hadoop2.7\
setx SPARK_HOME C:\bin\spark-2.4.1-bin-hadoop2.7\Once you've installed, open a new command prompt and run the following command:
%SPARK_HOME%\bin\spark-submit --versionIf the command runs, printing our version information, you're good to go.
If you receive a 'spark-submit' is not recognized as an internal or external command error, make sure you opened a new command prompt. If you can't resolve the issue, use the I ran into an issue button to get help fixing the problem.
Apache Spark is downloaded as a .tgz file.
Download Apache Spark 2.4.1In your terminal, move to the folder that contains the file you just downloaded then run the following command:
mkdir ~/bin
tar xvf spark-2.4.1-bin-hadoop2.7.tgz --directory ~/binRun the following command:
./spark-shellDownload and install .NET for Apache Spark:
.NET for Apache Spark requires WinUtils to be installed alongside Apache Spark.
Run the following command to set the DOTNET_WORKER_DIR Environment Variable. This is used by .NET apps to locate .NET for Apache Spark.
setx DOTNET_WORKER_DIR "C:\bin\Microsoft.Spark.Worker-0.6.0".NET for Apache Spark is downloaded as a .tgz file.
Download .NET for Apache SparkIn your terminal, move to the folder that contains the file you just downloaded then run the following command:
tar xvf Microsoft.Spark.Worker.netcoreapp2.1.linux-x64-0.6.0.tar.gz --directory ~/binRun the following command to set the DOTNET_WORKER_DIR Environment Variable. This is used by .NET apps to locate .NET for Apache Spark.
export DOTNET_WORKER_DIR="~/bin/Microsoft.Spark.Worker-0.6.0"In your command prompt, run the following commands:
In your terminal, run the following commands:
dotnet new console -o mySparkApp
cd mySparkAppThe dotnet command creates a new application of type console for you. The -o parameter creates a directory named mySparkApp where your app is stored, and populates it with the required files. The cd mySparkApp command puts you into the newly created app directory.
To use .NET for Apache Spark in an app, you need to install the Microsoft.Spark package.
In your command prompt, run the following command:
In your terminal, run the following command:
dotnet add package Microsoft.Spark --version 0.6.0Your app will be processing a file containing lines of text.
Create an input.txt file in your mySparkApp directory, containing the following text.
Hello World
This .NET app uses .NET for Apache Spark
This .NET app counts words with Apache SparkOpen Program.cs in any text editor and replace all of the code with the following:
using Microsoft.Spark.Sql;
namespace MySparkApp
{
class Program
{
static void Main(string[] args)
{
// Create a Spark session
var spark = SparkSession
.Builder()
.AppName("word_count_sample")
.GetOrCreate();
// Create initial DataFrame
DataFrame dataFrame = spark.Read().Text("input.txt");
// Count words
var words = dataFrame
.Select(Functions.Split(Functions.Col("value"), " ").Alias("words"))
.Select(Functions.Explode(Functions.Col("words"))
.Alias("word"))
.GroupBy("word")
.Count()
.OrderBy(Functions.Col("count").Desc());
// Show results
words.Show();
// Stop Spark session
spark.Stop();
}
}
}Run the following command to build you application:
dotnet buildRun the following command to submit your application to run on Apache Spark:
%SPARK_HOME%\bin\spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin\Debug\netcoreapp3.0\microsoft-spark-2.4.x-0.6.0.jar dotnet bin\Debug\netcoreapp3.0\mySparkApp.dll~/bin/spark-2.4.1-bin-hadoop2.7/bin/spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin/Debug/netcoreapp3.0/microsoft-spark-2.4.x-0.6.0.jar dotnet bin/Debug/netcoreapp3.0/mySparkApp.dllWhen your app runs, you will see the data written to the console.
... logging ...
+------+-----+
| word|count|
+------+-----+
| .NET| 3|
|Apache| 2|
| app| 2|
| This| 2|
| Spark| 2|
| World| 1|
|counts| 1|
| for| 1|
| words| 1|
| with| 1|
| Hello| 1|
| uses| 1|
+------+-----+
... logging ...Congratulations, you've created your first .NET for Apache Spark application!
