Use .NET Interactive to run .NET code in Jupyter Notebooks on an Azure Machine Learning compute instance
Introduction
In this post, I'll go over how you can run .NET code in Jupyter Notebooks using .NET Interactive and Azure Machine Learning compute instances.
Jupyter Notebooks are an graphical interactive computing environment used in disciplines like data science and education. When it comes to prototyping or learning scenarios, they can help users see near real-time what their code is doing. The interactive computing protocol used by Jupyter Notebooks is extensible and as a result, you're able to run a variety of languages inside of this environment via kernels. Kernels are processes that take the input from the user, execute the code, and return the output for display and further processing. Using .NET Interactive, a .NET Core global tool, that among its many features provides a kernel for Jupyter Notebooks to run .NET (C#/F#) and PowerShell code.
Although you can install and run Jupyter Notebooks on your local computer, services like Azure Machine Learning provide single VMs known as compute instances which have Jupyter Notebooks as well as many popular data science libraries and development tools like Docker preinstalled. Therefore, installing .NET Interactive on one of these VMs is a seamless experience. Additionally, Azure Machine Learning gives you the ability to share computing resources with teams of various sizes.
Let's get started!
Prerequisites
- Azure account. If you don't have one, create one.
Create Azure Machine Learning workspace
A workspace organizes all of you Azure Machine Learning resources and assets such as compute instances in a single place. In order to use Azure Machine Learning, you have to create a workspace.
There's multiple ways to create a workspace. For this writeup, I use the Azure Portal.
Navigate to portal.azure.com
In the portal, select Create a resource.
From the resource list, select AI + Machine Learning > Machine Learning
Fill in the form and select Review + create.
Review your information before creating the workspace and select Create. Deployment takes a few minutes. As part of your workspace deployment, additional Azure resources are created such as Azure Container Registry, Azure Storage account, Azure Application Insights, and Azure KeyVault.
For more information on workspaces, see the Azure Machine Learning workspace documentation.
Create an Azure Machine Learning compute instance
As mentioned earlier, an Azure Machine Learning compute instance is a single VM that comes with a variety of development tools and libraries commonly used for data science and machine learning workflows preinstalled.
Navigate to Azure Machine Learning studio. The studio is a web interface for managing your Azure Machine Learning resources such as data, experiments, models, and compute.
In studio, select Create new > Compute instance to create a new compute instance.
Provide a name for your instance and select Create. You can also customize the size of your VM and whether you want it to be GPU enabled. In this case, I just selected the preselected option.
Once your compute instance is provisioned, select the Jupyter link to launch Jupyter Notebooks.
For more information on compute instances, see the Azure Machine Learning compute instance documentation.
Install the .NET Core 3.1 SDK
At the time of this writing, the image used by compute instances is Ubuntu 16.04. Therefore, we'll be installing the Linux version of the .NET Core SDK.
Inside the Jupyter Notebooks, select New > Terminal to create a new terminal.
In the terminal, add the Microsoft package signing key to your list of trusted keys.
wget https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
Install the .NET Core 3.1 SDK with the following command:
sudo apt-get update; \
sudo apt-get install -y apt-transport-https && \
sudo apt-get update && \
sudo apt-get install -y dotnet-sdk-3.1
Note that as of the time of this writing, .NET Interactive requires .NET Core 3.1.
You can also find these installation instructions in the .NET Core installation documentation.
Install .NET Interactive
Start off by checking which kernels are installed. In the terminal, enter the following command.
jupyter kernelspec list
The output should look something like the output below
Available kernels:
python3 /anaconda/envs/azureml_py36/share/jupyter/kernels/python3
ir /usr/local/share/jupyter/kernels/ir
python3-azureml /usr/local/share/jupyter/kernels/python3-azureml
Then, install the .NET Interactive global tool
dotnet tool install -g --add-source "https://dotnet.myget.org/F/dotnet-try/api/v3/index.json" Microsoft.dotnet-interactive
Use the .NET Interactive tool to install the .NET and powershell kernels.
dotnet interactive jupyter install
Finally, run the following command to confirm that the kernels have been installed.
jupyter kernelspec list
In the output you should see the C#, F#, and PowerShell kernels listed
Available kernels:
.net-csharp /home/azureuser/.local/share/jupyter/kernels/.net-csharp
.net-fsharp /home/azureuser/.local/share/jupyter/kernels/.net-fsharp
.net-powershell /home/azureuser/.local/share/jupyter/kernels/.net-powershell
python3 /anaconda/envs/azureml_py36/share/jupyter/kernels/python3
ir /usr/local/share/jupyter/kernels/ir
python3-azureml /usr/local/share/jupyter/kernels/python3-azureml
Add .NET global tools directory to PATH
The Jupyter Notebook server is managed by a systemd service. Although during installation .NET Interactive is added to your PATH environment variable, it's not added to the environment file /etc/environment which is used by systemd. Therefore, in order to run .NET Interactive, you have to add the .NET global tools directory to that file.
The contents of the file should look similar to the one below
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
AML_CloudName=AzureCloud
Open the /etc/environment using your preferred text editor (vi/nano) and replace the PATH with the following content:
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/azureuser/.dotnet/tools"
Save your changes and restart the Jupyter Server. In the terminal, run the following command.
sudo service jupyter restart
This will close your terminal session since the notebook server has been restarted.
Close all Jupyter Notebook windows.
Create new .NET Jupyter Notebook
From Azure Machine Learning studio, launch Jupyter Notebooks again by selecting the Jupyter link.
Then, in Jupyter Notebooks select New and create a new notebook (C#/F#/PowerShell). In this case, I created an F# notebook.
Once the kernel is ready, enter code into the first cell and run it.
Congratulations! You should now be able to run .NET and PowerShell code inside of your Azure Machine Learning compute instance.
Conclusion
In this post, I showed how you can run .NET code in Jupyter Notebooks on an Azure Machine Learning compute instance with the help of .NET Interactive. This enables you to interactively prototype solutions remotely while still having control over your environment and dependencies. Now that you have .NET interactive setup, compute instances also give you the option of using JupyterLab. Feel free to tinker with the different environments and see which one works best for you. Happy coding!