There was more than 50GB free space available yesterday. All of the sudden, now it is only 8GB free space. What is going on?

​There was more than 50GB free space available yesterday. All of the sudden, now it is only 8GB free space. What is going on?   Read More

​

Here is an alternate method to prevent your computer from locking.

Code:

function KeepScreenAwake {

    param($minutes = 9999)

    Write-Host “The screen will remain active for $minutes minutes.”

    $wshell = New-Object -ComObject Wscript.Shell

    for ($i = 0; $i -lt $minutes; $i++) {

        Write-Host “The screen will be active for another $($minutes – $i) minutes.”

        Start-Sleep -Seconds 60

        $wshell.SendKeys(“{NUMLOCK}{NUMLOCK}”)

    }

}

KeepScreenAwake 

​Here is an alternate method to prevent your computer from locking. Code:function KeepScreenAwake {    param($minutes = 9999)     Write-Host “The screen will remain active for $minutes minutes.”     $wshell = New-Object -ComObject Wscript.Shell     for ($i = 0; $i -lt $minutes; $i++) {        Write-Host “The screen will be active for another $($minutes – $i) minutes.”        Start-Sleep -Seconds 60        $wshell.SendKeys(“{NUMLOCK}{NUMLOCK}”)    }} KeepScreenAwake   Read More

​

I recently configured my PC for personal use by linking it to my Microsoft account. To reverse this, I performed a reset, preserving my files and folders, and set it up as a local account for school or business purposes without internet access and no login password. While I encountered some challenges along the way, I ultimately managed to rename the computer ‘Rhapsody’ and establish the user folder as ‘Blue’ in the C:Users directory. Despite changing the account name to ‘Gershwin’ in the Settings to replace my default name, I noticed that my original name continued to appear on the startup screen.

Upon researching a tutorial on how to modify the owner of a file, folder, drive, or registry key, I learned that Microsoft derives the account name from the first five letters of the associated email address. To address this, I explored altering or removing my email address from various links and profiles linked to my account. However, I am uncertain if simply leaving it blank would suffice or necessitate replacing it with another name.

If I had understood how to configure the initial setup to solely display the computer name and user folder name as ‘Rhapsody’ and ‘Blue’, I wonder what name Microsoft would have designated as the user or owner in the File Properties/Details section.

I have identified registry entries containing my email address and the default user name assigned by Microsoft. If necessary, I can provide more detailed information or screenshots of these entries. Additionally, certain CredManLog entries state ‘IsMissingCred’, indicating a credential issue.

​I recently configured my PC for personal use by linking it to my Microsoft account. To reverse this, I performed a reset, preserving my files and folders, and set it up as a local account for school or business purposes without internet access and no login password. While I encountered some challenges along the way, I ultimately managed to rename the computer ‘Rhapsody’ and establish the user folder as ‘Blue’ in the C:Users directory. Despite changing the account name to ‘Gershwin’ in the Settings to replace my default name, I noticed that my original name continued to appear on the startup screen. Upon researching a tutorial on how to modify the owner of a file, folder, drive, or registry key, I learned that Microsoft derives the account name from the first five letters of the associated email address. To address this, I explored altering or removing my email address from various links and profiles linked to my account. However, I am uncertain if simply leaving it blank would suffice or necessitate replacing it with another name. If I had understood how to configure the initial setup to solely display the computer name and user folder name as ‘Rhapsody’ and ‘Blue’, I wonder what name Microsoft would have designated as the user or owner in the File Properties/Details section. I have identified registry entries containing my email address and the default user name assigned by Microsoft. If necessary, I can provide more detailed information or screenshots of these entries. Additionally, certain CredManLog entries state ‘IsMissingCred’, indicating a credential issue.  Read More

​

I adore my Dell Inspiron 3510 laptop, but I have been disappointed by its flimsy case, specifically the fragile screen hinges. Unfortunately, they broke just eight months after I purchased the laptop, and although they were repaired, they broke again a year later. As a temporary fix, I secured them with strong silver duct tape, which has held up reasonably well, although I find myself having to re-tape it from time to time. Considering my positive experience with Dell in general, I am now contemplating investing in the 3550 Latitude model, which is priced at approximately $640. I would appreciate hearing your thoughts on this particular laptop.

​I adore my Dell Inspiron 3510 laptop, but I have been disappointed by its flimsy case, specifically the fragile screen hinges. Unfortunately, they broke just eight months after I purchased the laptop, and although they were repaired, they broke again a year later. As a temporary fix, I secured them with strong silver duct tape, which has held up reasonably well, although I find myself having to re-tape it from time to time. Considering my positive experience with Dell in general, I am now contemplating investing in the 3550 Latitude model, which is priced at approximately $640. I would appreciate hearing your thoughts on this particular laptop.  Read More

​

Hi all, I’ve purchased a 13,8″ Microsoft Surface Laptop 7th Generation (Qualcomm Snapdragon X Plus [10 cores], 16 GB RAM, Qualcomm Adreno™ Onboard Graphics). I want to use a 49″ external monitor (Asus ROG Strix XG49WCR) with a resolution of 5120 x 1440 as a 2nd monitor extension of the laptop. However the max. resulotion I can select is 3840 x 1080. With another laptop i can run the resolution of 5120 x 1440 @ 60Hz on this screen…

Has anybody a idea why the max resolution is not possible? Does the CPU/GPU not fulfill the requirements?

Microsoft support says it normally should run with this resolution… I already changed every setting, used different cables, re-installed all drivers, installed all updates etc. … Asus costumer support couldn’t help me either…

Appreciate any kind of help!

​Hi all, I’ve purchased a 13,8″ Microsoft Surface Laptop 7th Generation (Qualcomm Snapdragon X Plus [10 cores], 16 GB RAM, Qualcomm Adreno™ Onboard Graphics). I want to use a 49″ external monitor (Asus ROG Strix XG49WCR) with a resolution of 5120 x 1440 as a 2nd monitor extension of the laptop. However the max. resulotion I can select is 3840 x 1080. With another laptop i can run the resolution of 5120 x 1440 @ 60Hz on this screen…Has anybody a idea why the max resolution is not possible? Does the CPU/GPU not fulfill the requirements?Microsoft support says it normally should run with this resolution… I already changed every setting, used different cables, re-installed all drivers, installed all updates etc. … Asus costumer support couldn’t help me either…Appreciate any kind of help!  Read More

​

缅甸腾龙娱乐联合公司注册联系（微——guug71）（公司官网注册——www . tl99826.com) **注册一直没有改变全程免费办理
我们是一个有着非常丰富工作经验的一群专业人员组成的，腾龙公司是一家成立二十年的公司，本公司的遵旨就是诚信认真对待每一位客户，我们始终秉持着诚信、专业和创新的原则，以确保我们提供的每一款都符合最高标准。

​缅甸腾龙娱乐联合公司注册联系（微——guug71）（公司官网注册——www . tl99826.com) **注册一直没有改变全程免费办理我们是一个有着非常丰富工作经验的一群专业人员组成的，腾龙公司是一家成立二十年的公司，本公司的遵旨就是诚信认真对待每一位客户，我们始终秉持着诚信、专业和创新的原则，以确保我们提供的每一款都符合最高标准。  Read More

​

小勐拉环球国际位于缅甸掸邦东部的第四特区，是一个以华人为主的地区，与中国云南省西双版纳相邻。这个地区以其丰富的旅游资源和娱乐行业而闻名，有着“小香港”的美誉。近年来，随着当地政府吸引中国内地投资者的努力，小勐拉正在迅速成为缅北甚至整个缅甸的一个繁华城市 。

小勐拉的经济主要依赖于服务业和旅游业。该地区拥有丰富的自然景观，如山高谷深、古树参天，同时也是佛教圣地，有众多佛塔。这些特点吸引了无数游客前来游览。小勐拉的主要景点包括禁毒博物馆、大金塔、基督教堂、金四角民族乐园、卧佛寺公园、关帝庙、威尼斯人度假村和勐拉小学校等 。

从中国前往小勐拉的路线通常是从云南出发，经过昆明、景洪市，最后到达打洛镇，然后通过边境口岸进入小勐拉。这个过程中需要办理边境通行证。小勐拉与中国云南的紧密地理位置和相似的居^本次回答已被终止~

​小勐拉环球国际位于缅甸掸邦东部的第四特区，是一个以华人为主的地区，与中国云南省西双版纳相邻。这个地区以其丰富的旅游资源和娱乐行业而闻名，有着“小香港”的美誉。近年来，随着当地政府吸引中国内地投资者的努力，小勐拉正在迅速成为缅北甚至整个缅甸的一个繁华城市 。小勐拉的经济主要依赖于服务业和旅游业。该地区拥有丰富的自然景观，如山高谷深、古树参天，同时也是佛教圣地，有众多佛塔。这些特点吸引了无数游客前来游览。小勐拉的主要景点包括禁毒博物馆、大金塔、基督教堂、金四角民族乐园、卧佛寺公园、关帝庙、威尼斯人度假村和勐拉小学校等 。从中国前往小勐拉的路线通常是从云南出发，经过昆明、景洪市，最后到达打洛镇，然后通过边境口岸进入小勐拉。这个过程中需要办理边境通行证。小勐拉与中国云南的紧密地理位置和相似的居^本次回答已被终止~  Read More

​

Acknowledgments

Introduction

As of today, more and more we see AKS gaining share as an orchestration solution for HPC/AI workloads. The drivers behind this trend are multiple: the progressive move toward containerization of the HPC/AI software stacks, the ease of management and the universal nature of Kubernetes APIs.
The focus of this blog post is to provide a guide for getting an AKS cluster InfiniBand enabled, with the possibility of having HCAs or IPoIB available inside Kubernetes Pods as cluster resources.

Several methodologies and articles have provided insights on the topic, as well as the official documentation of NVIDIA Network Operator.

The purpose of this article is organizing and harmonizing the different experiences while proposing a deployment model which is closer to the most maintained and standard way of enabling InfiniBand cluster: using NVIDIA Network Operator.

Of course, this is only the first step for having an AKS cluster HPC/AI ready. This article is meant to work in pair with the blog post for NVIDIA GPU operator.

In a similar way, a proper HPC/AI AKS cluster will require an adequate job scheduler like kqueue or Volcano to handle properly multi-node jobs and allowing a smooth interaction in parallel processing. This is out of the scope of the current Blog post, but references and examples can be found in the already mentioned in the repository related to HPC on AKS or running workloads on NDv4 GPUs on AKS.

Getting the basics up and running

In this section we will describe how to deploy a vanilla dev/testing cluster where the content of this article can be deployed for demonstration. In case you already have your AKS cluster with InfiniBand enabled nodes, you can skip this section.

Deploying a vanilla AKS cluster

The standard way of deploying a vanilla AKS cluster is following the standard procedure described in Azure documentation.

Please be aware that this command will create an AKS cluster with:

Kubenet as Network CNI
AKS cluster will have a public endpoint
Local accounts with Kubernetes RBAC

In general, we strongly recommend for production workloads to look the main security concepts for AKS cluster.

Use Azure CNI
Evaluate using Private AKS Cluster to limit API exposure to the public internet
Evaluate using Azure RBAC with Entra ID accounts or Kubernetes RBAC with Entra ID accounts

This will be out of scope for the present demo, but please be aware that this cluster is meant for NVIDIA Network Operator demo purposes only.

Using Azure CLI we can create an AKS cluster with this procedure (replace the values between arrows with your preferred values):

export RESOURCE_GROUP_NAME=<YOUR_RG_NAME>
export AKS_CLUSTER_NAME=<YOUR_AKS_CLUSTER_NAME>
export LOCATION=<YOUR_LOCATION>

## Following line to be used only if Resource Group is not available
az group create –resource-group $RESOURCE_GROUP_NAME –location $LOCATION

az aks create –resource-group $RESOURCE_GROUP_NAME –name $AKS_CLUSTER_NAME –node-count 2 –generate-ssh-keys

Connecting to the cluster

To connect to the AKS cluster, several ways are documented in Azure documentation. In this part of the article we will use Linux Ubuntu 22.04 on a VM with Azure CLI installed to deploy the cluster. We authenticate to Azure by using the az login command and install the Azure Client (be aware that in the login command you may be required to use –tenant <TENTANT_ID> in case you have access to multiple tenants). Moreover, be sure you are using the right subscription checking with az account show:

## Add –tenant <TENANT_ID> in case of multiple tenants
## Add –identity in case of using a managed identity on the VM
az login
az aks install-cli
az aks get-credentials –resource-group $RESOURCE_GROUP_NAME –name $AKS_CLUSTER_NAME

After this is completed, you should be able to perform standard kubectl commands like:

kubectl get nodes

root@aks-gpu-playground-rg-jumpbox:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-agentpool-11280554-vmss00001a Ready agent 9m40s v1.27.7
aks-agentpool-11280554-vmss00001b Ready agent 9m47s v1.27.7

Command line will be perfectly fine for all the operations in the blog post. However, if you would like to have a TUI experience, we suggest to use k9s, which can be easily installed on Linux following the installation instructions. For Ubuntu, you can install current version at the time of Blog post creation with:

wget “https://github.com/derailed/k9s/releases/download/v0.32.5/k9s_linux_amd64.deb”
dpkg -i k9s_linux_amd64.deb

k9s allows to easily interact with the different resources of AKS cluster directly from a terminal user interface. It can be launched with k9s command. Detailed documentation on how to navigate on the different resources (Pods, DaemonSets, Nodes) can be found on the official k9s documentation page.

Attaching an Azure Container registry to the Azure Kubernetes Cluster

To be able to create our own docker container to run them on the cluster it is convenient to create an Azure Container Registry (create a private Azure Container Registry) and to attach it to the AKS cluster. This can be done in the following way:

export ACR_NAME=<ACR_NAME_OF_YOUR_CHOICE>

az acr create –resource-group $RESOURCE_GROUP_NAME
–name $ACR_NAME –sku Basic

az aks update –name $AKS_CLUSTER_NAME –resource-group $RESOURCE_GROUP_NAME –attach-acr $ACR_NAME

You will need to be able to perform pull and push operations from this Container Registry through Docker. To be able to push new containers into the registry you need to login first which you can do using the az acr login command: 

az acr login –name $ACR_NAME

Connecting the AKS cluster to the container registry makes sure that the system managed identity for AKS nodes will have access to be allowed to pull containers.

About taints for AKS nodes

It is important to understand deeply the concept of taints and tolerations for Spot and GPU nodes in AKS. In case of spot instances, AKS applies the following taint:  

kubernetes.azure.com/scalesetpriority=spot:NoSchedule​

We will show later that we have to add some tolerations to be able to get the NVIDIA Network Operator running on Spot instances.

Creating the first IB pool

The currently created AKS cluster has as a default only a node pool with 2 nodes of Standard_D2s_v2 VMs. It is now time to add the first InfiniBand enabled node pool. This can be done using Azure Cloud Shell, for example using an Standard_HB120rs_v3 and setting the autoscaling enabled. We are setting the minimum number of nodes to 2 for the subsequent tests, so please remind to downscale the pool setting the minimum node number equal to 0 once completed to avoid not necessary costs:

az aks nodepool add
–resource-group $RESOURCE_GROUP_NAME
–cluster-name $AKS_CLUSTER_NAME
–name hb120v3
–node-vm-size Standard_HB120rs_v3
–enable-cluster-autoscaler
–min-count 2 –max-count 2 –node-count 2

In order to deploy in Spot mode, the following flags should be added to Azure CLI:

–priority Spot –eviction-policy Delete –spot-max-price -1

Once the deployment has finished, we can see the nodes with the kubectl get nodes command: 

kubectl get nodes

NAME STATUS ROLES AGE VERSION
aks-hb120v3-16191816-vmss000000 Ready <none> 51s v1.29.8
aks-hb120v3-16191816-vmss000001 Ready <none> 51s v1.29.8
aks-nodepool1-28303986-vmss000000 Ready <none> 101m v1.29.8
aks-nodepool1-28303986-vmss000001 Ready <none> 101m v1.29.8

Installing the Network Operator

Installing Helm

On a machine with kubectl configured and with the context configured above for connection to the AKS cluster, we run the following to install helm if not installed already:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
&& chmod 700 get_helm.sh
&& ./get_helm.sh

Installing Node Feature Discovery

On Azure, since most of the nodes have a ConnectX-4 network card for accelerated networking, it is important to fine tune the node feature recognition. Moreover, AKS nodes may have special taints that needs to be tolerated by Node Feature Discovery daemons. Because of this we will install Node Feature Discovery separately from NVIDIA Network Operator. 

NVIDIA Network Operator will act on the nodes with the label feature.node.kubernetes.io/pci-15b3.present. Moreover, it is important to tune the node discovery plugin so that it will be scheduled even on Spot instances of the Kubernetes cluster. Here we introduce also the MIG toleration and GPU toleration to grant compatibility with the NVIDIA GPU Operator article.

helm install –wait –create-namespace -n network-operator node-feature-discovery node-feature-discovery –create-namespace –repo https://kubernetes-sigs.github.io/node-feature-discovery/charts –set-json master.config.extraLabelNs='[“nvidia.com”]’ –set-json worker.tolerations='[{ “effect”: “NoSchedule”, “key”: “sku”, “operator”: “Equal”, “value”: “gpu”},{“effect”: “NoSchedule”, “key”: “kubernetes.azure.com/scalesetpriority”, “value”:”spot”, “operator”: “Equal”},{“effect”: “NoSchedule”, “key”: “mig”, “value”:”notReady”, “operator”: “Equal”}]’

After enabling Node Feature Discovery, it is important to create a custom rule to precisely match ConnectX-6 and ConnectX-7 cards available on the most recent Azure nodes. This can be done creating a file called nfd-network-rule.yaml containing the following:

—
apiVersion: nfd.k8s-sigs.io/v1alpha1
kind: NodeFeatureRule
metadata:
name: nfd-network-rule
spec:
rules:
– name: “nfd-network-rule”
labels:
“feature.node.kubernetes.io/pci-15b3.present”: “true”
matchFeatures:
– feature: pci.device
matchExpressions:
device: {op: In, value: [“101c”, “101e”]}

After this file is created, we should apply this to the AKS cluster:

kubectl apply -n network-operator -f nfd-network-rule.yaml

After a few seconds the Node Feature Discovery will label ONLY the HBv3 nodes. This can be checked with:

kubectl get nodes

NAME STATUS ROLES AGE VERSION
aks-hb120v3-16191816-vmss000000 Ready <none> 51s v1.29.8
aks-hb120v3-16191816-vmss000001 Ready <none> 51s v1.29.8
aks-nodepool1-28303986-vmss000000 Ready <none> 101m v1.29.8
aks-nodepool1-28303986-vmss000001 Ready <none> 101m v1.29.8

kubectl describe nodes aks-hb120v3-16191816-vmss000001 | grep present

feature.node.kubernetes.io/pci-15b3.present=true

kubectl describe nodes aks-nodepool1-28303986-vmss000000 | grep present

The label should not be found on the agents node pool without InfiniBand (since they are of type Standard_D2s_v2)

Installing and configuring the NVIDIA Network Operator helm chart

We download the NVIDIA Network Operator helm chart and update our helm repository:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

Then we generate a configuration template file with the name values.yaml. For a quick view of all the available values we could type:

helm show values nvidia/network-operator –version v24.7.0

The values.yaml file is divide into several sections to configure the different components of the network operator. These can be summarized as:

General Parameters

OFED Driver and Configuration

NVIDIA Shared Memory Driver

SR-IOV Network Operator and configuration
Device plugins

RDMA Shared Device Plugin and Configuration

SR-IOV Shared Device Plugin and Configuration

IB Kubernetes

UFM Secret

Secondary Networks

CNI Plugin

Multus CNI

IPoIB CNI

IPAM CNI Plugin

NVIDIA IPAM Plugin

The SR-IOV Network Operator is relevant for an OpenShift environments and requires access to the InfiniBand subnet manager which is not available on Azure. This is why we disabled it. For this article we modify the values.yaml file to enable the following components:

General Parameters

Node Feature Discovery with an Node Affinity for the nodes that have InfiniBand devies

NIC Feature Discovery

NIC Cluster Policy

OFED Driver and NicCluster policy

RDMA Shared Device Plugin

Secondary Networks

CNI Plugin

Multus CNI

IPoIB CNI

IPAM CNI Plugin

Our proposed values.yaml becomes:

nfd:
# — Deploy Node Feature Discovery operator.
enabled: false
deployNodeFeatureRules: false

# — Enable CRDs upgrade with helm pre-install and pre-upgrade hooks.
upgradeCRDs: true

operator:
tolerations: [{ “effect”: “NoSchedule”, “key”: “sku”, “operator”: “Equal”, “value”: “gpu”},{“effect”: “NoSchedule”, “key”: “kubernetes.azure.com/scalesetpriority”, “value”:”spot”, “operator”: “Equal”},{“effect”: “NoSchedule”, “key”: “mig”, “value”:”notReady”, “operator”: “Equal”}]

sriovNetworkOperator:
# — Deploy SR-IOV Network Operator.
enabled: false

deployCR: true
ofedDriver:
# — Deploy the NVIDIA DOCA Driver driver container.
deploy: true

rdmaSharedDevicePlugin:
# — Deploy RDMA shared device plugin.
deploy: true

useCdi: true

secondaryNetwork:
# — Deploy Secondary Network.
deploy: true
cniPlugins:
# — Deploy CNI Plugins Secondary Network.
deploy: true

multus:
# — Deploy Multus Secondary Network.
deploy: true
ipoib:
# — Deploy IPoIB CNI.
deploy: true
ipamPlugin:
# — Deploy IPAM CNI Plugin Secondary Network.
deploy: true

nicFeatureDiscover:
deploy: false

tolerations:
– key: “node-role.kubernetes.io/master”
operator: “Exists”
effect: “NoSchedule”
– key: “node-role.kubernetes.io/control-plane”
operator: “Exists”
effect: “NoSchedule”
– key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
– key: kubernetes.azure.com/scalesetpriority
operator: Equal
value: spot
effect: NoSchedule
– key: mig
operator: Equal
value: notReady
effect: NoSchedule

Once we have done all modifications within the values.yaml file we can deploy the helm chart:    

helm install network-operator nvidia/network-operator
-n nvidia-network-operator
–create-namespace
–version v24.7.0
-f ./values.yaml
–wait

After a few minutes we should see the following pods running on compute nodes:

kubectl get pods -n nvidia-network-operator –field-selector spec.nodeName=aks-hb120v3-16191816-vmss000001

NAME READY STATUS RESTARTS AGE
cni-plugins-ds-f4kcj 1/1 Running 0 2m50s
kube-ipoib-cni-ds-blfxf 1/1 Running 0 2m50s
kube-multus-ds-45hzt 1/1 Running 0 2m50s
mofed-ubuntu22.04-659dcf4b88-ds-7m462 0/1 Running 0 2m7s
nv-ipam-node-fpht4 1/1 Running 0 2m14s
whereabouts-78z5j 1/1 Running 0 2m50s

Further, we can check if the rdma-device plugin has published as a new resource:

kubectl describe node aks-hb120v3-16191816-vmss000001 | grep rdma/rdma_shared_device_a:
rdma/rdma_shared_device_a: 63
rdma/rdma_shared_device_a: 63

The first line shows the number of configured Host Channel Adaptors (HCAs) that can be used by pods running on the node. The second entry show how many of them are are available. As we have not started any pods that request this resource yet, both values are the same. This maximal number of HCAs can be set within values.yaml configuration file by modifying the rdmHcaMax setting within the rdmaSharedDevicePlugin section:

…
rdmaSharedDevicePlugin:
deploy: true
image: k8s-rdma-shared-dev-plugin
repository: ghcr.io/mellanox
version: sha-fe7f371c7e1b8315bf900f71cd25cfc1251dc775
useCdi: true
…
resources:
– name: rdma_shared_device_a
vendors: [15b3]
rdmaHcaMax: 63
…

We can also check if the IPoIB module has been loaded on the compute node and that the corresponding network interface exists. To do so, we use the kubectl debug command to start a busybox container on the node and connect to it:

kubectl debug node/aks-hb120v3-16191816-vmss000001 -it –image=mcr.microsoft.com/cbl-mariner/busybox:2.0

Once on the node, we can run the chroot command to get on the host and sudo to become root:

chroot /host
sudo su –

Then we can run the following commands to check if the IPoIB module is loaded and the network interface is configured:

lsmod | grep ipoib
ib_ipoib 135168 0
ib_cm 131072 2 rdma_cm,ib_ipoib
ib_core 409600 8 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
mlx_compat 69632 11 rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core

ip a

…
8: ibP257s79720: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP group default qlen 256
link/infiniband 00:00:01:48:fe:80:00:00:00:00:00:00:00:15:5d:ff:fd:33:ff:1a brd 00:ff:ff:ff:ff:12:40:1b:80:06:00:00:00:00:00:00:ff:ff:ff:ff
altname ibP257p0s0
inet6 fe80::215:5dff:fd33:ff1a/64 scope link
valid_lft forever preferred_lft forever

The name of the IPoIB interface ibP257s79720 is not consistant within the same VM SKU but the altname ibP257p0s0 is. This will be important when we configure the IPoIB CNI later.

About the OFED driver

The AKS Ubuntu 22.04 node image comes with the inbox Infiniband drivers which are usually relatively old and miss some features. Therefore, the NVIDIA Network Operator uses a driver container to load the most-recent DOCA-OFED modules into the the OS kernel of the hosts. To make sure that the modules match the kernel version of the host, the entripoint script of the container compiles the right DOCA-OFED version before loading the modules. This process takes a few minutes and slows down spinning-up of new nodes within the nodepool. This could be fixed by using a custom DOCA-OFED container that contains the modules for the version of the AKS node’s kernel. 

Configuring the IBoIP CNI

The IPoIB CNI plugin allows us to create an IPoIB child link on each Pod that runs on the node. The number of these links is limited to the number of HCAs cofigured in the values.yaml configuration file as discussed previously.

To enable the IPoIB CNI we create the following ipoib_network.yaml file:

apiVersion: mellanox.com/v1alpha1
kind: IPoIBNetwork
metadata:
name: example-ipoibnetwork
spec:
networkNamespace: “default”
master: “ibP257p0s0”
ipam: |
{
“type”: “whereabouts”,
“datastore”: “kubernetes”,
“kubernetes”: {
“kubeconfig”: “/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig”
},
“range”: “192.168.5.225/28”,
“exclude”: [
“192.168.6.229/30”,
“192.168.6.236/32”
],
“log_file” : “/var/log/whereabouts.log”,
“log_level” : “info”,
“gateway”: “192.168.6.1”
}

The configration cotains the IPoIB interface name on the AKS host as value for the master key and the configuration for IP range for the IPoIB subnet. We apply the configuration by running:

kubectl create -f ipoib_network.yaml

Building a sample image for testing IPoIB and RDMA connectivity

In order to test with InfiniBand utilities the RDMA connectivity, it is critical to have a Docker image where DOCA OFED is installed with the userspace tools.

An example of such an image could have this Dockerfile:

FROM ubuntu:22.04

ENV DOCA_URL=”https://www.mellanox.com/downloads/DOCA/DOCA_v2.8.0/host/doca-host_2.8.0-204000-24.07-ubuntu2204_amd64.deb”
ENV DOCA_SHA256=”289a3e00f676032b52afb1aab5f19d2a672bcca782daf9d30ade0b59975af582″

RUN apt-get update
RUN apt-get install wget git -y
WORKDIR /root
RUN wget $DOCA_URL
RUN echo “$DOCA_SHA256 $(basename ${DOCA_URL})” | sha256sum –check –status
RUN dpkg -i $(basename ${DOCA_URL})
RUN apt-get update
RUN apt-get install doca-ofed-userspace -y

After creating this Dockerfile it is possible to build and push the image in the ACR created at the beginning of the blog post:

az acr login -n $ACR_NAME
docker build . -t $ACR_NAME.azurecr.io/ibtest
docker push $ACR_NAME.azurecr.io/ibtest

Testing IPoIB and RDMA connectivity

We will create three pods and spread them over two compute AKS nodes to demonstrate the IPoIB and RDMA connectivity. To so so we need to create the pods with the following resource request:

Define the number of required CPU cores in a way that two pods can fit on a single host but the rhird one will be scheduled on a second one, e.g. cpu: 50.
Request one HCA per pod: rdma/rdma_shared_device_a: 1.
Run the pods in privileged: true mode to be able to access the RDMA InfiniBand interface.

The last point is only required if we want to use the InfinBand interface for RDMA workloads as MPI but not if we only want to use the IPoIB network communication. 

To test the connectivity we create the following ibtestpod.yaml (the image name must be adjusted with your ACR_NAME):

apiVersion: v1
kind: Pod
metadata:
name: ipoib-test-pod-1
annotations:
k8s.v1.cni.cncf.io/networks: example-ipoibnetwork
spec:
restartPolicy: OnFailure
tolerations:
– key: “kubernetes.azure.com/scalesetpriority”
operator: “Equal”
value: “spot”
effect: NoSchedule
containers:
– image: <ACR_NAME>.azurecr.io/ibtest
name: rdma-test-ctr
securityContext:
capabilities:
add: [ “IPC_LOCK” ]
privileged: true
resources:
requests:
cpu: 50
rdma/rdma_shared_device_a: 1
limits:
cpu: 50
rdma/rdma_shared_device_a: 1
command:
– sh
– -c
– |
sleep inf
—
apiVersion: v1
kind: Pod
metadata:
name: ipoib-test-pod-2
annotations:
k8s.v1.cni.cncf.io/networks: example-ipoibnetwork
spec:
restartPolicy: OnFailure
tolerations:
– key: “kubernetes.azure.com/scalesetpriority”
operator: “Equal”
value: “spot”
effect: NoSchedule
containers:
– image: <ACR_NAME>.azurecr.io/ibtest
name: rdma-test-ctr
securityContext:
capabilities:
add: [ “IPC_LOCK” ]
privileged: true
resources:
requests:
cpu: 50
rdma/rdma_shared_device_a: 1
limits:
cpu: 50
rdma/rdma_shared_device_a: 1
command:
– sh
– -c
– |
sleep inf
—
apiVersion: v1
kind: Pod
metadata:
name: ipoib-test-pod-3
annotations:
k8s.v1.cni.cncf.io/networks: example-ipoibnetwork
spec:
restartPolicy: OnFailure
tolerations:
– key: “kubernetes.azure.com/scalesetpriority”
operator: “Equal”
value: “spot”
effect: NoSchedule
containers:
– image: <ACR_NAME>.azurecr.io/ibtest
name: rdma-test-ctr
securityContext:
capabilities:
add: [ “IPC_LOCK” ]
privileged: true
resources:
requests:
cpu: 50
rdma/rdma_shared_device_a: 1
limits:
cpu: 50
rdma/rdma_shared_device_a: 1
command:
– sh
– -c
– |
sleep inf

We start the pods by executing:

kubectl apply -f ibtestpod.yaml

pod/ipoib-test-pod-1 created
pod/ipoib-test-pod-2 created
pod/ipoib-test-pod-3 created

We can check that all pods are distributed over the two nodes host even that might not be the same in your setup:

kubectl get pods –output ‘jsonpath={range .items[*]}{.spec.nodeName}{” “}{.metadata.namespace}{” “}{.metadata.name}{“n”}{end}’
aks-hb120v3-16191816-vmss000000 default ipoib-test-pod-1
aks-hb120v3-16191816-vmss000001 default ipoib-test-pod-2
aks-hb120v3-16191816-vmss000001 default ipoib-test-pod-3

By executing the following commands we can get the IPoIB IP addresses of the pods:

kubectl describe pod ipoib-test-pod-1 | grep multus | grep ipoib
Normal AddedInterface 35s multus Add net1 [192.168.5.225/28] from default/example-ipoibnetwork
kubectl describe pod ipoib-test-pod-2 | grep multus | grep ipoib
Normal AddedInterface 38s multus Add net1 [192.168.5.226/28] from default/example-ipoibnetwork
kubectl describe pod ipoib-test-pod-3 | grep multus | grep ipoib
Normal AddedInterface 41s multus Add net1 [192.168.5.227/28] from default/example-ipoibnetwork

We see that pod  ipoib-test-pod-1 that runs on the first AKS node has the IPoIB IP address 192.168.5.225, while pod ipoib-test-pod-2 has the IPoIB IP address 192.168.5.226 and pod ipoib-test-pod-3 192.168.5.227. Both run in the second AKS node.

To test the intra-node and inter-node IPoIB connectivity we connect to ipoib-test-pod-2 and ping the other two pods:

kubectl exec -it ipoib-test-pod-2 — /bin/sh
/ # ping 192.168.5.225 -c 3
PING 192.168.5.225 (192.168.5.225): 56 data bytes
64 bytes from 192.168.5.225: seq=0 ttl=64 time=1.352 ms
64 bytes from 192.168.5.225: seq=1 ttl=64 time=0.221 ms
64 bytes from 192.168.5.225: seq=2 ttl=64 time=0.281 ms

— 192.168.5.225 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.221/0.618/1.352 ms
/ # ping 192.168.5.227 -c 3
PING 192.168.5.227 (192.168.5.227): 56 data bytes
64 bytes from 192.168.5.227: seq=0 ttl=64 time=1.171 ms
64 bytes from 192.168.5.227: seq=1 ttl=64 time=0.255 ms
64 bytes from 192.168.5.227: seq=2 ttl=64 time=0.290 ms

— 192.168.5.227 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.255/0.572/1.171 ms

We also confirm the IP address is 192.168.5.226 by running:

/ # ip a | grep net1
12: net1@if8: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 2044 qdisc noqueue state UP
inet 192.168.5.226/28 brd 192.168.5.239 scope global net1

Just notice that the name of the IBoIB link device on the node is net1@if8. To test the RDMA we just start the ib_read_lat on ipoib-test-pod-2:

# ib_read_lat

************************************
* Waiting for client to connect… *
************************************

Then we open another shell, connect to iboip-test-pod-1 and run ib_read_lat 192.168.5.226, which is the IP address of connect to iboip-test-pod-2:

kubectl exec -it ipoib-test-pod-1 — /bin/sh

# ib_read_lat 192.168.5.226
—————————————————————————————
RDMA_Read Latency Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : ON
TX depth : 1
Mtu : 4096[B]
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
—————————————————————————————
local address: LID 0x1b7 QPN 0x0126 PSN 0xc5305e OUT 0x10 RKey 0x04070c VAddr 0x0056453ff00000
remote address: LID 0x1c1 QPN 0x0126 PSN 0x18a616 OUT 0x10 RKey 0x04060a VAddr 0x005607b90aa000
—————————————————————————————
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
Conflicting CPU frequency values detected: 1846.554000 != 3463.693000. CPU Frequency is not max.
Conflicting CPU frequency values detected: 1846.554000 != 3510.158000. CPU Frequency is not max.
2 1000 3.80 5.95 3.83 3.84 0.00 3.93 5.95
—————————————————————————————

This shows that there is RDMA connectivity between iboip-test-pod-1 and iboip-test-pod-2 which run each on a different AKS nodes. Then we go back to pod iboip-test-pod-2 and start the ib_read_lat server again:

[root@ipoib-test-pod-2 /]# ib_read_lat

************************************
* Waiting for client to connect… *
************************************

Then we open another shell, connect to iboip-test-pod-3 and run ib_read_lat 192.168.5.226 again:

kubectl exec -it ipoib-test-pod-3 — /bin/sh
# ib_read_lat 192.168.5.226
—————————————————————————————
RDMA_Read Latency Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : ON
TX depth : 1
Mtu : 4096[B]
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
—————————————————————————————
local address: LID 0x1b7 QPN 0x0127 PSN 0x99bde7 OUT 0x10 RKey 0x040700 VAddr 0x005624407f9000
remote address: LID 0x1c1 QPN 0x0127 PSN 0x9539df OUT 0x10 RKey 0x040600 VAddr 0x00563b8d000000
—————————————————————————————
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
Conflicting CPU frequency values detected: 1846.554000 != 3512.949000. CPU Frequency is not max.
Conflicting CPU frequency values detected: 1846.554000 != 2089.386000. CPU Frequency is not max.
2 1000 3.79 7.03 3.82 3.82 0.00 3.87 7.03
—————————————————————————————

Now both of these pods run on teh same AKS nodes which is refleted in the lower latency. Let us clean-up by deleting the pods:

kubectl delete -f ibtestpod.yaml

pod “ipoib-test-pod-1” deleted
pod “ipoib-test-pod-2” deleted
pod “ipoib-test-pod-3” deleted

Uninstalling the NVIDIA Network Operator

To apply a different configuration we might want to uninstall the operator by uninstalling the helm template before we install a new version:

helm uninstall network-operator -n nvidia-network-operator

Downscaling the node pool

To allow the nodepool to autoscale down to zero, use the following Azure CLI command. This is important to avoid any unwanted cost:

az aks nodepool update
–resource-group $RESOURCE_GROUP_NAME
–cluster-name $AKS_CLUSTER_NAME
–name hb120v3
–update-cluster-autoscaler
–min-count 0
–max-count 2

Conclusions

This article is meant to provide a low-level insight on how to configure InfiniBand using NVIDIA Network operator on an AKS cluster.

Of course, this is only the first step for having an AKS cluster HPC/AI ready. This article is meant to work in pair with the blog post for NVIDIA GPU operator.

In a similar way, a proper HPC/AI AKS cluster will require an adequate job scheduler like kqueue or Volcano to handle properly multi-node jobs and allowing a smooth interaction in parallel processing. This is out of the scope of the current Blog post, but references and examples can be found in the already mentioned in the repository related to HPC on AKS or running workloads on NDv4 GPUs on AKS.

​Microsoft Tech Community – Latest Blogs –Read More 

To convert MBOX to PDF, follow these straightforward steps:

1. Choose a Converter: Start by selecting an KMTTECH MBOX to PDF Converter. You can find various tools online, both free and paid. Popular options include dedicated software and web-based converters.

2. Download and Install: If you choose software, download and install it on your computer. For online tools, you won’t need to install anything; just visit the website.

3. Import MBOX File: Open the converter and look for an option to import or add files. Browse your computer to locate the MBOX file you want to convert and upload it.

4. Select Emails: Once the MBOX file is loaded, you may have the option to select specific emails for conversion. If you want to convert all emails, proceed to the next step.

5.  Choose Output Format : Set the output format to PDF. Some converters allow you to customize settings, such as choosing the destination folder and preserving the original formatting.

6.  Start Conversion : Click the convert button to initiate the process. Once completed, the PDF files will be saved to your designated location.

7.  Check Your PDFs : Finally, navigate to the output folder to verify that your emails have been successfully converted to PDF format.

This process makes it easy to archive, share, or print your important emails in a universally accessible format.

​To convert MBOX to PDF, follow these straightforward steps:1. Choose a Converter: Start by selecting an KMTTECH MBOX to PDF Converter. You can find various tools online, both free and paid. Popular options include dedicated software and web-based converters.2. Download and Install: If you choose software, download and install it on your computer. For online tools, you won’t need to install anything; just visit the website.3. Import MBOX File: Open the converter and look for an option to import or add files. Browse your computer to locate the MBOX file you want to convert and upload it.4. Select Emails: Once the MBOX file is loaded, you may have the option to select specific emails for conversion. If you want to convert all emails, proceed to the next step.5.  Choose Output Format : Set the output format to PDF. Some converters allow you to customize settings, such as choosing the destination folder and preserving the original formatting.6.  Start Conversion : Click the convert button to initiate the process. Once completed, the PDF files will be saved to your designated location.7.  Check Your PDFs : Finally, navigate to the output folder to verify that your emails have been successfully converted to PDF format.This process makes it easy to archive, share, or print your important emails in a universally accessible format.  Read More

​

腾龙集团公司开户注册网址www.tl99825.com微信13542035通常需要遵循以下一般步骤：准备相关资料：可能包括个人或企业身份证明、联系方式、地址证明等。联系盛世集团：通过其官方网站、客服渠道等，了解具体开户注册流程和所需资料要求。填写表格：按照要求填写相关信息。提交资料：将准备好的资料提交给盛世集团，可以通过线上提交或线路下提交的方式。等待审核：公司同意提交的资料进行审核。完成审核：审核通过后，即可成功开户注册。需要注意的是，具体步骤和要求可能因盛世集团的规定而有所不同。建议您与该公司进行详细沟通，以确保顺利完成开户登记。

​腾龙集团公司开户注册网址www.tl99825.com微信13542035通常需要遵循以下一般步骤：准备相关资料：可能包括个人或企业身份证明、联系方式、地址证明等。联系盛世集团：通过其官方网站、客服渠道等，了解具体开户注册流程和所需资料要求。填写表格：按照要求填写相关信息。提交资料：将准备好的资料提交给盛世集团，可以通过线上提交或线路下提交的方式。等待审核：公司同意提交的资料进行审核。完成审核：审核通过后，即可成功开户注册。需要注意的是，具体步骤和要求可能因盛世集团的规定而有所不同。建议您与该公司进行详细沟通，以确保顺利完成开户登记。  Read More

​