Link Search Menu Expand Document

AKS

deployment kubernetes azure aks

On this page

  1. What are Kubernetes and Azure Kubernetes Services (AKS)?
  2. Deploy AKS
    1. Role-Based Access Control (RBAC)
    2. Azure CLI
    3. ARM Template
  3. Sizing
    1. More than five million records
    2. Less than five million records

What are Kubernetes and Azure Kubernetes Services (AKS)?

Kubernetes is open-source software that helps deploy and manage containerized applications at scale. It orchestrates a cluster of Azure virtual machines, schedules containers, automatically manages service discovery, incorporates load balancing, and tracks resource allocation. It also checks the health of individual resources and heals apps with auto-restart and auto-replication. AKS provides a managed Kubernetes service with automated provisioning, upgrading, monitoring, and on-demand scaling. (Source: https://azure.microsoft.com/en-us/services/kubernetes-service/#faq)

There are several ways to install an AKS cluster: ARM templates, Azure CLI, Terraform - the choice is yours.

To deploy Azure resources for CluedIn, you need to provide CluedIn Partner GUID (e64d9978-e282-4d1c-9f2e-0eccb50582e4 ). The way you provide the Partner GUID depends on the way you deploy AKS:

Deploy AKS

Role-Based Access Control (RBAC)

To deploy and manage Azure resources, you need sufficient access rights. You can read more about it in Microsoft Documentation: Manage access to your Azure environment with Azure role-based access control Azure built-in roles You need a Contributor role on the Subscription level. If it’s not possible to have this role, you need to ask someone with enough permissions to create an AKS cluster. When you create a new AKS cluster in a particular resource group, Microsoft Azure automatically creates an infrastructure resource group (with “MC_” prefix) to keep AKS-related resources: disks, public IP, identity, etc. Therefore, you should have enough permissions to create resource groups in a given subscription to create a cluster. Then, to manage the cluster, you need to be a Contributor in two AKS resource groups - the group where you have created the AKS and the related infrastructure group.

Azure CLI

Walkthrough (Microsoft Docs): https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough Microsoft’s instructions to deploy the Partner GUID: https://docs.microsoft.com/en-us/azure/marketplace/azure-partner-customer-usage-attribution#example-azure-cli

To install the Partner GUID, you need to add an environment variable to your terminal session.

Bash:

export AZURE_HTTP_USER_AGENT='pid-e64d9978-e282-4d1c-9f2e-0eccb50582e4' ;
echo AZURE_HTTP_USER_AGENT # should print pid-e64d9978-e282-4d1c-9f2e-0eccb50582e4

PowerShell:

$env:AZURE_HTTP_USER_AGENT='pid-e64d9978-e282-4d1c-9f2e-0eccb50582e4' ;
$env:AZURE_HTTP_USER_AGENT # should print pid-e64d9978-e282-4d1c-9f2e-0eccb50582e4

ARM Template

Walkthrough (Microsoft Docs): https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-rm-template

Walkthrough with Partner GUID (CluedIn Docs): https://documentation.cluedin.net/kb/azure-customer-usage-attribution.

Microsoft’s instructions to deploy the Partner GUID: https://docs.microsoft.com/en-us/azure/marketplace/azure-partner-customer-usage-attribution#add-a-guid-to-a-resource-manager-template

To deploy with the Partner GUID, you only need to add this deployment to the resources section:

    { 
      "apiVersion": "2020-06-01",
      "name": "pid-e64d9978-e282-4d1c-9f2e-0eccb50582e4",
      "type": "Microsoft.Resources/deployments",
      "properties": {
          "mode": "Incremental",
          "template": {
              "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
              "contentVersion": "1.0.0.0",
              "resources": []
          }
      }
    }, 

Sizing

The recommended size of the AKS cluster depends on the amount of data you plan to process. But, of course, you can always scale it up and down as you need.

More than five million records

NodepoolNodepool TypeNodepool SizeNodeWorkloadCPU Request (Cores)RAM RequestCPU Limit (Cores)RAM LimitDisk TypeDisk SizePurposeTaintToleration
ProcessingF16s_v215.7 Cores, 28 GB (Allocatable)1CluedIn Processing1420GB1524GB--Processing incoming and outgoing master data
2CluedIn Processing1420GB1524GB--Additional processing for historical ingestion (temporary)
DatalayerD8s_v47.6 Cores, 28 GB (Allocatable)1Neo4j728728Standard SSD500GBGraph DatabasedatalayerPool=truedatalayerPool=true
2ElasticSearch728728Standard SSD500GBSearch IndexdatalayerPool=truedatalayerPool=true
3SQL Server38616Standard SSD750GBRelational DatabasedatalayerPool=truedatalayerPool=true
RabbitMQ1424Standard SSD150GBService BusdatalayerPool=truedatalayerPool=true
Redis0.5512Mi11Standard SSD32GBCachedatalayerPool=truedatalayerPool=true
GenericD4_v33.8 Cores, 14 GB (Allocatable)1Annotation0.12564Mi10.512--Annotation Service--
GQL0.264Mi11--GraphQL Layer--
Submitter0.25128Mi11--Clue Submitter Service--
UI0.250.75256Mi512Mi--User Interface--
Webapi0.25256Mi0.5512Mi--User Interface Wrapper--
2CluedIn API1.56Gi210Gi--CluedIn Server WebApi--
CluedIn Crawling28Gi212Gi--CluedIn Crawling Pod--

Less than five million records

NodepoolNodepool TypeNodepool SizeNodeWorkloadCPU Request (Cores)RAM RequestCPU Limit (Cores)RAM LimitDisk TypeDisk SizePurposeTaintToleration
ProcessingF8s_v27.6 Cores, 14GB (Allocatable)1CluedIn Processing712712--Processing incoming and outgoing master dataprocessingPool=trueprocessingPool=true
DatalayerD8s_v47.6 Cores, 28 GB (Allocatable)1Neo4j3.5123.512Standard SSD250GBGraph DatabasedatalayerPool=truedatalayerPool=true
ElasticSearch3.5123.512Standard SSD500GBSearch IndexdatalayerPool=truedatalayerPool=true
2SQL Server38616Standard SSD750 GBRelational DatabasedatalayerPool=truedatalayerPool=true
RabbitMQ1424Standard SSD150 GBService BusdatalayerPool=truedatalayerPool=true
Redis0.5512Mi11Standard SSD32GBCachedatalayerPool=truedatalayerPool=true
GenericD4_v33.8 Cores, 14 GB (Allocatable)1Annotation0.12564Mi10.512--Annotation Service--
GQL0.264Mi11--GraphQL Layer--
Submitter0.25128Mi11--Clue Submitter Service--
UI0.250.75256Mi512Mi--User Interface--
Webapi0.25256Mi0.5512Mi--User Interface Wrapper--
2CluedIn API1.56Gi210Gi--CluedIn Server WebApi--
CluedIn Crawling28Gi212Gi--CluedIn Crawling Pod--