Required configuration

The cluster configuration file can be generated by using clusterctl generate cluster command. This command actually uses the template file and replace the values surrounded by ${} with environment variables. You have to set all required environment variables in advance. The following sections explain some more details about what should be configured.

Note: You can use the template file by manually replacing values.

clusterctl generate cluster capi-quickstart \
  --kubernetes-version v1.21.3 \
  --control-plane-machine-count=1 \
  --worker-machine-count=1 \
  > capi-quickstart.yaml

In order to fetch the configuration parameters via the terminal, please install cmk and jq

Cluster Level Configurations

These configurations are passed while defining the CloudStackCluster and apply to the entire cluster.

Zone

The Zone must be exposed as an environment variable CLOUDSTACK_ZONE_NAME and is a mandatory parameter. As of now, only advanced zones without security groups is supported.

The list of zones can be fetched using the cmk cli as follows :

cmk list zones listall=true | jq '.zone[] | {name, id}'

Network

The network must be exposed as an environment variable CLOUDSTACK_NETWORK_NAME and is a mandatory parameter. As of now, only isolated and shared networks are supported. If the specified network does not exist, a new isolated network will be created.

The list of networks for the specific zone can be fetched using the cmk cli as follows :

cmk list networks listall=true zoneid=<zoneid> | jq '.network[] | {name, id, type}'

Endpoint

The endpoint of the workload cluster. It can either be an IP or an FQDN which is resolvable from the management cluster

If on an isolated network, and the endpoint is an IP, it must be an IP in the Public IP range. The necessary Firewall and LoadBalancing rules will be automatically created on Apache CloudStack for the specified IP.

If on a shared network, and the endpoint is an IP, it must belong to the shared network range and not allocated to any other resource on CloudStack.

The Endpoint is exposed in two parts, as the CLUSTER_ENDPOINT_IP and CLUSTER_ENDPOINT_PORT environment variables.

The list of Public IPs for the specific zone can be fetched using the cmk cli as follows :

cmk list publicipaddresses listall=true zoneid=<zone-id> forvirtualnetwork=true allocatedonly=false | jq '.publicipaddress[] | select(.state == "Free" or .state == "Reserved") | .ipaddress'

Machine Level Configurations

These configurations are passed while defining the CloudStackMachine. They can differ based on the MachineSet mapped to it.

Service Offerings

The service offerings for the Control Plane Nodes are specified by the CLOUDSTACK_CONTROL_PLANE_MACHINE_OFFERING environment variable. It must have a minimum of 2GB RAM and 2 vCPU

The service offerings for the Worker Nodes are specified by the CLOUDSTACK_WORKER_MACHINE_OFFERING environment variable.

The list of Service offerings for the specific zone can be fetched using the cmk cli as follows :

cmk list serviceofferings listall=true zoneid=<zone-id> cpunumber=2 memory=2048 | jq '.serviceoffering[] | {name, id}'

Virtual Machine Template

We currently depend on an up-to-date version of cloud-init otherwise the operating system choice is yours. The kubeadm bootstrap provider we’re using also depends on some pre-installed software like a container runtime, kubelet, kubeadm, etc. For an examples how to build such an image take a look at CloudStack CAPI Images.

Prebuilt images can be found here

The image can be referenced by exposing it as an environment variable CLOUDSTACK_TEMPLATE_NAME.

The list of Templates for the specific zone can be fetched using the cmk cli as follows :

cmk list templates zoneid=<zone-id> templatefilter=executable | jq '.template[] | {name, id}'

Optional Configurations

Cluster Level Configurations

These configurations are passed while defining the CloudStackCluster and apply to the entire cluster.

Account

The account in which the CAPC cluster resources are to be created. Please note that the credentials of the user passed to CAPC via the CLOUDSTACK_B64ENCODED_SECRET must have access the the specified account.

The account can be specified by adding the CloudStackCluster.spec.account field in the yaml specification

The list of accounts can be fetched using the cmk cli as follows :

cmk list accounts listall=true | jq '.account[] | {name, id}'

Please note that if the optional configurations of account and domainid are passed, the corresponding account must have access to the specified resources on CloudStack such as the Network, Public IP, VM Template, Service Offering, SSH Key, Affinity Group, etc

Domain

The domain / subdomain in which the CAPC cluster resources are to be created. Please note that the credentials of the user passed to CAPC via the CLOUDSTACK_B64ENCODED_SECRET must have access the the specified domain

The domain can be specified by adding the CloudStackCluster.spec.domain field in the yaml specification

The list of domains can be fetched using the cmk cli as follows :

cmk list domains listall=true | jq '.domain[] | {name, id, path}'

Please note that if the optional configurations of account and domainid are passed, the corresponding account must have access to the specified resources on CloudStack such as the Network, Public IP, VM Template, Service Offering, SSH Key, Affinity Group, etc

Machine Level Configurations

These configurations are passed while defining the CloudStackMachine. They can differ based on the MachineSet mapped.

SSH KeyPair

The SSH key pair must be an existing key registered in Apache CloudStack and is exposed as the environment variable CLOUDSTACK_SSH_KEY_NAME.

The ssh keypair can be specified by adding the CloudStackMachine.spec.sshKey field in the yaml specification

The list of SSH Keypairs can be fetched using the cmk cli as follows :

cmk list sshkeypairs listall=true | jq '.sshkeypair[] | {name, id}'

If the user wishes to pass a public key not registered in CloudStack directly to the node, it can be done by adding the KubeadmConfigTemplate.spec.template.spec.users spec in the cluster definition yaml. Eg:

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: ${CLUSTER_NAME}-md-0
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            provider-id: "cloudstack:///'{{ ds.meta_data.instance_id }}'"
          name: '{{ local_hostname }}'
      users:
        - name: ${OS_USERID}
          sshAuthorizedKeys:
            - ${SSH_KEY_MATERIAL}
          sudo: ALL=(ALL) NOPASSWD:ALL

To learn how to configure the required network access in order to SSH into the node, please see here

Affinity Groups

The nodes in the MachineDeployment mapped to a corresponding CloudStackMachine can have a specific host affinity or be assigned to affinity groups. The affinity can either be specified pro (host affinity) or anti (host anti affinity) in the CloudStackMachine.spec.affinity field in the yaml specification and the required affinity groups will be created in CloudStack If existing affinity groups in CloudStack wish to be used, the group IDs can be passed as a list in the CloudStackMachine.spec.affinitygroupids field in the yaml specification

The list of existing affinity groups can be fetched using the cmk cli as follows :

cmk list affinitygroups listall=true | jq '.affinitygroup[] | {name, id}'

VM Details

These are arbitrary key value pairs which are passed as VM details while deploying the nodes.

The VM details can be specified by adding the CloudStackMachine.spec.details field in the yaml specification

Log level

TODO / Maybe add feature ?

Timeout settings

TODO / Add feature

Apache CloudStack Credentials

  1. Generate the API and Secret Key for your Apache CloudStack instance either via the UI (Accounts > User > API Key) or the getUserKeys API.

  2. Create a file named cloud-config, substituting in your own environment’s values

    [Global]
api-url = <cloudstackApiUrl>
api-key = <cloudstackApiKey>
secret-key = <cloudstackSecretKey>

  3. Run the following command to save the above Apache CloudStack connection info into an environment variable :

    export CLOUDSTACK_B64ENCODED_SECRET=$(base64 -w0 -i cloud-config)

  4. Once exported, the management cluster can be initialized with Apache CloudStack as the underlying infrastructure provider

    clusterctl init --infrastructure cloudstack