ansible_proxmox_cloudinit_kubernetes_minio_ipxe_kickstart_cluster/README.md

What is this demo?

A tech demo to apply some cloud tools, concepts and IaC to build PXE boot environment without TFTP. Ansible is the glue to bind the various components together, Proxmox an easy to script cloud-init capable hypervisor and Kubernetes a place to initialise our applications with dynamic configuration.

How does it work?

• Downloads and validates disk images and uploads to Proxmox using the API.
• Builds a 3 nodes on Proxmox booting them with cloud-init to setup networking and install packages necessary for a Kubernetes cluster.
• Uses CFSSL to generate a certificate authority and service certificates for use in container services.
• Installs the Kubernetes cluster and joins the worker nodes.
• Render and apply the deployment files for the Kubernetes dashboard and user authentication credential files.
• Render and apply the deployment files for Kubernetes loadbalancer with ownership of a range of external IP addresses to be assigned to services.
• Render and apply the deployment files for a Docker registry with
  Minio storage, this runs a sidecar job container to create the Minio buckets used by the registry service and pxe boot files.
• Builds a customised dnsmasq container to serve dhcp/pxe config to farm nodes, pushes the container to the Docker registry.
• Builds 3 additional farm nodes in Proxmox setup to pxe boot and retrieves their mac addresses for dnsmasq and pxe.
• Render and apply the deployment with nested runtime configurations for dnsmasq, pxe and kickstart.
• Boot the farm nodes to retrieve pxe config and node specific kickstart from Minio over http, kickstart pulls build install files over public http/s, avoiding any tftp/nfs or interference with other dhcp servers on the network segment.

What is in the Ansible?

Interact with Proxmox using built-in Ansible modules, Proxmox API calls and commands that can only be run via the cli in shell.

Import images to lvm disk slices, expanding lvm disks and seed cloud-init data in Proxmox.

Using custom cloud init userdata in Proxmox to boot hosts, install software and signify to Ansible the target host is ready (useful where dhcp may not yet exist)

Chaining roles that have dependencies, accessing variables and facts between plays using in-built variables.

Ways to loop, looping blocks of code.

Delegate tasks to different hosts in the same play, Ansible privilege escalation, run_as, delegate and local_action examples.

General flow control and strategies to run tasks on multiple hosts in parallel / series including a solution to check for a host to be provisioned, updated then rebooted before being ready for the playbook to continue.

Generating dynamic inventories for re-use inline and in subsequent plays or re-runs (or outputted as a stateful record of a system build).

Comparing lists, matching / eliminating / duplicates.

Using a local Ansible configuration to override default behaviour, used to ensure the inventory doesn't have to be built or specified during debugging.

Querying json command output and in-built {{ vars }} with jmespath queries and building dicts of selected output.

Jinja2 template loops with inline variable declaration.

Various string and url manipulation examples.

What is in the Kubernetes config and templates?

A collection of config files illustrating the relationships between common configuration items, Helm often abstracts the user from exploring and understanding the API, recently being described as a Kubernetes package manager. IMHO it is helpful to better understand these components and how to trawl the API documentation to understand how Helm can be implemented.

Components such as Configmaps and Jobs are more powerful and flexible than many tutorials suggest, when used with Deployment parameters we often find we don't need to build our own custom containers to include additional data, scripts and modified entrypoints.

How to run

Edit groupvars/all for Proxmox credentials and network attributes

ansible-playbook site.yml

Enhancements

• Explore using Ansible roles and modules to build and manage Kubernetes, for expediency the installation was derived from the manual steps to install a cluster (shell commands). Explore using Ansible roles and modules to write yaml for
• Add Ansible install to kickstart and a Systemd unit/timer to the farm nodes for the purposes of an Ansible pull job to install packages.
• Create an etcd/consul service for the farm nodes to run discovery to pull and populate environment/customer specific parameters for Ansible playbooks.
• Populate a Minio bucket with the Ansible playbooks for ansible-pull.
• Demonstrate how to pull unique host/environment/customer specific parameters through pxe configs for the operating system to evaluate (example: address of etcd keys, secondary network adapter vlan/bonds)