tseed
/
ansible_proxmox_cloudinit_kubernetes_minio_ipxe_kickstart_cluster
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

initial commit

main
tseed 2022-10-26 17:18:32 +01:00
commit 8e86d5e9df
43 changed files with 2584 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

87
README.md Executable file
View File

@ -0,0 +1,87 @@
## What is this demo?
A tech demo to apply some cloud tools, concepts and IaC to build PXE boot environment without TFTP. Ansible is the glue to bind the various components together, Proxmox an easy to script cloud-init capable hypervisor and Kubernetes a place to initialise our applications with dynamic configuration.
## How does it work?
- Downloads and validates disk images and uploads to Proxmox using the
API.
- Builds a 3 nodes on Proxmox booting them with cloud-init to setup
networking and install packages necessary for a Kubernetes cluster.
- Uses CFSSL to generate a certificate authority and service
certificates for use in container services.
- Installs the Kubernetes cluster and joins the worker nodes.
- Render and apply the deployment files for the Kubernetes dashboard and user
authentication credential files.
- Render and apply the deployment files for Kubernetes loadbalancer
with ownership of a range of external IP addresses to be assigned to
services.
- Render and apply the deployment files for a Docker registry with
Minio storage, this runs a sidecar job container to create the Minio
buckets used by the registry service and pxe boot files.
- Builds a customised dnsmasq container to serve dhcp/pxe config to
farm nodes, pushes the container to the Docker registry.
- Builds 3 additional farm nodes in Proxmox setup to pxe boot and
retrieves their mac addresses for dnsmasq and pxe.
- Render and apply the deployment with nested runtime configurations
for dnsmasq, pxe and kickstart.
- Boot the farm nodes to retrieve pxe config and node specific
kickstart from Minio over http, kickstart pulls build install files
over public http/s, avoiding any tftp/nfs or interference with other
dhcp servers on the network segment.
## What is in the Ansible?
Interact with Proxmox using built-in Ansible modules, Proxmox API calls and commands that can only be run via the cli in shell.
Import images to lvm disk slices, expanding lvm disks and seed cloud-init data in Proxmox.
Using custom cloud init userdata in Proxmox to boot hosts, install software and signify to Ansible the target host is ready (useful where dhcp may not yet exist)
Chaining roles that have dependencies, accessing variables and facts between plays using in-built variables.
Ways to loop, looping blocks of code.
Delegate tasks to different hosts in the same play, Ansible privilege escalation, run_as, delegate and local_action examples.
General flow control and strategies to run tasks on multiple hosts in parallel / series including a solution to check for a host to be provisioned, updated then rebooted before being ready for the playbook to continue.
Generating dynamic inventories for re-use inline and in subsequent plays or re-runs (or outputted as a stateful record of a system build).
Comparing lists, matching / eliminating / duplicates.
Using a local Ansible configuration to override default behaviour, used to ensure the inventory doesn't have to be built or specified during debugging.
Querying json command output and in-built {{ vars }} with jmespath queries and building dicts of selected output.
Jinja2 template loops with inline variable declaration.
Various string and url manipulation examples.
## What is in the Kubernetes config and templates?
A collection of config files illustrating the relationships between common configuration items, Helm often abstracts the user from exploring and understanding the API, recently being described as a Kubernetes package manager. IMHO it is helpful to better understand these components and how to trawl the API documentation to understand how Helm can be implemented.
Components such as Configmaps and Jobs are more powerful and flexible than many tutorials suggest, when used with Deployment parameters we often find we don't need to build our own custom containers to include additional data, scripts and modified entrypoints.
## How to run
Edit groupvars/all for Proxmox credentials and network attributes
ansible-playbook site.yml
## Enhancements
- Explore using Ansible roles and modules to build and manage Kubernetes, for
expediency the installation was derived from the manual steps to
install a cluster (shell commands).
Explore using Ansible roles and modules to write yaml for
- Add Ansible install to kickstart and a Systemd unit/timer to the farm
nodes for the purposes of an Ansible pull job to install packages.
- Create an etcd/consul service for the farm nodes to run discovery to
pull and populate environment/customer specific parameters for
Ansible playbooks.
- Populate a Minio bucket with the Ansible playbooks for ansible-pull.
- Demonstrate how to pull unique host/environment/customer specific
parameters through pxe configs for the operating system to evaluate
(example: address of etcd keys, secondary network adapter vlan/bonds)

10
ansible.cfg Executable file
View File

@ -0,0 +1,10 @@
[defaults]
inventory = inventory/nodes.ini,inventory/farm.ini
[privilege_escalation]
[paramiko_connection]
[ssh_connection]
[persistent_connection]
[accelerate]
[selinux]
[colors]
[diff]

0
bionic-server-cloudimg-amd64.img Executable file
View File

14
generate_certificates/files/ca-csr.json Executable file
View File

@ -0,0 +1,14 @@
{
"CN": "Test Root CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "GB",
"L": "Sheffield",
"ST": "England"
}
]
}

13
generate_certificates/files/cfssl-profile.json Executable file
View File

@ -0,0 +1,13 @@
{
"signing": {
"default": {
"expiry": "8760h"
},
"profiles": {
"server": {
"usages": ["signing", "digital signing", "key encipherment", "server auth"],
"expiry": "8760h"
}
}
}
}

14
generate_certificates/files/registry-csr.json Executable file
View File

@ -0,0 +1,14 @@
{
"CN": "registry",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "GB",
"L": "Sheffield",
"ST": "England"
}
]
}

114
generate_certificates/tasks/main.yml Executable file
View File

@ -0,0 +1,114 @@
- name: install cfssl
apt:
name: golang-cfssl
- name: check certificate(s) exist on remote host
stat:
path: ~/{{ item }}
with_items:
- ca-key.pem
- ca.pem
- registry-key.pem
- registry.pem
register: stat_result
- name: query results of output files exist on remote host
set_fact:
query_certs: "{{ stat_result | json_query(jmesquery) }}"
vars:
jmesquery: "results[?(@.stat.exists)].item"
- name: report where certificate(s) exist
debug:
msg:
- "certificate(s) already exist, you may want to remove to make a fresh run"
- "{{ query_certs }}"
when: query_certs | length>0
- name: send CA certificate signing request to host
copy:
src: ca-csr.json
dest: "~/"
mode: '0640'
when: '"ca.pem" not in query_certs' # interesting escape sequence for yaml parsing, to remember when matching a string in a var/list
- name: send Registry certificate signing request to host
copy:
src: registry-csr.json
dest: "~/"
mode: '0640'
when: '"registry.pem" not in query_certs'
- name: send cfssl profile config to host
copy:
src: cfssl-profile.json
dest: "~/"
mode: '0640'
when: '"registry.pem" not in query_certs'
- name: generate CA certificate
shell:
cmd: cfssl gencert -initca ~/ca-csr.json | cfssljson -bare ~/ca -
when: '"ca.pem" not in query_certs'
# CN in registry-csr.json would usually contain an fqdn, we arent using dns for service discovery so use -hostname to add SAN (Subject Alternative Name) to specify ip's and shortnames for ssl validation by clients
# we could put /etc/hosts entries on our clients that match the certificate CN
#
# openssl x509 -in registry.pem -text -noout
#
# Subject: C = GB, ST = England, L = Sheffield, CN = registry
# DNS:localhost, DNS:registry, IP Address:192.168.101.200, IP Address:127.0.0.1
#
- name: generate Registry certificate
shell:
cmd: cfssl gencert -ca ~/ca.pem -ca-key ~/ca-key.pem -config ~/cfssl-profile.json -profile=server -hostname="{{ registry_service_metallb_ip_endpoint }}",127.0.0.1,localhost,registry ~/registry-csr.json | cfssljson -bare ~/registry
when: '"registry.pem" not in query_certs'
- name: get CA certificate contents
shell:
cmd: cat ~/ca.pem
register: ca_pem_out
- name: get Registry certificate contents
shell:
cmd: cat ~/registry.pem
register: registry_pem_out
- name: get Registry certificate key
shell:
cmd: cat ~/registry-key.pem
register: registry_key_out
- set_fact:
ca_pem: "{{ ca_pem_out.stdout }}"
registry_pem: "{{ registry_pem_out.stdout }}"
registry_key: "{{ registry_key_out.stdout }}"
# not the way to add a certificate on ubuntu, update-ca-certificates will remove the entry whenever executed
# - name: add CA certificate to k8s nodes
# become: yes
# blockinfile:
# path: /etc/ssl/certs/ca-certificates.crt
# block: "{{ ca_pem }}"
# insertafter: last
# delegate_to: "{{ item }}"
# with_items:
# - "{{ groups['control'] }}"
- name: copy CA certificate to node
copy:
dest: "/usr/local/share/ca-certificates/ocf-ca.crt"
content: "{{ ca_pem }}"
delegate_to: "{{ item }}"
with_items:
- "{{ groups['control'] }}"
- "{{ groups['worker'] }}"
- name: import CA certificate to root certificate store on node
shell:
cmd: update-ca-certificates
delegate_to: "{{ item }}"
with_items:
- "{{ groups['control'] }}"
- "{{ groups['worker'] }}"

34
get_cloud-init_image/tasks/main.yml Executable file
View File

@ -0,0 +1,34 @@
- set_fact:
image_name: "{{ ((image_url | urlsplit).path).split ('/')[-1] }}"
- name: check local cloud-init image present
stat:
path: ./{{ image_name }}
register: img_local_present
- name: check site is available
uri:
url: "{{ image_url }}"
follow_redirects: none
method: HEAD
register: _result
until: _result.status == 200
retries: 2
delay: 5 # seconds
when: not img_local_present.stat.exists
- name: download image
get_url:
url: "{{ image_url }}"
dest: .
when: not img_local_present.stat.exists
- name: check local image present
stat:
path: ./{{ image_name }}
register: img_local_present
- name: report image downloaded
fail:
msg: "image {{ image_name }} not present, download failed"
when: not img_local_present.stat.exists

39
group_vars/all Executable file
View File

@ -0,0 +1,39 @@
---
proxmox_host: 192.168.101.239
# Pmox API creds
proxmox_node: pve
proxmox_user: root@pam
proxmox_pass: Password0
# Pmox SSH creds
proxmox_ssh_user: root
proxmox_ssh_pass: Password0
# Pmox Storage
proxmox_vm_datastore: local-lvm
proxmox_img_datastore: local-image
proxmox_img_datastore_path: /var/lib/local-image
proxmox_node_disk_size: 10G
# Pmox Network
proxmox_vmbr: vmbr0
# K8S Cloud-init image
image_url: https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
# K8S nodes
number_control_nodes: 1 # this k8s setup isnt using a lbl or dns cname with entries for all control nodes as the primary api endpoint, stick to a single node
number_worker_nodes: 2 # have as many as you like
ip_range_24_prefix: 192.168.101
ip_range_gateway: 192.168.101.254
ip_range_subnet: 255.255.255.0
ip_range_dns: 192.168.101.254
ip_range_control_start: 70
ip_range_worker_start: 80
domain: local
node_account: ocfadmin
node_account_password: Password0
# K8S pods
pod_network_range: 10.200.0.0/16
metallb_ip_range_start: 192.168.101.200
metallb_ip_range_end: 192.168.101.210
registry_service_metallb_ip_endpoint: 192.168.101.200
minio_service_metallb_ip_endpoint: 192.168.101.201
# Farm
ip_range_farm_start: 60
number_farm_nodes: 3

9
inventory/farm.ini Executable file
View File

@ -0,0 +1,9 @@
[farm]
farm-01 ansible_host=192.168.101.61
farm-02 ansible_host=192.168.101.62
farm-03 ansible_host=192.168.101.63
[all:vars]
ansible_password=Password0
ansible_user=ocfadmin
ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"

36
inventory/nodes.ini Executable file
View File

@ -0,0 +1,36 @@
[control]
control-01 ansible_host=192.168.101.71
[all]
proxmox_server ansible_host=192.168.101.239
control-01 ansible_host=192.168.101.71
worker-01 ansible_host=192.168.101.81
worker-02 ansible_host=192.168.101.82
farm-01 ansible_host=192.168.101.61
farm-02 ansible_host=192.168.101.62
farm-03 ansible_host=192.168.101.63
[farm]
farm-01 ansible_host=192.168.101.61
farm-02 ansible_host=192.168.101.62
farm-03 ansible_host=192.168.101.63
[worker]
worker-01 ansible_host=192.168.101.81
worker-02 ansible_host=192.168.101.82
[ungrouped]
[proxmox]
proxmox_server ansible_host=192.168.101.239
[proxmox:vars]
ansible_ssh_user=root
ansible_ssh_pass=Password0
become=true
[all:vars]
ansible_password=Password0
ansible_user=ocfadmin
ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
ansible_python_interpreter=/usr/bin/python3

82
k8s_control_init/tasks/main.yml Executable file
View File

@ -0,0 +1,82 @@
# this role is only designed to run on a single host for testing, the --apiserver-advertise-address switch would point to a loadbalancer or fqdn otherwise
---
- name: k8s check tcp 6443, fail flag where k8s is already setup
wait_for:
port: 6443
state: stopped
timeout: 5
ignore_errors: true
register: result
- name: k8s initialize
shell:
cmd: kubeadm init --apiserver-advertise-address={{ hostvars[inventory_hostname].ansible_host }} --pod-network-cidr={{ pod_network_range }}
when: "{{ result.failed }} == false" # result.failed true when port 6443 already listening, dont re-init cluster
- name: create ~/.kube config directory
file:
path: "/home/{{ node_account }}/.kube"
state: directory
owner: "{{ node_account }}"
group: "{{ node_account }}"
mode: '0750'
- name: copy kubernets config (with access token) into service account home directory
copy:
src: /etc/kubernetes/admin.conf
dest: "/home/{{ node_account }}/.kube/config"
owner: "{{ node_account }}"
group: "{{ node_account }}"
mode: '0640'
remote_src: yes
- name: download calico network scheme
get_url:
url: https://docs.projectcalico.org/manifests/calico.yaml
dest: "/home/{{ node_account }}/pod_network.yaml"
owner: "{{ node_account }}"
group: "{{ node_account }}"
mode: '0660'
- name: prepare pod network scheme with ip range
replace:
path: "/home/{{ node_account }}/pod_network.yaml"
regexp: '192.168.0.0\/16'
replace: "{{ pod_network_range }}"
- name: apply pod networking scheme
become: yes
become_user: "{{ node_account }}"
shell:
cmd: kubectl apply -f /home/{{ node_account }}/pod_network.yaml
- name: get pod cidr
become: yes
become_user: "{{ node_account }}"
shell:
cmd: kubeadm config view | awk '/podSubnet/ {print $2}'
register: pod_cidr
- name: fail if pod CIDR not applied correctly
assert:
that:
- pod_cidr.stdout == pod_network_range
fail_msg: "pod CIDR {{ pod_cidr.stdout }} should be {{ pod_network_range }}, investigate further"
- name: pause to wait for cluster to init
pause:
seconds: 30
when: "{{ result.failed }} == false"
- name: k8s health check
become: yes
become_user: "{{ node_account }}"
shell:
cmd: kubectl get componentstatus | awk 'NR>1 {print $4}' | awk 'NF>0'
#cmd: kubectl get componentstatus | awk '{print $4}' | awk 'NF>0' # force health check fail
register: k8s_health
- name: fail if k8s components in error state
fail:
msg: k8s in error state, investigate further with "kubectl get componentstatus"
when: k8s_health.stdout | length > 0

21
k8s_dashboard/files/dashboard_rbac.yaml Executable file
View File

@ -0,0 +1,21 @@
---
# service account for dashboard access
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
# cluster role binding to existing role cluster-admin
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard

52
k8s_dashboard/tasks/main.yml Executable file
View File

@ -0,0 +1,52 @@
---
- name: deploy dashboard pod to k8s cluster
become: yes
become_user: "{{ node_account }}"
shell:
cmd: kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc2/aio/deploy/recommended.yaml
- name: copy dashboard rbac to control host
copy:
src: dashboard_rbac.yaml
dest: "/home/{{ node_account }}"
owner: "{{ node_account }}"
group: "{{ node_account }}"
mode: '0640'
- name: apply dashboard rbac to k8s cluster
become: yes
become_user: "{{ node_account }}"
shell:
cmd: kubectl apply -f dashboard_rbac.yaml
- name: get barer token to access dashboard
become: yes
become_user: "{{ node_account }}"
shell:
#cmd: kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}') # not easily parable
#cmd: kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}') --template='{{.data.token}}' | base64 -d # fails with jinja special characters {{ }}, base64 decode required
#cmd: kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}') --template='{{ '{{' }}.data.token{{ '}}' }}' | base64 -d # escape characters, ugly
cmd: kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}') --template={% raw %}'{{.data.token}}'{% endraw %} | base64 -d # civilised
register: barer_token
# - name: print barer token
# debug:
# msg:
# - "barer token to access k8s dashboard"
# - "{{ barer_token.stdout }}"
- name: output bearer token instruction
debug:
msg:
- "ensure your workstation has kubectl installed and a populated ~/.kube/config with rights to the kubernetes-dashboard namespace"
- ""
- "for quick testing you can copy /etc/kubernetes/admin.conf OR /home/ocfadmin/.kube/config @ {{ groups['control'][0] }} to <workstation>:~/.kube/config"
- "then run 'kubectl proxy' on your workstation and open URL; http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/"
- ""
- "when prompted enter the barer token"
- ""
- "{{ barer_token.stdout }}"
- ""
- "to access the url across the local network, run the proxy with lax security"
- ""
- "kubectl proxy --address='0.0.0.0' --accept-hosts='^*'"

21
k8s_dhcp/files/Dockerfile Executable file
View File

@ -0,0 +1,21 @@
FROM alpine:edge
LABEL maintainer="dev@jpillora.com"
# webproc release settings
ENV WEBPROC_VERSION 0.2.2
ENV WEBPROC_URL https://github.com/jpillora/webproc/releases/download/$WEBPROC_VERSION/webproc_linux_amd64.gz
# fetch dnsmasq and webproc binary
RUN apk update \
&& apk --no-cache add dnsmasq \
&& apk add --no-cache --virtual .build-deps curl \
&& curl -sL $WEBPROC_URL | gzip -d - > /usr/local/bin/webproc \
&& chmod +x /usr/local/bin/webproc \
&& apk del .build-deps
#configure dnsmasq
RUN mkdir -p /etc/default/
RUN echo -e "ENABLED=1\nIGNORE_RESOLVCONF=yes" > /etc/default/dnsmasq
RUN mkdir -p /etc/confdnsmasq
COPY dnsmasq.conf /etc/confdnsmasq/dnsmasq.conf
COPY leases.conf /etc/confdnsmasq/leases.conf
#run!
#ENTRYPOINT ["webproc","--config","/etc/confdnsmasq/dnsmasq.conf","--","dnsmasq","--conf-dir=/etc/confdnsmasq","--no-daemon"]
ENTRYPOINT ["webproc","--config","/etc/confdnsmasq/leases.conf","--","dnsmasq","--conf-dir=/etc/confdnsmasq","--no-daemon"]

9
k8s_dhcp/files/dnsmasq.conf Executable file
View File

@ -0,0 +1,9 @@
port=0
dhcp-range=192.168.1.20,192.168.1.100,255.255.255.0,2h
dhcp-option=option:router,192.168.1.1
dhcp-option=option:dns-server,192.168.1.1
#dhcp-authoritative # disable where other dhcp server may exist on the range
dhcp-ignore=tag:!known # only reply to hosts with a static allocation
log-async
log-queries
log-dhcp

16
k8s_dhcp/files/docker-compose.yml Executable file
View File

@ -0,0 +1,16 @@
version: "3"
services:
dnsmasq:
build:
context: .
dockerfile: Dockerfile
image: ocf-dnsmasq:latest
container_name: ocf-dnsmasq
privileged: true
network_mode: host
environment:
- HTTP_USER=admin
- HTTP_PASS=password
ports:
- "67:67/udp"
- "8080:8080/tcp"

1
k8s_dhcp/files/leases.conf Executable file
View File

@ -0,0 +1 @@
#dhcp-host=F2:7E:85:48:29:EB,myhost,192.168.1.30,infinite # example static allocation

79
k8s_dhcp/tasks/main.yml Executable file
View File

@ -0,0 +1,79 @@
- set_fact:
registry_user: "{{ node_account }}" # whole demo uses the same creds on every service, these creds are used for docker login to registry
registry_pass: "{{ node_account_password }}"
dhcp_webapp_user: "{{ node_account }}"
dhcp_webapp_pass: "{{ node_account_password }}"
ip_range_farm_end: "{{ ip_range_farm_start|int + number_farm_nodes|int + 2 }}"
- name: generate docker registry access config with token
become: yes
become_user: root
shell:
cmd: docker login -u {{ registry_user }} -p {{ registry_pass }} {{ registry_service_metallb_ip_endpoint }}:5000
register: registry_login
ignore_errors: yes
- name: check token creation
debug:
msg:
- "docker login failed"
- "is the registry up? {{ registry_service_metallb_ip_endpoint }}:5000"
- "it the registry ssl certificate validating? curl https://{{ registry_service_metallb_ip_endpoint }}:5000"
- "is the CA certificate that signed the registry certificate present on this host? has docker been restarted since installing the certificate?"
- "does the registry htpasswd file contain matching credentials?"
when: '"Login Succeeded" not in registry_login.stdout'
- name: fail token creation
fail:
msg: "failed docker registry client token creation"
when: '"Login Succeeded" not in registry_login.stdout'
- name: get docker registry access token as base64
become: yes
become_user: root
shell:
cmd: base64 -w 0 ~/.docker/config.json
register: registry_token_out
- set_fact:
registry_token: "{{ registry_token_out.stdout }}"
- name: create docker build directory
become: yes
become_user: root
file:
path: ~/ocf-dhcp
state: directory
mode: '0750'
- name: copy build files to directory
become: yes
become_user: root
copy:
src: "{{ item }}"
dest: ~/ocf-dhcp
owner: root
mode: '660'
with_fileglob:
- files/* # relative path for role
# not building with "docker-compose build" (referencing the docker-compose.yml), installing docker-compose from apt currently causes docker login issues (install compose from official docs if required)
# when building manually ensure container is tagged with version (latest for simplicity) and the address of the registry
- name: build docker container then push to registry
become: yes
become_user: root
shell:
cmd: |
docker build -t ocf-dnsmasq:latest ~/ocf-dhcp
docker tag ocf-dnsmasq {{ registry_service_metallb_ip_endpoint }}:5000/ocf-dnsmasq:latest
docker push {{ registry_service_metallb_ip_endpoint }}:5000/ocf-dnsmasq:latest
docker rmi {{ registry_service_metallb_ip_endpoint }}:5000/ocf-dnsmasq:latest ocf-dnsmasq:latest
- name: render dhcp deployment
template:
src: deployment.j2
dest: ~/dhcp_deployment.yaml
- name: apply dhcp configuration
shell:
cmd: kubectl apply -f ~/dhcp_deployment.yaml

144
k8s_dhcp/templates/deployment.j2 Executable file
View File

@ -0,0 +1,144 @@
apiVersion: v1
kind: Namespace
metadata:
name: ocf-dhcp
labels:
name: dev
owner: ocfadmin
stage: dev
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ocf-dhcp-service-account
namespace: ocf-dhcp
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: ocf-dhcp-role
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: ocf-dhcp-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ocf-dhcp-role
subjects:
- kind: ServiceAccount
name: ocf-dhcp-service-account
namespace: ocf-dhcp
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ocf-dhcp-config
namespace: ocf-dhcp
data:
dnsmasq.conf: |
port=0
dhcp-range={{ip_range_24_prefix}}.{{ip_range_farm_start}},{{ip_range_24_prefix}}.{{ip_range_farm_end}},{{ip_range_subnet}},2h
dhcp-option=option:router,{{ip_range_gateway}}
dhcp-option=option:dns-server,{{ip_range_dns}}
#dhcp-authoritative # disable where other dhcp server may exist on the range
dhcp-ignore=tag:!known # only reply to hosts with a static allocation
log-async
log-queries
log-dhcp
leases.conf: |
#dhcp-host=F2:7E:85:48:29:EB,myhost,192.168.1.30,infinite # example static allocation
---
apiVersion: v1
kind: Secret
metadata:
name: ocf-registry-key
namespace: ocf-dhcp
data:
.dockerconfigjson: {{registry_token}}
type: kubernetes.io/dockerconfigjson
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ocf-dhcp
namespace: ocf-dhcp
spec:
replicas: 1
selector:
matchLabels:
app: ocf-dhcp
template:
metadata:
labels:
app: ocf-dhcp
spec:
nodeName: {{groups['worker'][0]}}
hostNetwork: true
restartPolicy: Always
serviceAccount: ocf-dhcp-service-account
serviceAccountName: ocf-dhcp-service-account
terminationGracePeriodSeconds: 10
imagePullSecrets:
- name: ocf-registry-key
containers:
- name: ofc-dhcp
imagePullPolicy: IfNotPresent
image: {{registry_service_metallb_ip_endpoint}}:5000/ocf-dnsmasq:latest
securityContext:
privileged: true
env:
- name: HTTP_USER
value: "{{node_account}}"
- name: HTTP_PASS
value: "{{node_account_password}}"
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "100m"
memory: "512Mi"
volumeMounts:
- mountPath: /etc/confdnsmasq
name: config-volume
ports:
- name: http
protocol: TCP
hostPort: 8080
containerPort: 8080
- name: dhcp
protocol: UDP
hostPort: 67
containerPort: 67
volumes:
- name: config-volume
configMap:
defaultMode: 0750
name: ocf-dhcp-config
items:
- key: dnsmasq.conf
path: dnsmasq.conf
- key: leases.conf
path: leases.conf

13
k8s_metallb/tasks/main.yml Executable file
View File

@ -0,0 +1,13 @@
- name: install MetalLB
shell:
cmd: kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.3/manifests/metallb.yaml
- name: render MetalLB ConfigMap
template:
src: metallb_configmap.j2
dest: ~/metallb_configmap.yaml
- name: apply MetalLB ConfigMap
shell:
cmd: kubectl apply -f metallb_configmap.yaml

12
k8s_metallb/templates/metallb_configmap.j2 Executable file
View File

@ -0,0 +1,12 @@
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- {{metallb_ip_range_start}}-{{metallb_ip_range_end}}

50
k8s_registry/tasks/main.yml Executable file
View File

@ -0,0 +1,50 @@
- name: install htpasswd utility
become: yes
become_user: root
apt:
name: apache2-utils
- set_fact:
htpasswd_user: "{{ node_account }}" # whole demo uses the same creds on every service, these creds are used for docker login to registry
htpasswd_pass: "{{ node_account_password }}"
- name: generate htpasswd file
shell:
cmd: htpasswd -Bbn {{ htpasswd_user }} {{ htpasswd_pass }} > htpasswd
- name: get htpasswd file content
shell:
cmd: cat htpasswd
register: htpasswd_out
- name: get registry certificate content
become: yes
become_user: root
shell:
cmd: cat /root/registry.pem
register: registry_pem_out
- name: get registry certificate key content
become: yes
become_user: root
shell:
cmd: cat /root/registry-key.pem
register: registry_key_out
- set_fact:
htpasswd: "{{ htpasswd_out.stdout }}"
registry_cert: "{{ registry_pem_out.stdout }}"
registry_key: "{{ registry_key_out.stdout }}"
- name: render registry deployment
template:
src: deployment.j2
dest: ~/registry_deployment.yaml
- name: apply registry deployment
shell:
cmd: kubectl apply -f ~/registry_deployment.yaml
- name: pause to wait for registry container to init
pause:
seconds: 30

326
k8s_registry/templates/deployment.j2 Executable file
View File

@ -0,0 +1,326 @@
apiVersion: v1
kind: Namespace
metadata:
name: ocf-registry
labels:
name: dev
owner: ocfadmin
stage: dev
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ocf-registry-service-account
namespace: ocf-registry
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: ocf-registry-role
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: ocf-registry-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ocf-registry-role
subjects:
- kind: ServiceAccount
name: ocf-registry-service-account
namespace: ocf-registry
---
apiVersion: v1
kind: Service
metadata:
namespace: ocf-registry
name: ocf-registry-service-5000
annotations:
metallb.universe.tf/address-pool: default
#metallb.universe.tf/allow-shared-ip: ocf-registry-sharing-key # can share different services to same deployment when wanting TCP + UDP, cannot share for two different deployments
spec:
selector:
app: ocf-registry
ports:
- port: 5000
targetPort: 5000
protocol: TCP
externalTrafficPolicy: Local
type: LoadBalancer
loadBalancerIP: {{registry_service_metallb_ip_endpoint}}
---
apiVersion: v1
kind: Service
metadata:
namespace: ocf-registry
name: ocf-minio-service-9000
annotations:
metallb.universe.tf/address-pool: default
#metallb.universe.tf/allow-shared-ip: ocf-registry-sharing-key
spec:
selector:
app: ocf-minio
ports:
- port: 9000
targetPort: 9000
protocol: TCP
externalTrafficPolicy: Local
type: LoadBalancer
loadBalancerIP: {{minio_service_metallb_ip_endpoint}}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ocf-registry-config
namespace: ocf-registry
data:
htpasswd: |
{{htpasswd}}
registry.pem: |
{{registry_cert | indent( width=6, indentfirst=False)}}
registry-key.pem: |
{{registry_key | indent( width=6, indentfirst=False)}}
createbucket.sh: |
#!/bin/bash
until nc -z {{minio_service_metallb_ip_endpoint}} 9000
do
echo "waiting for minio"
sleep 1
done
sleep 3
/usr/bin/mc config host add minioinstance http://{{minio_service_metallb_ip_endpoint}}:9000 minio Password0
#/usr/bin/mc rm -r --force minioinstance/docker # dont want to delete every start, easy to enable through k8s dashboard edit console and re-run
/usr/bin/mc mb minioinstance/docker
/usr/bin/mc policy set download minioinstance/docker
exit 0
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-pv-volume
namespace: ocf-registry
labels:
type: local
spec:
storageClassName: manual
#nodeAffinity: worker-01 # will influence pod affinity, really should be using network backed volume so pods can fail over to other nodes
capacity:
storage: 4Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/opt/minio_data1"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: minio-pv-claim
namespace: ocf-registry
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ocf-minio
namespace: ocf-registry
spec:
replicas: 1
selector:
matchLabels:
app: ocf-minio
template:
metadata:
labels:
app: ocf-minio
spec:
restartPolicy: Always
serviceAccount: ocf-registry-service-account
serviceAccountName: ocf-registry-service-account
terminationGracePeriodSeconds: 10
containers:
- name: ocf-minio
imagePullPolicy: IfNotPresent
image: minio/minio:latest
env:
- name: MINIO_ACCESS_KEY
value: "minio"
- name: MINIO_SECRET_KEY
value: "Password0"
resources:
requests:
cpu: "500m"
memory: "1024Mi"
limits:
cpu: "500m"
memory: "1024Mi"
volumeMounts:
- mountPath: "/data1"
name: minio-pv-storage
ports:
- name: http
protocol: TCP
containerPort: 9000
args:
- server
- /data1
volumes:
- name: minio-pv-storage
persistentVolumeClaim:
claimName: minio-pv-claim
---
# one time job to create minio bucket
apiVersion: batch/v1
kind: Job
metadata:
name: ocf-minio-bucket
namespace: ocf-registry
spec:
template:
spec:
serviceAccount: ocf-registry-service-account
serviceAccountName: ocf-registry-service-account
restartPolicy: OnFailure
containers:
- name: ocf-create-bucket
imagePullPolicy: IfNotPresent
image: minio/mc:latest
command:
- /bin/sh
- /tmp/createbucket.sh
volumeMounts:
- mountPath: "/tmp"
name: config-volume
volumes:
- name: config-volume
configMap:
defaultMode: 0750
name: ocf-registry-config
items:
- key: createbucket.sh
path: createbucket.sh
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ocf-registry
namespace: ocf-registry
spec:
replicas: 1
selector:
matchLabels:
app: ocf-registry
template:
metadata:
labels:
app: ocf-registry
spec:
restartPolicy: Always
serviceAccount: ocf-registry-service-account
serviceAccountName: ocf-registry-service-account
terminationGracePeriodSeconds: 10
containers:
- name: ocf-registry
imagePullPolicy: IfNotPresent
image: registry:latest
env:
- name: REGISTRY_HTTP_TLS_CERTIFICATE
value: /certs/domain.crt
- name: REGISTRY_HTTP_TLS_KEY
value: /certs/domain.key
- name: REGISTRY_HTTP_ADDR
value: 0.0.0.0:5000
- name: REGISTRY_AUTH
value: htpasswd
- name: REGISTRY_AUTH_HTPASSWD_REALM
value: basic-realm
- name: REGISTRY_AUTH_HTPASSWD_PATH
value: /certs/htpasswd
- name: REGISTRY_STORAGE_REDIRECT_DISABLE
value: "true"
- name: REGISTRY_STORAGE
value: s3
- name: REGISTRY_STORAGE_S3_ACCESSKEY
value: minio
- name: REGISTRY_STORAGE_S3_SECRETKEY
value: Password0
- name: REGISTRY_STORAGE_S3_REGION
value: us-east-1
- name: REGISTRY_STORAGE_S3_REGIONENDPOINT
value: http://{{minio_service_metallb_ip_endpoint}}:9000
- name: REGISTRY_STORAGE_S3_BUCKET
value: docker
- name: REGISTRY_STORAGE_S3_SECURE
value: "true"
- name: REGISTRY_STORAGE_S3_V4AUTH
value: "true"
- name: REGISTRY_STORAGE_S3_CHUNKSIZE
value: "5242880"
- name: REGISTRY_STORAGE_S3_ROOTDIRECTORY
value: /
- name: REGISTRY_STORAGE_S3_DELETE_ENABLED
value: "true"
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "500m"
memory: "512Mi"
ports:
- name: http
protocol: TCP
containerPort: 5000
volumeMounts:
- mountPath: "/certs"
name: config-cert
volumes:
- name: config-cert
configMap:
defaultMode: 0750
name: ocf-registry-config
items:
- key: htpasswd
path: htpasswd
- key: registry.pem
path: domain.crt
- key: registry-key.pem
path: domain.key

87
k8s_worker_init/tasks/main.yml Executable file
View File

@ -0,0 +1,87 @@
---
- name: generate access token
become: yes
become_user: "{{ node_account }}"
shell:
cmd: kubeadm token create
delegate_to: "{{ groups['control'][0] }}"
register: k8s_token_out
- name: get CA hash
become: yes
become_user: "{{ node_account }}"
shell:
cmd: openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
delegate_to: "{{ groups['control'][0] }}"
register: k8s_ca_hash_out
- set_fact:
k8s_token: "{{ k8s_token_out.stdout }}"
k8s_ca_hash: "{{ k8s_ca_hash_out.stdout }}"
# - debug:
# msg:
# - "{{ k8s_token_out.stdout }}"
# - "{{ k8s_ca_hash_out.stdout }}"
# the only task that really runs on the worker node
- name: register node
shell:
cmd: kubeadm join {{ hostvars[groups['control'][0]]['ansible_host'] }}:6443 --token {{ k8s_token }} --discovery-token-ca-cert-hash sha256:{{ k8s_ca_hash }}
- name: pause to wait for worker nodes to init # really there should be a health check here as in k8s_control_init to loop the next few tasks
pause:
seconds: 90
- name: check nodes online
become: yes
become_user: "{{ node_account }}"
shell:
cmd: kubectl get nodes -o json
delegate_to: "{{ groups['control'][0] }}"
register: k8s_out
- name: extract json
set_fact:
jsonnodes: "{{ k8s_out.stdout | from_json }}"
- name: get ready nodes from query of json
set_fact:
nodes: "{{ jsonnodes | json_query(jmesquery) }}"
vars:
jmesquery: "items[?(@.kind == 'Node') && status.conditions[?(@.type=='Ready') && (@.status=='True')]].metadata.name" # @ is current element reference, only gets node name when conditions met
delegate_to: localhost
- debug:
msg: "{{ nodes }}"
- name: build a list of ready nodes
set_fact:
ansible_ready_nodes: "{{ ansible_ready_nodes | default([]) + [item] }}"
loop: "{{ nodes }}"
delegate_to: localhost
- name: build a list of target nodes
set_fact:
ansible_target_nodes: "{{ ansible_target_nodes | default([]) + [item] }}"
loop: "{{ groups['worker'] }}"
delegate_to: localhost
- name: find unready nodes
set_fact:
unready_node: "{{ unready_node | default([]) + [item] }}"
loop: "{{ ansible_target_nodes }}"
when: item not in ansible_ready_nodes
delegate_to: localhost
# - name: spoof fail condition
# set_fact:
# unready_node: "{{ unready_node + ['worker-N'] }}"
# delegate_to: localhost
- name: fail with unready node
fail:
msg: "node {{ unready_node | list | join(', ') }} in a unready state, did worker node join the cluster?"
when: unready_node | length > 0
delegate_to: localhost

4
k8s_worker_init/vars/main.yml Executable file
View File

@ -0,0 +1,4 @@
---
ansible_target_nodes: []
ansible_ready_nodes: []
unready_node: []

128
proxmox_farm_provision/files/ks-example.cfg Executable file
View File

@ -0,0 +1,128 @@
install
url --url http://mirror.freethought-internet.co.uk/centos/7.7.1908/os/x86_64/
keyboard 'uk'
# Root password is 12345678
rootpw --iscrypted $6$EM1co8P73TyS65Cn$wZxxE6aw0TSlQs//FCVmSA/QR8QGaroYLH3kYpbZ0wJTGrIdKHtVX0zItAkEKvAchtwkBL5Ommo2fVJxv.kI0/
group --name=ocfadmin --gid=1000
user --groups=ocfadmin --homedir=/home/ocfadmin --shell=/bin/bash --name=ocfadmin --uid=1000 --gid=1000 --password=$6$EM1co8P73TyS65Cn$wZxxE6aw0TSlQs//FCVmSA/QR8QGaroYLH3kYpbZ0wJTGrIdKHtVX0zItAkEKvAchtwkBL5Ommo2fVJxv.kI0/ --iscrypted
timezone Europe/London --isUtc
lang en_GB
firewall --disabled
selinux --disabled
auth --useshadow --passalgo=sha512
text
skipx
firstboot --disable
reboot
bootloader --location=mbr
zerombr
clearpart --all clearpart
part /boot --asprimary --fstype="xfs" --ondisk=sda --size=250
part / --asprimary --fstype="xfs" --ondisk=sda --size=4096
part pv.2 --size=0 --grow --ondisk=sda
volgroup vg0 pv.2
logvol /usr --fstype="xfs" --name=lv_usr --vgname=vg0 --size=4092
logvol /var --fstype="xfs" --name=lv_var --vgname=vg0 --size=8192
logvol /home --fstype="xfs" --name=lv_home --vgname=vg0 --size=4096
logvol /opt --fstype="xfs" --name=lv_opt --vgname=vg0 --size=4096 --grow
logvol swap --fstype="swap" --name=lv_swap --vgname=vg0 --size=1024
# Configure network for CNDN interface naming scheme
%include /tmp/network.ks
%pre
ip addr | grep -i broadcast | awk '{ print $2 }' > /tmp/interface
sed -i 's/:/\ /g' /tmp/interface
interface=`cat /tmp/interface`
echo "network --bootproto static --device $interface --ip 192.168.140.81 --netmask 255.255.255.0 --gateway 192.168.140.2 --noipv6 --nodns --nameserver=192.168.140.2 --hostname=VLSDEVAPP02 --activate" >/tmp/network.ks
%end
services --enabled=NetworkManager,sshd,chronyd
eula --agreed
# Disable kdump
%addon com_redhat_kdump --disable
%end
%packages --ignoremissing
@core
@base
-iwl6000g2b-firmware
-iwl7260-firmware
-iwl105-firmware
-iwl135-firmware
-alsa-firmware
-iwl2030-firmware
-iwl2000-firmware
-iwl3160-firmware
-alsa-tools-firmware
-aic94xx-firmware
-atmel-firmware
-b43-openfwwf
-bfa-firmware
-ipw2100-firmware
-ipw2200-firmware
-ivtv-firmware
-iwl100-firmware
-iwl1000-firmware
-iwl3945-firmware
-iwl4965-firmware
-iwl5000-firmware
-iwl5150-firmware
-iwl6000-firmware
-iwl6000g2a-firmware
-iwl6050-firmware
-libertas-usb8388-firmware
-ql2100-firmware
-ql2200-firmware
-ql23xx-firmware
-ql2400-firmware
-ql2500-firmware
-rt61pci-firmware
-rt73usb-firmware
-xorg-x11-drv-ati-firmware
-zd1211-firmware
%end
%post --log=/root/kickstart.log
yum install -y nano \
chrony \
postfix \
mailx \
sharutils \
dos2unix \
gzip \
bzip2 \
which \
mlocate \
iproute \
net-tools \
traceroute \
ethtool \
tcpdump \
nmap-ncat \
telnet \
iptraf-ng \
procps-ng \
lsof \
iotop \
sysstat \
psacct \
dstat \
coreutils \
ncurses \
wget \
curl \
rsync \
lftp \
time \
dmidecode \
pciutils \
usbutils \
util-linux \
yum-utils
yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sed -i -e '/\[epel\]/{:a;n;/^$/!ba;i\includepkgs=htop' -e '}' /etc/yum.repos.d/epel.repo
yum -y install htop
%end

90
proxmox_farm_provision/tasks/main.yml Executable file
View File

@ -0,0 +1,90 @@
---
- set_fact:
dhcp_webapp_user: "{{ node_account }}"
dhcp_webapp_pass: "{{ node_account_password }}"
ip_range_farm_end: "{{ ip_range_farm_start|int + number_farm_nodes|int + 2 }}"
- name: add proxmox host to in-memory inventory
add_host: >
name=proxmox_server
groups=proxmox
ansible_ssh_host={{ proxmox_host }}
ansible_host={{ proxmox_host }}
ansible_ssh_user={{ proxmox_ssh_user }}
ansible_user={{ proxmox_ssh_user }}
ansible_ssh_pass="{{ proxmox_ssh_pass }}"
- name: provision nodes on proxmox kvm host
include_tasks: provision.yml
with_sequence:
- start=1 end={{ number_farm_nodes }} format=farm-%s
# inventory is only read at runtime, used aid debug so we dont have to re-provision hosts as we update this playbook
- name: create farm inventory
template:
src: ansible-hosts.j2
dest: "inventory/farm.ini"
# used in debug to test kickstart of without rebuilding farm nodes
# - debug:
# msg: "{{ farm_leases }}"
# used in debug to test kickstart of without rebuilding farm nodes
# - set_fact:
# farm_leases: ["dhcp-host=62:d6:f3:07:c9:69,farm-01,192.168.101.61,infinite","dhcp-host=ca:59:33:b3:cc:3d,farm-02,192.168.101.62,infinite","dhcp-host=ee:2b:95:2c:5e:fc,farm-03,192.168.101.63,infinite"]
- name: render dhcp configmap
template:
src: update_configmap.j2
dest: ~/update_dhcp_configmap.yaml
delegate_to: "{{ groups['control'][0] }}"
- name: render dhcp job
template:
src: update_job.j2
dest: ~/update_dhcp_job.yaml
delegate_to: "{{ groups['control'][0] }}"
# a job will not restart if updated (or its configmap is updated), it will be re-run when replaced. jobs are designed to run once when deployed
- name: remove dhcp job, ensure k8s job is invoked with with configmap change (expect this to fail on first run)
shell:
cmd: kubectl delete job ocf-ipxe-bucket -n ocf-dhcp
ignore_errors: yes
delegate_to: "{{ groups['control'][0] }}"
- name: apply dhcp configmap and job (ipxe minio bucket creation)
shell:
cmd: |
kubectl apply -f ~/update_dhcp_configmap.yaml
kubectl apply -f ~/update_dhcp_job.yaml
delegate_to: "{{ groups['control'][0] }}"
# in a prod env you'd use a service like wave to ensure configmaps and secret changes auto rollout new pods
- name: restart pod for dhcp leases to become live (new configmap)
shell:
cmd: |
kubectl scale deployment ocf-dhcp --replicas=0 -n ocf-dhcp
kubectl scale deployment ocf-dhcp --replicas=1 -n ocf-dhcp
delegate_to: "{{ groups['control'][0] }}"
- name: start nodes on proxmox kvm host
proxmox_kvm:
api_user : "{{ proxmox_user }}"
api_password: "{{ proxmox_pass }}"
api_host : "{{ proxmox_host }}"
node : "{{ proxmox_node }}"
name : "{{ item }}"
state : started
with_items:
- "{{ groups['farm'] }}"
- debug:
msg:
- "minio"
- " http://{{minio_service_metallb_ip_endpoint}}:9000"
- " u:minio"
- " p:{{ node_account_password }}"
- ""
- "dnsmasq live log"
- " http://{{ hostvars[groups['worker'][0]]['ansible_host'] }}:8080"
- " u:{{ node_account }}"
- " p:{{ node_account_password }}"

89
proxmox_farm_provision/tasks/provision.yml Executable file
View File

@ -0,0 +1,89 @@
---
- name: generate instance vars
set_fact:
node_type: "{{ item.split('-')[0] }}"
node_number: "{{ item.split('-')[1] }}"
node_name: "{{ item.split('-')[0] }}-{{ '%02d' | format (item.split('-')[1]|int) }}"
node_ip: "{{ip_range_24_prefix}}.{{ (ip_range_farm_start|int + item.split('-')[1]|int) }}"
- name: get proxmox API auth cookie
uri:
url: "https://{{ proxmox_host }}:8006/api2/json/access/ticket"
validate_certs: no
method: POST
body_format: form-urlencoded
body:
username: "{{ proxmox_user }}"
password: "{{ proxmox_pass }}"
status_code: 200
register: login
- name: query proxmox next free vmid
uri:
url: "https://{{ proxmox_host }}:8006/api2/json/cluster/nextid"
validate_certs: no
method: GET
headers:
Cookie: "PVEAuthCookie={{ login.json.data.ticket }}"
CSRFPreventionToken: "{{ login.json.data.CSRFPreventionToken }}"
register: next_pvid
- name: provision nodes on proxmox kvm host
proxmox_kvm:
api_user : "{{ proxmox_user }}"
api_password: "{{ proxmox_pass }}"
api_host : "{{ proxmox_host }}"
node : "{{ proxmox_node }}"
vmid : "{{ next_pvid.json.data }}"
boot : cn # n network,d cdrom,c harddisk, combine in any order cdn
kvm : yes
agent : yes
name : "{{ node_name }}"
sockets : 1
cores : 2
memory : 2048 # rhel 7 min requirement for kickstart now
vga : std
scsihw : virtio-scsi-single
virtio : '{"virtio0":"{{ proxmox_vm_datastore }}:10,format=raw"}'
net : '{"net0":"virtio,bridge={{ proxmox_vmbr }},firewall=0"}'
ostype : l26
state : present
#ide : '{"ide0":"{{ proxmox_iso_datastore }}:iso/{{ iso_image_file }},media=cdrom"}'
register: _result
- name: end run with failure where host(s) pre-exist
fail:
msg: "node {{ node_name }} already exists"
when: _result.msg == "VM with name <{{ node_name }}> already exists"
- name: add node to in-memory inventory
add_host: >
name="{{ node_name }}"
groups="{{ node_type }}"
ansible_host="{{ node_ip }}"
ansible_ssh_user="{{ node_account }}"
ansible_ssh_pass="{{ node_account_password }}"
ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
- name: get node MAC
uri:
url: "https://{{ proxmox_host }}:8006/api2/json/nodes/{{ proxmox_node }}/qemu/{{ next_pvid.json.data }}/config"
validate_certs: no
method: GET
headers:
Cookie: "PVEAuthCookie={{ login.json.data.ticket }}"
CSRFPreventionToken: "{{ login.json.data.CSRFPreventionToken }}"
register: _result
- name: register node MAC
set_fact:
node_mac: "{{ ((_result.json.data.net0).split(',')[0]).split('=')[1] | lower }}"
# mac must be lower (already is) as ipxe returns ${net0/mac:hexhyp} lowercase
- name: build dhcp lease
set_fact:
lease: "dhcp-host={{ node_mac|lower }},{{ node_name }},{{ node_ip }},infinite"
- name: add dhcp lease to list
set_fact:
farm_leases: "{{ farm_leases | default([]) + [lease] }}"

10
proxmox_farm_provision/templates/ansible-hosts.j2 Executable file
View File

@ -0,0 +1,10 @@
[farm]
{% for entry in groups['farm'] %}
{% set ip = hostvars[entry].ansible_host -%}
{{ entry }} ansible_host={{ ip }}
{% endfor %}
[all:vars]
ansible_password={{ node_account_password }}
ansible_user={{ node_account }}
ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"

137
proxmox_farm_provision/templates/update_configmap.j2 Executable file
View File

@ -0,0 +1,137 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ocf-dhcp-config
namespace: ocf-dhcp
data:
dnsmasq.conf: |
port=0
dhcp-range={{ip_range_24_prefix}}.{{ip_range_farm_start}},{{ip_range_24_prefix}}.{{ip_range_farm_end}},{{ip_range_subnet}},2h
dhcp-option=option:router,{{ip_range_gateway}}
dhcp-option=option:dns-server,{{ip_range_dns}}
#dhcp-authoritative # disable where other dhcp server may exist on the range
dhcp-ignore=tag:!known # only reply to hosts with a static allocation
dhcp-match=set:ipxe,175
dhcp-boot=tag:ipxe,http://{{minio_service_metallb_ip_endpoint}}:9000/ipxe/farm.ipxe
log-async
log-queries
log-dhcp
leases.conf: |
{% for item in farm_leases %}
{{item | indent( width=6, indentfirst=True)}}
{% endfor %}
createbucket.sh: |
#!/bin/bash
until nc -z {{minio_service_metallb_ip_endpoint}} 9000
do
echo "waiting for minio"
sleep 1
done
sleep 3
/usr/bin/mc config host add minioinstance http://{{minio_service_metallb_ip_endpoint}}:9000 minio Password0
/usr/bin/mc rm -r --force minioinstance/ipxe
/usr/bin/mc mb minioinstance/ipxe
/usr/bin/mc policy set public minioinstance/ipxe
/usr/bin/mc cp /tmp/farm.ipxe minioinstance/ipxe
/usr/bin/mc cp /tmp/*.cfg minioinstance/ipxe
exit 0
farm.ipxe: |
#!ipxe
kernel http://anorien.csc.warwick.ac.uk/mirrors/CentOS-vault/7.1.1503/os/x86_64/isolinux/vmlinuz ks=http://{{minio_service_metallb_ip_endpoint}}:9000/ipxe/${net0/mac:hexhyp}.cfg
initrd http://anorien.csc.warwick.ac.uk/mirrors/CentOS-vault/7.1.1503/os/x86_64/isolinux/initrd.img
boot
{% for item in farm_leases %}
{% set mac = ((item.split(',')[0]).split('=')[1]).replace(':','-') %}
{% set host = item.split(',')[1] %}
{% set ip = item.split(',')[2] %}
{{mac}}.cfg: |
install
url --url http://anorien.csc.warwick.ac.uk/mirrors/CentOS-vault/7.1.1503/os/x86_64/
keyboard 'uk'
# Root password is 12345678
rootpw --iscrypted $6$EM1co8P73TyS65Cn$wZxxE6aw0TSlQs//FCVmSA/QR8QGaroYLH3kYpbZ0wJTGrIdKHtVX0zItAkEKvAchtwkBL5Ommo2fVJxv.kI0/
group --name={{node_account}} --gid=1000
user --groups={{node_account}} --homedir=/home/{{node_account}} --shell=/bin/bash --name={{node_account}} --uid=1000 --gid=1000 --password=$6$EM1co8P73TyS65Cn$wZxxE6aw0TSlQs//FCVmSA/QR8QGaroYLH3kYpbZ0wJTGrIdKHtVX0zItAkEKvAchtwkBL5Ommo2fVJxv.kI0/ --iscrypted
timezone Europe/London --isUtc
lang en_GB
firewall --disabled
selinux --disabled
auth --useshadow --passalgo=sha512
text
skipx
firstboot --disable
reboot
bootloader --location=mbr
zerombr
clearpart --all clearpart
part /boot --asprimary --fstype="xfs" --ondisk=vda --size=250
part / --asprimary --fstype="xfs" --ondisk=vda --size=4096
part pv.2 --size=0 --grow --ondisk=vda
volgroup vg0 pv.2
logvol /home --fstype="xfs" --name=lv_home --vgname=vg0 --size=1024 --grow
logvol swap --fstype="swap" --name=lv_swap --vgname=vg0 --size=1024
# Configure network for CNDN interface naming scheme
%include /tmp/network.ks
%pre
ip addr | grep -i broadcast | awk '{ print $2 }' > /tmp/interface
sed -i 's/:/\ /g' /tmp/interface
interface=`cat /tmp/interface`
echo "network --bootproto static --device $interface --ip {{ip}} --netmask {{ip_range_subnet}} --gateway {{ip_range_gateway}} --noipv6 --nodns --nameserver={{ip_range_dns}} --hostname={{host}} --activate" >/tmp/network.ks
%end
services --enabled=NetworkManager,sshd,chronyd
eula --agreed
# Disable kdump
%addon com_redhat_kdump --disable
%end
%packages --ignoremissing
@core
@base
-iwl6000g2b-firmware
-iwl7260-firmware
-iwl105-firmware
-iwl135-firmware
-alsa-firmware
-iwl2030-firmware
-iwl2000-firmware
-iwl3160-firmware
-alsa-tools-firmware
-aic94xx-firmware
-atmel-firmware
-b43-openfwwf
-bfa-firmware
-ipw2100-firmware
-ipw2200-firmware
-ivtv-firmware
-iwl100-firmware
-iwl1000-firmware
-iwl3945-firmware
-iwl4965-firmware
-iwl5000-firmware
-iwl5150-firmware
-iwl6000-firmware
-iwl6000g2a-firmware
-iwl6050-firmware
-libertas-usb8388-firmware
-ql2100-firmware
-ql2200-firmware
-ql23xx-firmware
-ql2400-firmware
-ql2500-firmware
-rt61pci-firmware
-rt73usb-firmware
-xorg-x11-drv-ati-firmware
-zd1211-firmware
%end
{% endfor %}

29
proxmox_farm_provision/templates/update_job.j2 Executable file
View File

@ -0,0 +1,29 @@
---
# one time job to create minio bucket
apiVersion: batch/v1
kind: Job
metadata:
name: ocf-ipxe-bucket
namespace: ocf-dhcp
spec:
template:
spec:
serviceAccount: ocf-dhcp-service-account
serviceAccountName: ocf-dhcp-service-account
restartPolicy: OnFailure
containers:
- name: ocf-create-ipxe-bucket
imagePullPolicy: IfNotPresent
image: minio/mc:RELEASE.2020-03-14T01-23-37Z
command:
- /bin/sh
- /tmp/createbucket.sh
volumeMounts:
- mountPath: "/tmp"
name: config-volume
volumes:
- name: config-volume
configMap:
defaultMode: 0750
name: ocf-dhcp-config

280
proxmox_node_provision/tasks/main.yml Executable file
View File

@ -0,0 +1,280 @@
# needed ansible from official ppa (not vanilla apt) due to bug in older ansible included proxmox_kvm module
# requires pip installed python modules - requests, proxmoxer
#
# this task is not idempotent, we arent spefifying the vmid, rather taking the next available (we could query pmox for the node name then query that vmid and remove the node)
# for safety we cannot add additional nodes with the same name so exit gracefully
#
# ideally we want to not use ssh transport only the proxmox API, however qm import isn't yet supported in the API
#
---
- set_fact:
image_name: "{{ ((image_url | urlsplit).path).split ('/')[-1] }}"
- name: add proxmox host to in-memory inventory
add_host: >
name=proxmox_server
groups=proxmox
ansible_ssh_host={{ proxmox_host }}
ansible_host={{ proxmox_host }}
ansible_ssh_user={{ proxmox_ssh_user }}
ansible_user={{ proxmox_ssh_user }}
ansible_ssh_pass="{{ proxmox_ssh_pass }}"
- name: generate password hash for cloud-init user-data
set_fact:
node_account_password_hash: "{{ node_account_password | password_hash('sha512') }}"
- name: check if raw cloud-init image is present
stat:
path: "{{ proxmox_img_datastore_path }}/template/iso/raw.{{ image_name }}"
register: raw_img_remote_attributes
delegate_to: proxmox_server
# the ubuntu cloud image is in qcow2 sparse format, it should be in img/iso or raw format for qm importdisk (we could check image format and do convert action conditionally)
# the API only doesnt allow qcow file suffix to be uploaded
- name: convert qcow2 to raw
shell:
cmd: |
qemu-img convert -f qcow2 -O raw {{ image_name }} raw.{{ image_name }}
chdir: "{{ proxmox_img_datastore_path }}/template/iso/"
#register: _result
delegate_to: proxmox_server
when: raw_img_remote_attributes.stat.exists == false
- name: provision nodes on proxmox kvm host
include_tasks: provision.yml # loop sequential list of dependent tasks (ansible block module doesnt work with loops)
with_sequence:
- start=1 end={{ number_control_nodes }} format=control-%s
- start=1 end={{ number_worker_nodes }} format=worker-%s
# inventory is only read at runtime
- name: create ansible inventory - useful if we want to disable build node roles and proceed with only subseqent roles on an already built cluster
template:
src: ansible-hosts.j2
dest: "inventory/nodes.ini"
# - name: empty list for nodes
# set_fact:
# nodes_attributes: {}
# - debug:
# msg: "{{ nodes_attributes }}"
# - debug:
# msg: "{{ vars }}"
# - name: create file
# template:
# src: ansible-hosts.j2
# dest: "inventory/nodes.ini"
# - set_fact:
# stuff:
# controlnode:
# control01: 192.168.140.71
# workernode:
# worker01: 192.168.140.81
# worker02: 192.168.140.82
# worker03: 192.168.140.83
# - set_fact:
# stuff1:
# - controlnode:
# - control01: 192.168.140.71
# - workernode:
# - worker01: 192.168.140.81
# worker02: 192.168.140.82
# worker03: 192.168.140.83
# - set_fact:
# stuff: "{{ stuff }}"
# - set_fact:
# parent_dict: [{'A':'val1','B':'val2'},{'C':'val3','D':'val4'}]
# - set_fact:
# parent_dict1:
# - A: val1
# B: val2
# - C: val3
# D: val4
# - debug:
# msg: "{{ parent_dict }}"
# - debug:
# msg: "{{ parent_dict1 }}"
# - set_fact:
# stuff: "{{ nodes_attributes }}"
# - debug:
# msg: "{{ stuff1 }}"
# ok: [localhost] => {
# "msg": {
# "control": {
# "control-01": "192.168.140.71"
# },
# "worker": {
# "worker-01": "192.168.140.81",
# "worker-02": "192.168.140.82",
# "worker-03": "192.168.140.83"
# }
# }
# }
# user our list of lists to now loop and create an inventory - i think we can make a dict (long hand), populate it with 'item.' and push out to yaml thus a yaml inventory file?
# will the next task see a yaml inventory file or is it only parsed at init?
#### notes
# - name: get proxmox API auth cookie
# uri:
# url: "https://{{ proxmox_host }}:8006/api2/json/access/ticket"
# validate_certs: no
# method: POST
# body_format: form-urlencoded
# body:
# username: "{{ proxmox_user }}"
# password: "{{ proxmox_pass }}"
# status_code: 200
# register: login
# - name: query proxmox next free vmid
# uri:
# url: "https://{{ proxmox_host }}:8006/api2/json/cluster/nextid"
# validate_certs: no
# method: GET
# headers:
# Cookie: "PVEAuthCookie={{ login.json.data.ticket }}"
# CSRFPreventionToken: "{{ login.json.data.CSRFPreventionToken }}"
# register: next_pvid
# - name: provision cloud-init node on proxmox kvm host
# proxmox_kvm:
# api_user : "{{ proxmox_user }}"
# api_password: "{{ proxmox_pass }}"
# api_host : "{{ proxmox_host }}"
# node : "{{ proxmox_node }}"
# vmid : "{{ next_pvid.json.data }}"
# boot : c # n network,d cdrom,c harddisk, combine in any order cdn
# kvm : yes
# agent : yes
# name : test
# vcpus : 1
# cores : 2
# memory : 512
# vga : std
# scsihw : virtio-scsi-single
# #virtio : '{"virtio0":"{{ proxmox_vm_datastore }}:10,format=raw"}' # not adding a disk here
# #net : '{"net0":"virtio=d0:3a:3b:d9:40:7d,bridge=vmbr1,tag=2,firewall=0"}' # keep for later when we start multiple nodes with specific macs
# net : '{"net0":"virtio,bridge=vmbr1,tag=2,firewall=0"}'
# ostype : l26
# state : present
## need a task to "qm importdisk 9000 bionic-server-cloudimg-amd64.img local-lvm" -- how?
## then "qm set 9000 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-9000-disk-1" -- can be done with proxmox_kvm update command
## how do we expand storage then? -- HTTP: PUT /api2/json/nodes/{node}/qemu/{vmid}/resize
## resize https://stackoverflow.com/questions/55323050/how-i-resize-hard-disk-with-ansible-module-proxmox-kvm
## MAYBE we have to create one host then clone it X times!
## create own cloud-init iso https://www.historiantech.com/automatically-provision-fortigate-vm-with-cloud-init-in-proxmox/
# - name: provision nodes on proxmox kvm host
# proxmox_kvm:
# api_user : "{{ proxmox_user }}"
# api_password: "{{ proxmox_pass }}"
# api_host : "{{ proxmox_host }}"
# node : "{{ proxmox_node }}"
# boot : cd # n network,d cdrom,c harddisk, combine in any order cdn
# kvm : yes
# agent : yes
# name : "{{ item }}"
# vcpus : 1
# cores : 2
# memory : 512
# vga : std
# scsihw : virtio-scsi-single
# virtio : '{"virtio0":"{{ proxmox_vm_datastore }}:10,format=raw"}'
# #net : '{"net0":"virtio=d0:3a:3b:d9:40:7d,bridge=vmbr1,tag=2,firewall=0"}' # keep for later when we start multiple nodes with specific macs
# net : '{"net0":"virtio,bridge=vmbr1,tag=2,firewall=0"}'
# ostype : l26
# state : present
# args : '-append ks=url/path/to/file.ks'
# #args : "console=ttyS0,115200n8 serial"
# ide : '{"ide0":"{{ proxmox_iso_datastore }}:iso/{{ iso_image_file }},media=cdrom"}'
# with_sequence:
# - start=1 end={{ number_control_nodes }} format=control-%02x
# - start=1 end={{ number_worker_nodes }} format=worker-%02x
# - name: start nodes on proxmox kvm host
# proxmox_kvm:
# api_user : "{{ proxmox_user }}"
# api_password: "{{ proxmox_pass }}"
# api_host : "{{ proxmox_host }}"
# node : "{{ proxmox_node }}"
# name : "{{ item }}"
# state : started
# with_sequence:
# - start=1 end={{ number_control_nodes }} format=control-%02x
# - start=1 end={{ number_worker_nodes }} format=worker-%02x
# run up a web server with a preseed hosted https://www.reddit.com/r/ansible/comments/9sys65/setting_up_an_asynchronous_tasks_simplehttpserver/
# start hosts with ability to get preseed (should not have ip on install only after reboot) + ip
# poll ip - continue/timeout
# handler kill web server
#
# start new role here - why? if hosts have timed out (bad ip etc) then been fixed the first role can be disabled and the rest of the playbook continues
# use this new feature - we want async tasks which are killed on a sthreaded failure
# https://stackoverflow.com/questions/35892455/execute-task-or-handler-if-any-task-failed
# run up a web server with a preseed hosted https://www.reddit.com/r/ansible/comments/9sys65/setting_up_an_asynchronous_tasks_simplehttpserver/
# start hosts with ability to get preseed (should not have ip on install only after reboot) + ip
# poll ip - until - continue/timeout
# handler kill web server
#
# start new role here - why? if hosts have timed out (bad ip etc) then been fixed the first role can be disabled and the rest of the playbook continues

154
proxmox_node_provision/tasks/provision.yml Executable file
View File

@ -0,0 +1,154 @@
---
- name: generate instance vars
set_fact:
node_type: "{{ item.split('-')[0] }}"
node_number: "{{ item.split('-')[1] }}"
node_name: "{{ item.split('-')[0] }}-{{ '%02d' | format (item.split('-')[1]|int) }}"
node_ip: "{{ip_range_24_prefix}}.{{ (ip_range_control_start|int + item.split('-')[1]|int) if item.split('-')[0] == 'control' else (ip_range_worker_start|int + item.split('-')[1]|int) if item.split('-')[0] == 'worker' }}"
- name: get proxmox API auth cookie
uri:
url: "https://{{ proxmox_host }}:8006/api2/json/access/ticket"
validate_certs: no
method: POST
body_format: form-urlencoded
body:
username: "{{ proxmox_user }}"
password: "{{ proxmox_pass }}"
status_code: 200
register: login
- name: query proxmox next free vmid
uri:
url: "https://{{ proxmox_host }}:8006/api2/json/cluster/nextid"
validate_certs: no
method: GET
headers:
Cookie: "PVEAuthCookie={{ login.json.data.ticket }}"
CSRFPreventionToken: "{{ login.json.data.CSRFPreventionToken }}"
register: next_pvid
# # should fail on error with uri return (also if field exists and is int) - else nodes will be created but subsequent steps fail
- name: provision nodes on proxmox kvm host
proxmox_kvm:
api_user : "{{ proxmox_user }}"
api_password: "{{ proxmox_pass }}"
api_host : "{{ proxmox_host }}"
node : "{{ proxmox_node }}"
vmid : "{{ next_pvid.json.data }}"
boot : d # n network,d cdrom,c harddisk, combine in any order cdn
kvm : yes
agent : yes
name : "{{ node_name }}"
sockets : 1
cores : 4
memory : 4096
vga : std
scsihw : virtio-scsi-single
net : '{"net0":"virtio,bridge={{ proxmox_vmbr }},firewall=0"}'
args : '' # create empty qemu bootstrap, this is setup along with cloud-init userdata and the disk image import
ostype : l26
state : present
register: _result
- name: end run with failure where host(s) pre-exist
fail:
msg: "node {{ node_name }} already exists"
when: _result.msg == "VM with name <{{ node_name }}> already exists"
- name: get node MAC
uri:
url: "https://{{ proxmox_host }}:8006/api2/json/nodes/{{ proxmox_node }}/qemu/{{ next_pvid.json.data }}/config"
validate_certs: no
method: GET
headers:
Cookie: "PVEAuthCookie={{ login.json.data.ticket }}"
CSRFPreventionToken: "{{ login.json.data.CSRFPreventionToken }}"
register: _result
- name: register node MAC
set_fact:
node_mac: "{{ ((_result.json.data.net0).split(',')[0]).split('=')[1] | lower }}"
# proxmox cloud-init only works with snippets enabled on the datastore - you can write your own "cloud-init" iso containing the network/userdata and mount a cdrom, this is an official cloud-init method
# prep datastore:
# pvesm set local --content backup,iso,vztmpl,snippets
# or edit /etc/pve/storage.cfg
# ls /var/lib/vz/snippets/
# pvesm list local
# should write these snippets over the API, not checked if supported
- name: write cloud-init network-data to proxmox server
template:
src: network-data.j2
dest: "{{ proxmox_img_datastore_path }}/snippets/network-data.{{ next_pvid.json.data }}.yaml"
owner: root
group: root
mode: '0755'
delegate_to: proxmox_server
- name: write cloud-init user-data to proxmox server
template:
src: user-data.j2
dest: "{{ proxmox_img_datastore_path }}/snippets/user-data.{{ next_pvid.json.data }}.yaml"
owner: root
group: root
mode: '0755'
delegate_to: proxmox_server
# using default cloudinit image in proxmox only allows meta-data settings for network, user-data settings for account/ssh, no other cloud-init function
# proxmox 6 requires the citype setting of nocloud
# cicustom entry adds an entry in /etc/pve/nodes/pve/qemu-server/103.conf, seems the cloudinit drive is regenerated on boot so no update command is required when edit/rebooting
# the cloudinit userdata is selectively idempotent, it will skip if all the predefined modules have run but maybe not boot commands or scripts - it might be better to register to a config service for idempotent recipes/playbooks (on every boot with no ill effect) or run scripts that first check if they have run before successfully
#
- name: create primary disk using cloud-init ready image
shell:
cmd: |
qm importdisk "{{ next_pvid.json.data }}" raw."{{ image_name }}" "{{ proxmox_vm_datastore }}"
qm set "{{ next_pvid.json.data }}" --scsihw virtio-scsi-pci --scsi0 "{{ proxmox_vm_datastore }}":vm-"{{ next_pvid.json.data }}"-disk-0
qm set "{{ next_pvid.json.data }}" --citype nocloud
qm set "{{ next_pvid.json.data }}" --cicustom "network={{ proxmox_img_datastore }}:snippets/network-data.{{ next_pvid.json.data }}.yaml,user={{ proxmox_img_datastore }}:snippets/user-data.{{ next_pvid.json.data }}.yaml"
qm set "{{ next_pvid.json.data }}" --ide2 "{{ proxmox_vm_datastore }}":cloudinit
qm set "{{ next_pvid.json.data }}" --boot c --bootdisk scsi0
qm set "{{ next_pvid.json.data }}" --serial0 socket --vga serial0
chdir: "{{ proxmox_img_datastore_path }}/template/iso/"
register: _result
delegate_to: proxmox_server
# should error check _result failed or rc code, for example we run out of space in VG
# - debug:
# msg: "{{ _result }}"
# cloud-init will expand to all available disk as an initramfs boot task, resize the proxmox base disk here
- name: resize node disk
uri:
url: "https://{{ proxmox_host }}:8006/api2/json/nodes/{{ proxmox_node }}/qemu/{{ next_pvid.json.data }}/resize"
validate_certs: no
method: PUT
headers:
Cookie: "PVEAuthCookie={{ login.json.data.ticket }}"
CSRFPreventionToken: "{{ login.json.data.CSRFPreventionToken }}"
body_format: form-urlencoded
body:
disk: scsi0
size: "{{ proxmox_node_disk_size }}"
- name: start nodes on proxmox kvm host
proxmox_kvm:
api_user : "{{ proxmox_user }}"
api_password: "{{ proxmox_pass }}"
api_host : "{{ proxmox_host }}"
node : "{{ proxmox_node }}"
name : "{{ node_name }}"
state : started
register: _result
- name: add node to in-memory inventory
add_host: >
name="{{ node_name }}"
groups="{{ node_type }}"
ansible_host="{{ node_ip }}"
ansible_ssh_user="{{ node_account }}"
ansible_ssh_pass="{{ node_account_password }}"
ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"

18
proxmox_node_provision/templates/ansible-hosts.j2 Executable file
View File

@ -0,0 +1,18 @@
{% for item in groups %}
[{{item}}]
{% for entry in groups[item] %}
{% set ip = hostvars[entry].ansible_host -%}
{{ entry }} ansible_host={{ ip }}
{% endfor %}
{% endfor %}
[proxmox:vars]
ansible_ssh_user={{ hostvars.proxmox_server.proxmox_ssh_user }}
ansible_ssh_pass={{ hostvars.proxmox_server.proxmox_ssh_pass }}
become=true
[all:vars]
ansible_password={{ node_account_password }}
ansible_user={{ node_account }}
ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
ansible_python_interpreter=/usr/bin/python3

16
proxmox_node_provision/templates/network-data.j2 Executable file
View File

@ -0,0 +1,16 @@
version: 1
config:
- type: physical
name: eth0
mac_address: '{{ node_mac }}'
subnets:
- type: static
address: '{{ node_ip }}'
netmask: '{{ ip_range_subnet }}'
gateway: '{{ ip_range_gateway }}'
- type: nameserver
address:
- '{{ ip_range_dns }}'
search:
- '{{ domain }}'

79
proxmox_node_provision/templates/user-data.j2 Executable file
View File

@ -0,0 +1,79 @@
#cloud-config
hostname: {{ node_name }}
manage_etc_hosts: true
fqdn: {{ node_name }}.{{ domain }}
groups:
- {{ node_account }}
users:
- name: {{ node_account }}
primary_group: {{ node_account }}
passwd: {{ node_account_password_hash }}
sudo: ALL=(ALL) NOPASSWD:ALL
lock-passwd: false
shell: /bin/bash
ssh_pwauth: true
package_update: true
packages:
- iptables
- arptables
- ebtables
- apt-transport-https
- ca-certificates
- curl
- gnupg-agent
- software-properties-common
- qemu-guest-agent
- python
runcmd:
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
- add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
- curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
- add-apt-repository "deb https://apt.kubernetes.io/ kubernetes-xenial main"
- apt-get update -y
- apt-get install -y containerd.io=1.2.10-3 docker-ce=5:19.03.4~3-0~ubuntu-$(lsb_release -cs) docker-ce-cli=5:19.03.4~3-0~ubuntu-$(lsb_release -cs) kubelet kubeadm kubectl
- apt-mark hold kubelet kubeadm kubectl
- mkdir -p /etc/systemd/system/docker.service.d
- systemctl daemon-reload
- systemctl restart docker
- systemctl restart kubelet
- echo "@reboot /cloud-init-status.sh" | crontab -
write_files:
- path: /etc/docker/daemon.json
permissions: 0755
owner: root
content: |
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
- path: /cloud-init-status.sh
permissions: 0750
owner: root
content: |
#!/bin/bash
set -e
cloud-init status --wait > /dev/null 2>&1
touch /ansible-ready
final_message: "The system is up, init duration $UPTIME seconds"
# package upgrade gets us a new kernel, reboot if cloudinit has made system changes on first run
power_state:
#delay: "+2"
mode: reboot
message: post setup reboot initiated on {{ node_name }}
timeout: 30
condition: True

132
proxmox_upload/tasks/main.yml Executable file
View File

@ -0,0 +1,132 @@
---
- set_fact:
image_name: "{{ ((image_url | urlsplit).path).split ('/')[-1] }}"
- name: get local IMG attributes
stat:
path: ./{{ image_name }}
register: img_local_attributes
- fail:
msg: "{{ image_name }} not present"
when: not img_local_present.stat.exists
- set_fact:
img_size: "{{ img_local_attributes.stat.size }}"
when: img_local_attributes.stat.exists == true
- name: get proxmox API auth cookie
uri:
url: "https://{{ proxmox_host }}:8006/api2/json/access/ticket"
validate_certs: no
method: POST
body_format: form-urlencoded
body:
username: "{{ proxmox_user }}"
password: "{{ proxmox_pass }}"
status_code: 200
register: login
# - name: print proxmox PVEAuthCookie / CSRFPreventionToken tokens
# vars:
# msg: |
# ticket "{{ login.json.data.ticket }}"
# CSRF "{{ login.json.data.CSRFPreventionToken }}"
# debug:
# msg: "{{ msg.split('\n') }}"
- name: query proxmox IMG datastore status
uri:
url: "https://{{ proxmox_host }}:8006/api2/json/nodes/{{ proxmox_node }}/storage/{{ proxmox_img_datastore }}/status"
validate_certs: no
method: GET
headers:
Cookie: "PVEAuthCookie={{ login.json.data.ticket }}"
CSRFPreventionToken: "{{ login.json.data.CSRFPreventionToken }}"
register: img_datastore_status
- name: check proxmox IMG datastore status
debug:
msg:
- "storage {{ proxmox_img_datastore }}@{{ proxmox_node }} must have content type iso enabled"
- "content type: {{ img_datastore_status.json.data.content }}"
vars:
uristatus: "{{ img_datastore_status.failed }}"
datatype: "{{ img_datastore_status.json.data }}"
failed_when: uristatus or datatype is not search("iso")
- name: query proxmox IMG datastore content
uri:
url: "https://{{ proxmox_host }}:8006/api2/json/nodes/{{ proxmox_node }}/storage/{{ proxmox_img_datastore }}/content"
validate_certs: no
method: GET
headers:
Cookie: "PVEAuthCookie={{ login.json.data.ticket }}"
CSRFPreventionToken: "{{ login.json.data.CSRFPreventionToken }}"
register: get_datastore_content
- name: find IMG on proxmox storage
debug:
msg: checking
#var: "{{ item }}"
#var: "{{ item.name }} {{ item.size }}"
loop: "{{ get_datastore_content.json | json_query(query) }}"
vars:
query: "data[?volid=='{{ proxmox_img_datastore }}:iso/{{ image_name }}'].{name: volid, size: size}"
register: get_datastore_img
- name: check IMG present on storage
debug:
msg: "image {{ image_name }} not present on proxmox node {{ proxmox_node }} on storage {{ proxmox_img_datastore }}"
when: get_datastore_img.skipped is defined
# https://forum.proxmox.com/threads/problems-with-uploading-new-storage-via-proxmox-api.37164/
# https://github.com/ansible/ansible/issues/58823
# cant use uri module (issues) need to issue curl from the shell (not ideal)
- name: upload ISO content to proxmox storage
shell: 'curl -X POST "https://{{ proxmox_host }}:8006/api2/json/nodes/{{ proxmox_node }}/storage/{{ proxmox_img_datastore }}/upload" \
-k \
-b "PVEAuthCookie={{ login.json.data.ticket }}" \
-H "CSRFPreventionToken: {{ login.json.data.CSRFPreventionToken }}" \
-H "Content-Type: multipart/form-data" \
--form "content=iso" \
--form "filename=@{{ image_name }}" \
'
args:
warn: false
when: get_datastore_img.skipped is defined
# allow iso to be moved from API cache to datastore, if this executes too quickly the size of the iso will be reported incorrectly
# comment this section to test following fail logic
- pause:
seconds: 10
- name: query proxmox IMG datastore content
uri:
url: "https://{{ proxmox_host }}:8006/api2/json/nodes/{{ proxmox_node }}/storage/{{ proxmox_img_datastore }}/content"
validate_certs: no
method: GET
headers:
Cookie: "PVEAuthCookie={{ login.json.data.ticket }}"
CSRFPreventionToken: "{{ login.json.data.CSRFPreventionToken }}"
register: get_datastore_content
- name: find IMG on proxmox storage
debug:
msg: checking
#var: "{{ item }}"
#var: "{{ item.name }} {{ item.size }}"
loop: "{{ get_datastore_content.json | json_query(query) }}"
vars:
query: "data[?volid=='{{ proxmox_img_datastore }}:iso/{{ image_name }}'].{name: volid, size: size}"
register: get_datastore_img
- name: report image present on storage
fail:
msg: "image {{ image_name }} not present on proxmox node {{ proxmox_node }} storage {{ proxmox_img_datastore }}, upload failed"
when: get_datastore_img.skipped is defined
- name: compare local and proxmox image size
fail:
msg: "image {{ image_name }} present on proxmox node {{ proxmox_node }} storage {{ proxmox_img_datastore }} is a different size {{ get_datastore_img.results[0].item.size }} to local image {{ img_size }}, upload failed"
when: get_datastore_img.results[0].item.size|int != img_size|int

61
site.yml Executable file
View File

@ -0,0 +1,61 @@
---
- hosts: localhost
gather_facts: false
become: false
roles:
- get_cloud-init_image
- proxmox_upload
- proxmox_node_provision
- hosts: control,worker
gather_facts: false
become: false
vars:
role_action: wait
strategy: free # 'strategy: free' runs parallel roles against all hosts, 'serial: 1' can be used to control how many hosts actioned in parallel
roles:
- wait_for_nodes
- hosts: localhost
gather_facts: false
become: false
vars:
role_action: check
roles:
- wait_for_nodes
# these certificates arent used for the kubernetes cluster (generates its own with this basic install), just services that run on kubernetes (registry service)
# certificates generated before installing kubernetes so we dont have to do a docker restart to activate cert store on a running cluster (k8s gotcha)
- hosts: "{{ groups['control'][0] }}"
gather_facts: false
become: true
roles:
- generate_certificates
- hosts: control
gather_facts: false
become: true
roles:
- k8s_control_init
- hosts: worker
gather_facts: false
become: true
roles:
- k8s_worker_init
- hosts: "{{ groups['control'][0] }}"
gather_facts: false
become: true
become_user: "{{ node_account }}"
roles:
- k8s_metallb
- k8s_registry
- k8s_dhcp
- k8s_dashboard
- hosts: localhost
gather_facts: false
become: false
roles:
- proxmox_farm_provision

29
wait_for_nodes/tasks/check.yml Executable file
View File

@ -0,0 +1,29 @@
---
- name: build a list of target nodes
set_fact:
ansible_target_nodes: "{{ ansible_target_nodes | default([]) + [item] }}"
with_items:
- "{{ groups['control'] }}"
- "{{ groups['worker'] }}"
delegate_to: localhost
- name: build a list of ready nodes
set_fact:
ansible_ready_nodes: "{{ ansible_ready_nodes | default([]) + [item] }}"
with_items:
- "{{ groups['all'] }}"
when: hostvars[item].ansible_is_ready is defined and hostvars[item].ansible_is_ready
delegate_to: localhost
- name: find unready nodes
set_fact:
unready_node: "{{ unready_node | default([]) + [item] }}" # if the value is not a list dont fail but default to a list
loop: "{{ ansible_target_nodes }}"
when: item not in ansible_ready_nodes
delegate_to: localhost
- name: fail with unready node
fail:
msg: "node {{ unready_node | list | join(', ') }} in a potentially unready state for ansible, did cloud-init finish?"
when: unready_node | length > 0
delegate_to: localhost

22
wait_for_nodes/tasks/main.yml Executable file
View File

@ -0,0 +1,22 @@
---
- name: check in which mode the play is being run
vars:
message:
- "This role requires a mode of wait or check set by variable role_action."
- "It is designed to be run twice in a playbook"
- " once against the target hosts: <host-group(s)> with strategy: free and variable role_action: wait set"
- " once against hosts: localhost with role_action: check set"
fail:
msg: "{{ message }}"
when: role_action is undefined or not role_action in ["wait", "check"]
- name: wait for the node to build with cloud-init and reboot, then populate the filesystem with a flag (/ansible-ready)
include: wait.yml
when: role_action == "wait"
# run on localhost to validate results of target machines
- name: check nodes in ready state
include: check.yml
when: role_action == "check"
delegate_to: localhost

19
wait_for_nodes/tasks/wait.yml Executable file
View File

@ -0,0 +1,19 @@
---
- set_fact:
node_ip: "{{ hostvars[inventory_hostname]['ansible_host'] }}"
ansible_is_ready: false
# requires local apt install sshpass
- name: wait for cloud-init to finish and host to become available
# checking for /var/lib/cloud/instance/boot-finished is fine for cloud init, but what if the host was built another way?
# we simulate this condition by checking for the presence of different file used to indicate a node is ready (maybe after a build / update / reboot cycle to be robust)
#local_action: command sshpass -p "{{node_account_password|default('')}}" ssh -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" "{{ node_account }}@{{ node_ip }}" "sleep 5;ls /var/lib/cloud/instance/boot-finished"
local_action: command sshpass -p "{{node_account_password|default('')}}" ssh -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" "{{ node_account }}@{{ node_ip }}" "sleep 5;ls /ansible-ready"
changed_when: False
register: ready
until: ready.rc == 0
retries: 20
- set_fact:
ansible_is_ready: true
when: ready.rc == 0

4
wait_for_nodes/vars/main.yml Executable file
View File

@ -0,0 +1,4 @@
---
ansible_target_nodes: []
ansible_ready_nodes: []
unready_node: []