provision kubernetes cluster with ansible and multipass

This commit is contained in:
Bartal Læarsson
2025-03-01 22:00:05 +00:00
parent ef34e0cf99
commit 7534d67c66
20 changed files with 0 additions and 1511 deletions
-245
View File
@@ -1,245 +0,0 @@
# Kubernetes Cluster Deployment with Ansible
This repository contains Ansible playbooks to set up a Kubernetes cluster on existing Ubuntu 24.04 VMs.
## Prerequisites
1. Three Ubuntu 24.04 (Noble Numbat) VMs running
2. SSH access to all VMs
3. Ansible installed on your control machine
4. Sudo access configured on all VMs
## Repository Structure
```
kubernetes-ansible/
├── inventory.yml # Inventory file with VM information
├── main.yml # Main playbook with task orchestration
├── tasks/
│ ├── setup-hosts.yml # Configure /etc/hosts
│ ├── update-dns.yml # Configure DNS
│ ├── ssh.yml # Configure SSH settings
│ ├── setup-kernel.yml # Configure kernel modules and sysctl
│ ├── node-setup.yml # Install Kubernetes components
│ ├── control-plane-init.yml # Initialize control plane
│ └── worker-join.yml # Join worker nodes to cluster
└── README.md
```
## Configuration Steps
1. Clone this repository:
```bash
git clone <repository-url>
cd kubernetes-ansible
```
2. Install required Ansible collections:
```bash
ansible-galaxy collection install community.general
```
3. Edit `inventory.yml` and update the IP addresses of your VMs:
```yaml
controlplane:
ansible_host: 192.168.1.X # Replace with your control plane IP
node01:
ansible_host: 192.168.1.Y # Replace with your first worker node IP
node02:
ansible_host: 192.168.1.Z # Replace with your second worker node IP
```
4. Update the SSH credentials in the inventory file:
```yaml
vars:
ansible_user: ubuntu # Replace with your SSH user
# Use one of these depending on your authentication method:
# ansible_ssh_private_key_file: ~/.ssh/id_rsa
# ansible_ssh_pass: your_password
```
## Deployment
You can deploy the entire cluster at once:
```bash
ansible-playbook -i inventory.yml main.yml
```
Or run deployment incrementally using tags:
```bash
# 1. Configure hosts, DNS, SSH, and kernel
ansible-playbook -i inventory.yml main.yml --tags common
# 2. Install Kubernetes components on all nodes
ansible-playbook -i inventory.yml main.yml --tags k8s-setup
# 3. Initialize control plane
ansible-playbook -i inventory.yml main.yml --tags control-plane
# 4. Join worker nodes
ansible-playbook -i inventory.yml main.yml --tags workers
```
## Iterative Testing Approach
To methodically test the deployment step by step:
### 1. Test Basic Connectivity
```bash
# Verify SSH access to all nodes
ansible -i inventory.yml all -m ping
```
### 2. Test Basic System Configuration
```bash
# Apply common configuration to control plane only
ansible-playbook -i inventory.yml main.yml --tags common --limit controlplane
# Verify hosts file configuration
ansible -i inventory.yml controlplane -m command -a "cat /etc/hosts"
# Verify DNS configuration
ansible -i inventory.yml controlplane -m command -a "systemctl status systemd-resolved"
```
### 3. Test Kernel Configuration
```bash
# Verify kernel modules
ansible -i inventory.yml controlplane -m command -a "lsmod | grep -E 'overlay|br_netfilter'"
# Verify sysctl settings
ansible -i inventory.yml controlplane -m command -a "sysctl net.bridge.bridge-nf-call-iptables net.ipv4.ip_forward"
```
### 4. Test Kubernetes Components
```bash
# Install Kubernetes components on control plane
ansible-playbook -i inventory.yml main.yml --tags k8s-setup --limit controlplane
# Verify containerd is running
ansible -i inventory.yml controlplane -m command -a "systemctl status containerd"
# Verify Kubernetes binaries
ansible -i inventory.yml controlplane -m command -a "kubectl version --client"
```
### 5. Initialize Control Plane
```bash
# Initialize control plane
ansible-playbook -i inventory.yml main.yml --tags control-plane
# Verify control plane initialization
ansible -i inventory.yml controlplane -m command -a "kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes"
```
### 6. Join Worker Nodes One by One
```bash
# Join first worker node
ansible-playbook -i inventory.yml main.yml --tags workers --limit node01
# Verify first worker joined
ansible -i inventory.yml controlplane -m command -a "kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes"
# Join second worker node
ansible-playbook -i inventory.yml main.yml --tags workers --limit node02
# Verify all nodes
ansible -i inventory.yml controlplane -m command -a "kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes"
```
### 7. Test Network Plugin
```bash
# Check Flannel pods
ansible -i inventory.yml controlplane -m command -a "kubectl --kubeconfig=/etc/kubernetes/admin.conf get pods -n kube-system | grep flannel"
# Deploy test application
ansible -i inventory.yml controlplane -m command -a "kubectl --kubeconfig=/etc/kubernetes/admin.conf create deployment nginx --image=nginx"
ansible -i inventory.yml controlplane -m command -a "kubectl --kubeconfig=/etc/kubernetes/admin.conf expose deployment nginx --port=80 --type=NodePort"
```
### 8. Debugging Common Issues
If any step fails, try these debugging commands:
```bash
# Check service logs
ansible -i inventory.yml <node> -m command -a "journalctl -xeu <service-name>"
# Example for kubelet
ansible -i inventory.yml node01 -m command -a "journalctl -xeu kubelet"
# Check network connectivity
ansible -i inventory.yml controlplane -m command -a "ping -c 1 {{ hostvars['node01']['ansible_host'] }}"
```
### 9. Reset if Needed
To reset a node and start fresh:
```bash
# Reset kubeadm
ansible -i inventory.yml <node> -m command -a "kubeadm reset -f"
# Clean up directories
ansible -i inventory.yml <node> -m command -a "rm -rf /etc/kubernetes /var/lib/kubelet /var/lib/etcd ~/.kube"
```
## Key Components
- **Kernel Configuration**: Sets up required kernel modules (overlay, br_netfilter) and sysctl settings
- **Container Runtime**: Installs and configures containerd with systemd cgroup driver
- **Kubernetes**: Installs kubelet, kubeadm, and kubectl
- **Networking**: Configures Flannel as the CNI plugin with pod CIDR 10.244.0.0/16
## Ubuntu 24.04 Specific Notes
- This playbook has been adjusted for Ubuntu 24.04 (Noble Numbat)
- The containerd installation includes a fallback to the Docker repository if the default package isn't available
- Kernel module loading includes error handling for modules that might be built into the kernel
- DNS configuration is specifically adjusted for systemd-resolved in Ubuntu 24.04
## Validation
After deployment, verify your cluster:
```bash
ssh <your-user>@<controlplane-ip>
kubectl get nodes
```
You should see all nodes in the `Ready` state:
```
NAME STATUS ROLES AGE VERSION
controlplane Ready control-plane 10m v1.28.x
node01 Ready <none> 8m v1.28.x
node02 Ready <none> 8m v1.28.x
```
## Troubleshooting
If nodes don't join the cluster or show issues:
1. Check connectivity between nodes: `ping <node-ip>`
2. Verify firewall allows required ports (6443, 10250)
3. Review logs on problematic nodes: `journalctl -xeu kubelet`
4. Verify DNS resolution: `nslookup controlplane`
## Cleanup
To reset a node and remove Kubernetes:
```bash
sudo kubeadm reset
sudo apt-get purge -y kubeadm kubectl kubelet kubernetes-cni
sudo apt-get autoremove -y
sudo rm -rf ~/.kube /etc/kubernetes /var/lib/kubelet /var/lib/etcd
```
-22
View File
@@ -1,22 +0,0 @@
---
all:
children:
k8s_cluster:
children:
control_plane:
hosts:
controlplane:
ansible_host: 10.203.146.213
workers:
hosts:
node01:
ansible_host: 10.203.146.194
node02:
ansible_host: 10.203.146.61
node03:
ansible_host: 10.203.146.193
vars:
ansible_user: ubuntu
ansible_become: yes
ansible_ssh_private_key_file: ~/.ssh/id_rsa
-45
View File
@@ -1,45 +0,0 @@
---
- name: Common configuration for all nodes
hosts: k8s_cluster
become: yes
tags:
- common
tasks:
- name: Include hosts setup
include_tasks: tasks/setup-hosts.yml
- name: Include DNS setup
include_tasks: tasks/update-dns.yml
- name: Include SSH setup
include_tasks: tasks/ssh.yml
- name: Include kernel setup
include_tasks: tasks/setup-kernel.yml
- name: Kubernetes setup for all nodes
hosts: k8s_cluster
become: yes
tags:
- k8s-setup
tasks:
- name: Include Kubernetes node setup
include_tasks: tasks/node-setup.yml
- name: Initialize Kubernetes control plane
hosts: control_plane
become: yes
tags:
- control-plane
tasks:
- name: Initialize control plane
include_tasks: tasks/control-plane-init.yml
- name: Join worker nodes to cluster
hosts: workers
become: yes
tags:
- workers
tasks:
- name: Join worker nodes
include_tasks: tasks/worker-join.yml
@@ -1,98 +0,0 @@
---
# tasks/control-plane-init.yml
- name: Set Kubernetes network CIDRs for Calico
set_fact:
pod_cidr: "192.168.0.0/16" # Calico default CIDR
service_cidr: "10.96.0.0/16"
- name: Initialize Kubernetes control plane
command: >
kubeadm init
--pod-network-cidr={{ pod_cidr }}
--service-cidr={{ service_cidr }}
--apiserver-advertise-address={{ ansible_default_ipv4.address }}
register: kubeadm_init
args:
creates: /etc/kubernetes/admin.conf
- name: Create .kube directory for current user
file:
path: "/home/{{ ansible_user }}/.kube"
state: directory
owner: "{{ ansible_user }}"
group: "{{ ansible_user }}"
mode: '0755'
- name: Copy Kubernetes admin.conf to user's .kube/config
copy:
src: /etc/kubernetes/admin.conf
dest: "/home/{{ ansible_user }}/.kube/config"
remote_src: yes
owner: "{{ ansible_user }}"
group: "{{ ansible_user }}"
mode: '0644'
- name: Install Calico Operator
shell: kubectl --kubeconfig=/etc/kubernetes/admin.conf create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.1/manifests/tigera-operator.yaml
register: calico_operator_result
changed_when: "'created' in calico_operator_result.stdout"
failed_when:
- calico_operator_result.rc != 0
- "'already exists' not in calico_operator_result.stderr"
- name: Install Calico Custom Resources
shell: kubectl --kubeconfig=/etc/kubernetes/admin.conf create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.1/manifests/custom-resources.yaml
register: calico_resources_result
changed_when: "'created' in calico_resources_result.stdout"
failed_when:
- calico_resources_result.rc != 0
- "'already exists' not in calico_resources_result.stderr"
- name: Wait for calico-system namespace to be created
shell: kubectl --kubeconfig=/etc/kubernetes/admin.conf get namespace | grep calico-system
register: calico_ns
until: calico_ns.rc == 0
retries: 30
delay: 10
changed_when: false
- name: Wait for Calico pods to be ready
shell: |
# Give pods some time to be created
sleep 30
kubectl --kubeconfig=/etc/kubernetes/admin.conf -n calico-system wait --for=condition=ready pods --all --timeout=300s
register: calico_pods_ready
changed_when: false
failed_when: false # Don't fail if timeout occurs, we'll verify the node status later
- name: Wait for CoreDNS to be ready
shell: kubectl --kubeconfig=/etc/kubernetes/admin.conf -n kube-system wait --for=condition=ready pods -l k8s-app=kube-dns --timeout=120s
register: coredns_ready
changed_when: false
failed_when: false
- name: Wait for node to be ready
shell: kubectl --kubeconfig=/etc/kubernetes/admin.conf wait --for=condition=ready nodes --all --timeout=120s
register: node_ready
changed_when: false
- name: Remove taint from control plane node
shell: kubectl --kubeconfig=/etc/kubernetes/admin.conf taint nodes --all node-role.kubernetes.io/control-plane-
register: taint_result
changed_when: "'untainted' in taint_result.stdout"
failed_when: false # This might fail if the taint doesn't exist, which is fine
- name: Remove load balancer label from all nodes
shell: kubectl --kubeconfig=/etc/kubernetes/admin.conf label nodes --all node.kubernetes.io/exclude-from-external-load-balancers-
register: label_result
changed_when: "'labeled' in label_result.stdout"
failed_when: false # This might fail if the label doesn't exist, which is fine
- name: Get join command
command: kubeadm token create --print-join-command
register: join_command
changed_when: false
- name: Store join command
set_fact:
join_command: "{{ join_command.stdout }}"
@@ -1,169 +0,0 @@
---
# tasks/node-setup.yml
- name: Update apt cache
apt:
update_cache: yes
- name: Install apt transport packages
apt:
name:
- apt-transport-https
- ca-certificates
- curl
- gnupg
state: present
# Add Docker GPG key
- name: Add Docker GPG key
apt_key:
url: https://download.docker.com/linux/ubuntu/gpg
state: present
- name: Add Docker repository
apt_repository:
repo: "deb [arch=amd64] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
state: present
filename: docker
- name: Install containerd from Docker repository
apt:
name: containerd.io
state: present
update_cache: yes
- name: Create containerd directory
file:
path: /etc/containerd
state: directory
mode: '0755'
- name: Check if containerd config command works
command: which containerd
register: containerd_check
changed_when: false
failed_when: false
- name: Generate default containerd config (if containerd exists)
command: containerd config default
register: containerd_config
changed_when: false
when: containerd_check.rc == 0
- name: Create a basic containerd config (if command failed)
copy:
content: |
version = 2
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".containerd]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
dest: /etc/containerd/config.toml
owner: root
group: root
mode: '0644'
when: containerd_check.rc != 0
- name: Write containerd config from generated output
copy:
content: "{{ containerd_config.stdout }}"
dest: /etc/containerd/config.toml
owner: root
group: root
mode: '0644'
when: containerd_check.rc == 0 and containerd_config.stdout is defined and containerd_config.stdout != ""
- name: Enable SystemdCgroup in containerd config
replace:
path: /etc/containerd/config.toml
regexp: 'SystemdCgroup = false'
replace: 'SystemdCgroup = true'
when: containerd_check.rc == 0
- name: Restart containerd
systemd:
name: containerd
state: restarted
enabled: yes
- name: Get latest Kubernetes version
shell: curl -L -s https://dl.k8s.io/release/stable.txt | awk 'BEGIN { FS="." } { printf "%s.%s", $1, $2 }'
register: kube_latest
changed_when: false
- name: Create keyrings directory
file:
path: /etc/apt/keyrings
state: directory
mode: '0755'
- name: Download Kubernetes apt key
get_url:
url: "https://pkgs.k8s.io/core:/stable:/{{ kube_latest.stdout }}/deb/Release.key"
dest: /tmp/kubernetes-release.key
mode: '0644'
- name: Add Kubernetes apt key
shell: cat /tmp/kubernetes-release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
args:
creates: /etc/apt/keyrings/kubernetes-apt-keyring.gpg
- name: Add Kubernetes apt repository
copy:
content: "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/{{ kube_latest.stdout }}/deb/ /"
dest: /etc/apt/sources.list.d/kubernetes.list
owner: root
group: root
mode: '0644'
- name: Update apt cache
apt:
update_cache: yes
- name: Install Kubernetes components
apt:
name:
- kubelet
- kubeadm
- kubectl
state: present
- name: Hold Kubernetes packages
dpkg_selections:
name: "{{ item }}"
selection: hold
loop:
- kubelet
- kubeadm
- kubectl
- name: Check if crictl is installed
command: which crictl
register: crictl_check
changed_when: false
failed_when: false
- name: Configure crictl
shell: |
crictl config \
--set runtime-endpoint=unix:///run/containerd/containerd.sock \
--set image-endpoint=unix:///run/containerd/containerd.sock
changed_when: false
when: crictl_check.rc == 0
- name: Get primary network interface IP
shell: ip -4 addr show {{ ansible_default_ipv4.interface }} | grep "inet" | head -1 | awk '{print $2}' | cut -d/ -f1
register: node_ip
changed_when: false
- name: Configure kubelet node IP
copy:
content: "KUBELET_EXTRA_ARGS='--node-ip {{ node_ip.stdout }}'"
dest: /etc/default/kubelet
owner: root
group: root
mode: '0644'
@@ -1,33 +0,0 @@
---
- name: Remove unwanted entries from /etc/hosts
lineinfile:
path: /etc/hosts
regexp: "{{ item }}"
state: absent
loop:
- "^.*ubuntu-jammy.*$"
- "^.*{{ ansible_hostname }}.*$"
- name: Get primary IP address
shell: "ip route | grep default | grep -Pv '10\\.\\d+\\.\\d+\\.\\d+' | awk '{ print $9 }'"
register: primary_ip
changed_when: false
- name: Set PRIMARY_IP environment variable
lineinfile:
path: /etc/environment
line: "PRIMARY_IP={{ primary_ip.stdout }}"
state: present
- name: Add controlplane entry to /etc/hosts
lineinfile:
path: /etc/hosts
line: "{{ hostvars['controlplane']['ansible_host'] }} controlplane"
state: present
- name: Add worker nodes to /etc/hosts
lineinfile:
path: /etc/hosts
line: "{{ hostvars[item]['ansible_host'] }} {{ item }}"
state: present
loop: "{{ groups['workers'] }}"
@@ -1,44 +0,0 @@
---
# tasks/setup-kernel.yml
- name: Create Kubernetes kernel modules config file
copy:
content: |
overlay
br_netfilter
dest: /etc/modules-load.d/k8s.conf
owner: root
group: root
mode: '0644'
- name: Attempt to load overlay module
shell: modprobe overlay || true
changed_when: false
- name: Attempt to load br_netfilter module
shell: modprobe br_netfilter || true
changed_when: false
- name: Create Kubernetes sysctl config file
copy:
content: |
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
dest: /etc/sysctl.d/k8s.conf
owner: root
group: root
mode: '0644'
register: sysctl_conf
- name: Apply sysctl changes
command: sysctl --system
when: sysctl_conf.changed
- name: Check loaded modules
shell: "lsmod | grep -E 'overlay|br_netfilter' || echo 'Modules may be built into kernel'"
register: modules_check
changed_when: false
- name: Display module status
debug:
var: modules_check.stdout_lines
-18
View File
@@ -1,18 +0,0 @@
---
# tasks/ssh.yml
- name: Enable password authentication in sshd_config
replace:
path: /etc/ssh/sshd_config
regexp: '#PasswordAuthentication'
replace: 'PasswordAuthentication'
- name: Enable KbdInteractiveAuthentication in sshd_config
replace:
path: /etc/ssh/sshd_config
regexp: 'KbdInteractiveAuthentication no'
replace: 'KbdInteractiveAuthentication yes'
- name: Restart SSH service
systemd:
name: ssh
state: restarted
@@ -1,18 +0,0 @@
---
- name: Ensure systemd-resolved is running
systemd:
name: systemd-resolved
state: started
enabled: yes
- name: Configure systemd-resolved to use Google DNS
lineinfile:
path: /etc/systemd/resolved.conf
regexp: '^#?DNS='
line: 'DNS=8.8.8.8'
state: present
- name: Restart systemd-resolved service
systemd:
name: systemd-resolved
state: restarted
@@ -1,13 +0,0 @@
---
# tasks/worker-join.yml
- name: Get join command from control plane
delegate_to: controlplane
command: kubeadm token create --print-join-command
register: join_command_result
changed_when: false
run_once: true
- name: Join worker node to the cluster
command: "{{ join_command_result.stdout }}"
args:
creates: /etc/kubernetes/kubelet.conf
-246
View File
@@ -1,246 +0,0 @@
# Install Multipass
```bash
sudo apt update
sudo apt upgrade
sudo snap install multipass
```
To fast-track to build multiple instances from the cloudinit.yaml file run e.g.
```bash
./launch-instances.sh cp1 worker1 worker2 worker3 worker4
```
Then skip ahead to `# Install and Configure Network Plugin 'Calico'` further down in this file
# Launch
## Launch basic instance
```bash
multipass launch --name worker-01
```
## Launch instance with defined resources
```bash
multipass launch --name worker-02 --disk 20G --memory 8G --cpus 2
```
## Launch instance with defined resources and cloudinit
create a yaml file which defines the intial state of the ubuntu instance
for example `cloudinit.yaml`
and run a launch command like this after you have created the subsequent file
```bash
multipass launch --name cp1 --cloud-init cloudinit.yaml --disk 20G --memory 8G --cpus 2
```
```yaml
#cloud-config
package_update: true
package_upgrade: true
write_files:
- path: /etc/modules-load.d/containerd.conf
content: |
overlay
br_netfilter
- path: /etc/sysctl.d/99-kubernetes-cri.conf
content: |
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
runcmd:
# Disable swap
- swapoff -a
- sed -i '/swap/d' /etc/fstab
# Load necessary modules
- modprobe overlay
- modprobe br_netfilter
# Apply sysctl params
- sysctl --system
# Remove conflicting packages
- for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do apt-get remove -y $pkg; done
# Install required packages
- apt-get install -y ca-certificates curl gpg apt-transport-https
# Add Docker GPG key
- mkdir -p -m 0755 /etc/apt/keyrings
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
- chmod a+r /etc/apt/keyrings/docker.asc
# Add Docker repository
- echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo ${UBUNTU_CODENAME:-$VERSION_CODENAME}) stable" | tee /etc/apt/sources.list.d/docker.list
# Update package lists
- apt-get update
# Install containerd
- apt-get install -y containerd.io
# Configure containerd
- mkdir -p /etc/containerd
- containerd config default | tee /etc/containerd/config.toml
- sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
- systemctl restart containerd
- systemctl enable containerd
# Add Kubernetes GPG key
- curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
# Add Kubernetes repository
- echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
# Update package lists
- apt-get update
# Install kubelet, kubeadm, and kubectl
- apt-get install -y kubelet kubeadm kubectl
- apt-mark hold kubelet kubeadm kubectl
# Enable kubelet service
- systemctl enable --now kubelet
# pull kubeadm related image
- sudo kubeadm config images pull
```
When the control plane instance is running, you have to install a network plugin
from a third party provider.
# Install and Configure Network Plugin 'Calico'
Shell into the control plane instance with `multipass shell <instance-name>` and run
```bash
sudo kubeadm init --pod-network-cidr=192.168.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.1/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.1/manifests/custom-resources.yaml
watch kubectl get pods -n calico-system
```
Wait until all the pods are up and running.
Afterwards run
```bash
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
```
It should return something like
```bash
node/<your-hostname> untainted
```
Then taint the control plane to not get any schedules, only the workers
```bash
kubectl taint nodes <control-plane-name-or-ip> node-role.kubernetes.io/control-plane=:NoSchedule
```
Confirm that you have a node running in your cluster
```bash
kubectl get nodes -o wide
```
It should return something like this
```bash
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
<your-hostname> Ready master 52m v1.12.2 10.128.0.28 <none> Ubuntu 18.04.1 LTS 4.15.0-1023-gcp docker://18.6.1
```
Perhaps remove the load balancer label from the list of backend servers
```bash
kubectl label nodes --all node.kubernetes.io/exclude-from-external-load-balancers-
```
# Adding Linux Worker Nodes
Run the same `cloudinit.yaml` file as the control plane instance.
For example
```bash
multipass launch --name worker1 --cloud-init cloudinit.yaml --disk 20G --memory 8G --cpus 2
```
When the instance is ready, shell into the instance with `multipass shell worker1` and run
```bash
sudo kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash>
```
[Important]
To get the exact information for the join run
```bash
multipass shell <controlplane-name>
```
When inside, run
```bash
sudo kubeadm token create --print-join-command
```
The information/token was generated with the kubeadm init command.
If you need to recover the information, shell into `multipass shell cp1`
and run
```bash
# Run this on a control plane node
sudo kubeadm token list
```
If you do not want to shell into the control plane you should be able to run (from dev pc)
```bash
multipass exec cp1 sudo kubeadm token list
```
which should return something like this
```bash
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
evv51i.5p35yitsiva6sjer 23h 2025-02-05T20:52:23Z authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token
```
By default, node join tokens expire after 24 hours.
If you are joining a node to the cluster after the current token has expired,
you can create a new token by running the following command on the control plane node:
```bash
# Run this on a control plane node
sudo kubeadm token create
```
The output is similar to this:
```bash
5didvk.d09sbcov8ph2amjw
```
If you don't have the value of `--discovery-token-ca-cert-hash`,
you can get it by running the following commands on the control plane node:
```bash
# Run this on a control plane node
sudo cat /etc/kubernetes/pki/ca.crt | openssl x509 -pubkey | openssl rsa -pubin -outform der 2>/dev/null | \
openssl dgst -sha256 -hex | sed 's/^.* //'
```
The output is similar to:
```bash
8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78
```
The output of the kubeadm join command should look something like:
```bash
[preflight] Running pre-flight checks
... (log output of join workflow) ...
Node join complete:
* Certificate signing request sent to control-plane and response
received.
* Kubelet informed of new secure connection details.
Run 'kubectl get nodes' on control-plane to see this machine join.
```
A few seconds later, you should notice this node in the output from kubectl get nodes.
(for example, run kubectl on a control plane node).
[Note]
As the cluster nodes are usually initialized sequentially,
the CoreDNS Pods are likely to all run on the first control plane node.
To provide higher availability,
please rebalance the CoreDNS Pods with
```bash
kubectl -n kube-system rollout restart deployment coredns
```
after at least one new node is joined.
@@ -1,170 +0,0 @@
---
- name: Setup Kubernetes Node with Docker
hosts: k8s_nodes
become: true
tasks:
- name: Update package cache
apt:
update_cache: yes
- name: Upgrade all packages
apt:
upgrade: yes
- name: Create kernel module configuration
copy:
dest: /etc/modules-load.d/k8s.conf
content: |
overlay
br_netfilter
mode: '0644'
- name: Create sysctl configuration
copy:
dest: /etc/sysctl.d/99-kubernetes-cri.conf
content: |
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
mode: '0644'
- name: Disable swap
block:
- name: Disable swap immediately
command: swapoff -a
changed_when: false
- name: Remove swap from fstab
replace:
path: /etc/fstab
regexp: '^([^#].*?\sswap\s+sw\s+.*)$'
replace: ''
- name: Load kernel modules
command: "modprobe {{ item }}"
loop:
- overlay
- br_netfilter
changed_when: false
- name: Apply sysctl params
command: sysctl --system
changed_when: false
- name: Remove conflicting packages
apt:
name:
- docker
- docker.io
- docker-doc
- docker-compose
- docker-compose-v2
- podman-docker
- containerd
- runc
state: absent
- name: Install required packages
apt:
name:
- ca-certificates
- curl
- gnupg
state: present
- name: Create keyrings directory
file:
path: /etc/apt/keyrings
state: directory
mode: '0755'
- name: Add Docker GPG key
shell: |
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
- name: Add Docker repository
shell: |
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
- name: Update package lists
apt:
update_cache: yes
- name: Install Docker packages
apt:
name:
- docker-ce
- docker-ce-cli
- containerd.io
- docker-buildx-plugin
- docker-compose-plugin
state: present
- name: Create Docker daemon configuration
copy:
dest: /etc/docker/daemon.json
content: |
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
mode: '0644'
create: yes
directory_mode: '0755'
- name: Create Docker service directory
file:
path: /etc/systemd/system/docker.service.d
state: directory
mode: '0755'
- name: Restart Docker
systemd:
name: docker
state: restarted
daemon_reload: yes
enabled: yes
- name: Add Kubernetes GPG key
shell: |
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
- name: Add Kubernetes repository
shell: |
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
- name: Update package lists again
apt:
update_cache: yes
- name: Install Kubernetes packages
apt:
name:
- kubelet
- kubeadm
- kubectl
state: present
- name: Hold Kubernetes packages
dpkg_selections:
name: "{{ item }}"
selection: hold
loop:
- kubelet
- kubeadm
- kubectl
- name: Enable kubelet service
systemd:
name: kubelet
state: started
enabled: yes
- name: Pull kubeadm images
command: kubeadm config images pull
changed_when: false
@@ -1,73 +0,0 @@
#cloud-config
package_update: true
package_upgrade: true
write_files:
- path: /etc/modules-load.d/containerd.conf
content: |
overlay
br_netfilter
- path: /etc/sysctl.d/99-kubernetes-cri.conf
content: |
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
runcmd:
# Disable swap
- swapoff -a
- sed -i '/swap/d' /etc/fstab
# Load necessary modules
- modprobe overlay
- modprobe br_netfilter
# Apply sysctl params
- sysctl --system
# Remove conflicting packages
- for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do apt-get remove -y $pkg; done
# Install required packages
- apt-get install -y ca-certificates curl gpg apt-transport-https
# Add Docker GPG key
- mkdir -p -m 0755 /etc/apt/keyrings
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
- chmod a+r /etc/apt/keyrings/docker.asc
# Add Docker repository
- echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo ${UBUNTU_CODENAME:-$VERSION_CODENAME}) stable" | tee /etc/apt/sources.list.d/docker.list
# Update package lists
- apt-get update
# Install containerd
- apt-get install -y containerd.io
# Configure containerd
- mkdir -p /etc/containerd
- containerd config default | tee /etc/containerd/config.toml
- sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
- systemctl restart containerd
- systemctl enable containerd
# Add Kubernetes GPG key
- curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
# Add Kubernetes repository
- echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
# Update package lists
- apt-get update
# Install kubelet, kubeadm, and kubectl
- apt-get install -y kubelet kubeadm kubectl
- apt-mark hold kubelet kubeadm kubectl
# Enable kubelet service
- systemctl enable --now kubelet
# pull kubeadm related image
- sudo kubeadm config images pull
@@ -1,6 +0,0 @@
[k8s_nodes]
node1 ansible_host=10.203.146.243
[k8s_nodes:vars]
ansible_user=ubuntu
ansible_ssh_private_key_file=~/.ssh/id_rsa
@@ -1,8 +0,0 @@
[k8s_nodes]
node1 ansible_host=10.203.146.216
[k8s_nodes:vars]
ansible_user=ubuntu
ansible_ssh_private_key_file=~/.ssh/id_rsa
ansible_become=true
@@ -1,13 +0,0 @@
#!/bin/bash
if [ "$#" -eq 0 ]; then
echo "Usage: $0 instance_name [instance_name2 ... instance_nameN]"
exit 1
fi
# Iterate over all provided instance names
for INSTANCE in "$@"; do
echo "Launching instance: ${INSTANCE}..."
multipass launch --name "${INSTANCE}" --cloud-init cloudinit.yaml --disk 20G --memory 8G --cpus 2
done
@@ -1,36 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: asyncapi
name: asyncapi-deployment
namespace: asyncapi
spec:
replicas: 2
selector:
matchLabels:
app: asyncapi
template:
metadata:
labels:
app: asyncapi
spec:
containers:
- name: asyncapi
image: iitaks.azurecr.io/asyncapi:1.2
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: asyncapi-service
namespace: asyncapi
spec:
selector:
app: asyncapi
type: NodePort
ports:
- protocol: TCP
port: 80
targetPort: 80
@@ -1,97 +0,0 @@
# Installation
```bash
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
```
```bash
sudo apt-get install containerd.io
```
---
# Cheat Sheet
## **1. Images (Pull, List, Remove)**
| Action | Docker | containerd (`ctr`) |
|--------|--------|------------------|
| Pull an image | `docker pull busybox` | `sudo ctr images pull docker.io/library/busybox:latest` |
| List images | `docker images` | `sudo ctr images list` |
| Remove an image | `docker rmi busybox` | `sudo ctr images rm docker.io/library/busybox:latest` |
---
## **2. Containers (Run, List, Stop, Remove)**
| Action | Docker | containerd (`ctr`) |
|--------|--------|------------------|
| Run a container (interactive) | `docker run -it busybox sh` | `sudo ctr run --rm --tty docker.io/library/busybox:latest my-container sh` |
| Run a container (detached) | `docker run -d busybox` | `sudo ctr run -d --tty docker.io/library/busybox:latest my-container sh` |
| List running containers | `docker ps` | `sudo ctr tasks ls` |
| Stop a running container | `docker stop my-container` | `sudo ctr tasks kill my-container` |
| Remove a stopped container | `docker rm my-container` | `sudo ctr containers delete my-container` |
---
## **3. Container Execution (Attach, Exec, Logs)**
| Action | Docker | containerd (`ctr`) |
|--------|--------|------------------|
| Attach to a running container | `docker attach my-container` | `sudo ctr tasks exec --tty my-container sh` |
| Execute a command inside a container | `docker exec -it my-container sh` | `sudo ctr tasks exec --tty my-container sh` |
| View container logs | `docker logs my-container` | **Not directly available in `ctr`** (use `crictl logs <container-id>` if using Kubernetes) |
---
## **4. Container Lifecycle Management**
| Action | Docker | containerd (`ctr`) |
|--------|--------|------------------|
| Restart a container | `docker restart my-container` | `sudo ctr tasks kill -s SIGTERM my-container && sudo ctr run -d --tty docker.io/library/busybox:latest my-container sh` |
| Pause a container | `docker pause my-container` | **Not directly supported** |
| Unpause a container | `docker unpause my-container` | **Not directly supported** |
---
## **5. Volumes & Networking**
Unlike Docker, `containerd` does **not** manage networking or volumes directly. You need to use **CNI (Container Network Interface)** for networking and manually mount host volumes when running a container.
For example, in Docker:
```sh
docker run -v /host/path:/container/path busybox
```
In `containerd`, you must specify the mount manually:
```sh
sudo ctr run --mount type=bind,src=/host/path,dst=/container/path,options=rbind:rw docker.io/library/busybox:latest my-container sh
```
---
## **6. Checking System Info**
| Action | Docker | containerd (`ctr`) |
|--------|--------|------------------|
| Show system info | `docker info` | `sudo ctr version && sudo ctr plugins ls` |
---
## **7. Removing Everything (Clean Up)**
| Action | Docker | containerd (`ctr`) |
|--------|--------|------------------|
| Remove all stopped containers | `docker container prune` | `sudo ctr containers list -q | xargs -r sudo ctr containers delete` |
| Remove all images | `docker image prune -a` | `sudo ctr images list -q | xargs -r sudo ctr images rm` |
---
### **Summary**
- `ctr` is more **low-level** than Docker, so it lacks built-in networking, volume management, and logging.
- You must manually handle networking using **CNI**.
- For Kubernetes workloads, you usually use `crictl` instead of `ctr`.
- `containerd` focuses purely on running containers; it does not replace Docker Compose or Swarm.
@@ -1,138 +0,0 @@
# Install kubeadm for Kubernetes version 1.32
## Prerquisites
- A compatible Linux host
- 2 GB or more of RAM per machine
- 2 CPUs or more for control plane machines.
- Full network connectivity between all machines in the cluster (public or private network is fine).
- Unique hostname, MAC address, and product_uuid for every node.
- Verify the `MAC address` and `product_uuid` are unique for every node
You can get the `MAC address` of the network interfaces using the command `ip link` or `ifconfig -a`
The `product_uuid `can be checked by using the command `sudo cat /sys/class/dmi/id/product_uuid`
It is very likely that hardware devices will have unique addresses, although some virtual machines may have identical values.
Kubernetes uses these values to uniquely identify the nodes in the cluster.
If these values are not unique to each node, the installation process may fail.
- Certain ports are open on your machines:
### Control plane
|Protocol|Direction|Port Range|Purpose|Used By|
|---|---|---|---|---|
|TCP|Inbound|6443|Kubernetes API server|All|
|TCP|Inbound|2379-2380|etcd server client API|kube-apiserver, etcd|
|TCP|Inbound|10250|Kubelet API|Self, Control plane|
|TCP|Inbound|10259|kube-scheduler|Self|
|TCP|Inbound|10257|kube-controller-manager|Self|
Although etcd ports are included in control plane section,
you can also host your own etcd cluster externally or on custom ports.
### Worker node(s)
|Protocol|Direction|Port Range|Purpose| Used By |
|---|---|---|---|---|
|TCP|Inbound|10250|Kubelet API|Self, Control plane |
|TCP|Inbound|10256|kube-proxy|Self, Load balancers |
|TCP|Inbound|30000-32767|NodePort Services†|All|
† Default port range for NodePort Services.
All default port numbers can be overridden.
When custom ports are used those ports need to be open instead of defaults mentioned here.
One common example is API server port that is sometimes switched to 443.
Alternatively, the default port is kept as is and API server is put behind a load balancer that listens on 443
and routes the requests to API server on the default port.
### Check required ports
These required ports need to be open in order for Kubernetes components to communicate with each other.
You can use tools like netcat to check if a port is open. For example:
```bash
nc 127.0.0.1 6443 -v
```
The pod network plugin you use may also require certain ports to be open.
Since this differs with each pod network plugin,
please see the documentation for the plugins about what port(s) those need.
## Swap Configuration
The default behavior of a kubelet is to fail to start if swap memory is detected on a node.
This means that swap should either be disabled or tolerated by kubelet.
- To tolerate swap, add `failSwapOn:` false to kubelet configuration or as a command line argument.
Note: even if `failSwapOn: false` is provided, workloads wouldn't have swap access by default.
This can be changed by setting a `swapBehavior`, again in the kubelet configuration file.
To use swap, set a `swapBehavior` other than the default `NoSwap ` setting.
- To disable swap, `sudo swapoff -a` can be used to disable swapping temporarily.
To make this change persistent across reboots, make sure swap is disabled in config files like `/etc/fstab`, `systemd.swap`
depending how it was configured on your system
## Install Container Runtime
Use containerd
Make sure to remove anything docker/container related installations
```bash
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done
```
```bash
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
```
```bash
sudo apt-get install containerd.io
```
## Install kubeadm, kubelet, and kubectl
You will install these packages on all of your machines:
- `kubeadm`: the command to bootstrap the cluster.
- `kubelet`: the component that runs on all of the machines in your cluster and does things like starting pods and containers.
- `kubectl`: the command line util to talk to your cluster.
1. Update the apt package index and install packages needed to use the Kubernetes apt repository:
```bash
sudo apt-get update
# apt-transport-https may be a dummy package; if so, you can skip that package
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
```
2. Download the public signing key for the Kubernetes package repositories.
The same signing key is used for all repositories so you can disregard the version in the URL:
```bash
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
```
3. Add the appropriate Kubernetes apt repository.
Please note that this repository have packages only for Kubernetes 1.32;
for other Kubernetes minor versions, you need to change the Kubernetes minor version in the URL to match your desired minor version
(you should also check that you are reading the documentation for the version of Kubernetes that you plan to install).
```bash
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
```
4. Update the apt package index, install kubelet, kubeadm and kubectl, and pin their version:
```bash
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
```
(Optional) Enable the kubelet service before running kubeadm:
```bash
sudo systemctl enable --now kubelet
```
The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do.
@@ -1,19 +0,0 @@
# minimal-postgres.yaml
apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
name: acid-minimal-cluster
spec:
teamId: "acid"
volume:
size: 1Gi
numberOfInstances: 3 # One master and two replicas for HA
postgresql:
version: "15"
resources:
requests:
cpu: 100m
memory: 100Mi
limits:
cpu: 500m
memory: 500Mi