CNI Plugins — Calico, Flannel, Cilium

Kubernetes defines rules for how pods must be able to communicate — every pod gets a unique IP, pods on different nodes can talk directly without NAT. But Kubernetes itself doesn't implement any of this. That's the job of a CNI plugin. The choice of plugin determines how pods get IPs, what performance characteristics you get, and whether Network Policies are enforced at all.

Kubernetes Networking Model

Kubernetes mandates three guarantees, but leaves the implementation entirely to the CNI plugin:

• Every pod gets a unique IP — no two pods in the cluster share an IP address
• Pods can reach each other directly — using pod IPs, without NAT, regardless of which node they're on
• Nodes can reach pods directly — using pod IPs, without NAT

These rules form the "flat networking model". Every pod behaves as if it were on the same LAN, even though in reality they may be on different physical machines in different availability zones.

What Is CNI?

CNI (Container Network Interface) is a specification — a simple contract between the container runtime and a network plugin. When a pod is created, the kubelet calls the CNI plugin binary with the container details. The plugin allocates an IP, creates a virtual network interface in the pod's network namespace, and sets up routing so the pod can send and receive packets.

# 1. kubelet creates the pod (sandbox container)
# 2. kubelet calls the CNI plugin: ADD <container-id> <netns-path>
# 3. CNI plugin:
#    - allocates an IP from IPAM (e.g. 10.244.1.7)
#    - creates a veth pair: one end in pod netns, one end on the host
#    - assigns the IP to the pod-side veth (eth0 inside the pod)
#    - sets up routing rules on the host
#    - returns: { "ip": "10.244.1.7/24", "gateway": "10.244.1.1" }
# 4. pod can now send/receive packets

CNI plugins are just binaries dropped into /opt/cni/bin/ on every node, configured via JSON files in /etc/cni/net.d/. Multiple plugins can chain together — for example, a VXLAN plugin for routing combined with a bandwidth plugin for QoS.

How a Pod Gets Its IP

IP allocation is handled by the CNI's IPAM (IP Address Management) component. Most plugins divide the cluster's pod CIDR (e.g. 10.244.0.0/16) into per-node subnets, then allocate individual IPs from the node's subnet.

Overlay vs Underlay

To send a packet from pod-a on node-1 to pod-c on node-2, the CNI plugin must route the packet across the physical network. There are two main approaches:

Flannel

Flannel is the simplest option — VXLAN overlay, straightforward setup, no extras. It runs a flanneld DaemonSet on each node that creates a flannel.1 virtual network device and maintains a mapping of which node owns which pod CIDR.

• Networking: VXLAN overlay (can also use host-gw for single-subnet clusters)
• Network Policy: Not supported — create NetworkPolicy objects but nothing enforces them
• Performance: Good for most workloads; VXLAN overhead is typically small at modern speeds
• Best for: Simple clusters, dev/test environments, when you don't need Network Policies

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

Calico

Calico supports both overlay (VXLAN or IP-in-IP) and BGP underlay modes. It enforces standard Kubernetes Network Policies and extends them with its own GlobalNetworkPolicy and NetworkPolicy CRDs that add per-node egress rules, DNS-based policies, and more.

• Networking: VXLAN (default), IP-in-IP, or native BGP routing
• Network Policy: Full enforcement + extended CalicoNetworkPolicy (L4, egress gateways)
• Performance: BGP mode delivers near-native throughput; VXLAN adds minimal overhead
• Best for: Production clusters that need Network Policy; on-premises clusters with BGP fabric

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/custom-resources.yaml

Cilium

Cilium replaces iptables with eBPF programs loaded directly into the Linux kernel. This eliminates the kube-proxy iptables chain entirely, enables L7-aware policies (allow/deny by HTTP method and path, gRPC service name), and provides rich observability through Hubble.

• Networking: eBPF-native; VXLAN overlay or native routing; can replace kube-proxy
• Network Policy: Standard K8s + CiliumNetworkPolicy (L7 HTTP, DNS, Kafka, gRPC)
• Performance: Best throughput and latency — eBPF bypasses the iptables overhead for Service routing
• Observability: Hubble UI shows real-time pod-to-pod traffic flows with policy verdicts
• Best for: Performance-critical clusters, zero-trust security requirements, modern cloud-native stacks
• Minimum kernel: 4.9 (full features require 5.10+)

helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.16.0 \
  --namespace kube-system \
  --set kubeProxyReplacement=true   # replace kube-proxy with eBPF

Comparison

kubectl Commands

# Check which CNI is installed (look for DaemonSets in kube-system)
kubectl get daemonsets -n kube-system

# Inspect CNI config on a node (requires node access)
ls /etc/cni/net.d/
cat /etc/cni/net.d/10-flannel.conflist   # or calico/cilium equivalent

# Check pod IPs and which node they're on
kubectl get pods -o wide

# Verify pod-to-pod connectivity
kubectl exec -it pod-a -- ping 10.244.1.3   # another pod's IP

# Cilium: check plugin status
kubectl exec -n kube-system ds/cilium -- cilium status

# Cilium: view live traffic flows (requires Hubble)
kubectl exec -n kube-system ds/cilium -- hubble observe --follow

# Calico: check node BGP peers (BGP mode)
kubectl exec -n kube-system ds/calico-node -- calicoctl node status

# Check IPAM allocation on a node
kubectl get node node-1 -o jsonpath='{.spec.podCIDR}'