Workloads

DaemonSets & Node-level Workloads

● Intermediate ⏱ 12 min read

A DaemonSet ensures that exactly one copy of a pod runs on every node (or a selected subset of nodes) in the cluster. When a new node joins the cluster, the DaemonSet automatically schedules a pod on it. When a node is removed, that pod is garbage-collected. DaemonSets are the standard pattern for cluster-wide infrastructure agents: log collectors, metrics exporters, network plugins, and security scanners.

What Is a DaemonSet?

Deployments and ReplicaSets spread a fixed number of pods across available nodes. DaemonSets work differently: the number of pods equals the number of nodes. The DaemonSet controller watches for node additions and removals and adjusts pod count automatically.

3-node cluster — DaemonSet "node-exporter"

node-1

node-exporter-x4k2p

node-2

node-exporter-m9r7q

node-3

node-exporter-bk8v1

Add node-4 → DaemonSet automatically schedules node-exporter-<hash> on it

DaemonSets place exactly one pod per node — count tracks cluster size automatically

Use Cases

DaemonSets are the right tool whenever you need something running on every node, not just somewhere in the cluster:

Category	Examples
Log collection	Fluent Bit, Fluentd, Logstash — tail node logs and container stdout/stderr
Metrics & monitoring	Prometheus node-exporter — exposes kernel/hardware metrics per node
Network plugins (CNI)	Calico, Cilium, Flannel — each node needs its own CNI agent for pod networking
Storage drivers	Ceph, Longhorn — storage agents that interface with node-level block devices
Security & compliance	Falco, Sysdig — kernel-level syscall monitoring per node
Device plugins	NVIDIA GPU plugin — exposes GPU resources per node to the scheduler

DaemonSet YAML

daemonset.yaml — Prometheus node-exporter

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true          # bind to node's network namespace
      hostPID: true              # access host process tree
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.8.0
        args:
        - --path.rootfs=/host
        ports:
        - containerPort: 9100
          hostPort: 9100
        securityContext:
          readOnlyRootFilesystem: true
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "250m"
        volumeMounts:
        - name: root
          mountPath: /host
          readOnly: true
      volumes:
      - name: root
        hostPath:
          path: /

The DaemonSet spec is nearly identical to a Deployment spec — same selector, same template. There is no replicas field because the count is implicit: one per node.

Targeting Specific Nodes

By default a DaemonSet runs on every node, including control-plane nodes. You can restrict it to a subset using nodeSelector or node affinity.

Target nodes with a specific label

spec:
  template:
    spec:
      nodeSelector:
        kubernetes.io/os: linux          # only Linux nodes
        node-role.kubernetes.io/worker: ""  # skip control-plane nodes

For more expressive rules, use affinity.nodeAffinity:

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-type
                operator: In
                values:
                - gpu

💡

DaemonSets bypass the scheduler for node selection

Regular pods go through the scheduler, which considers resource availability and affinity. DaemonSets schedule pods directly via the DaemonSet controller. This means DaemonSet pods can be placed on nodes that are under memory pressure or have PodDisruptionBudgets in play — the controller doesn't check those conditions.

Tolerations for Tainted Nodes

Control-plane nodes typically carry a taint (node-role.kubernetes.io/control-plane:NoSchedule) that prevents regular pods from being scheduled there. DaemonSets that need to run on control-plane nodes must explicitly tolerate this taint.

spec:
  template:
    spec:
      tolerations:
      # Run on control-plane nodes too
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      # Run on nodes marked as not-ready (e.g. network agents need to start early)
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute

Network plugins (CNI) must tolerate not-ready and unreachable — they're responsible for bringing nodes into a ready state, so they must start before the node is marked Ready.

Host Path & Host Network

Node-level agents frequently need access to the host's filesystem, network, or process table — things that normal pods don't touch. DaemonSets routinely use:

Feature	Field	Use case
hostPath volume	`volumes[].hostPath`	Mount node directories (`/var/log`, `/proc`, `/sys`) into the pod
hostNetwork	`spec.hostNetwork: true`	Pod uses the node's network namespace — sees host interfaces, binds to host ports
hostPID	`spec.hostPID: true`	Pod sees all host processes (required for syscall monitors like Falco)
hostPort	`containerPort.hostPort`	Expose container port directly on the node's IP (used by node-exporter)

⚠️

Host access requires elevated permissions

hostNetwork, hostPID, and hostPath with sensitive paths break pod isolation. They require explicit allow-listing in Pod Security Standards (PSS). In production clusters, DaemonSets using these features must be deployed in namespaces with the privileged PSS policy — never in application namespaces.

Update Strategies

DaemonSets support the same two update strategies as StatefulSets:

Strategy	Behaviour	When to use
RollingUpdate (default)	Updates one pod at a time across nodes. Old pod is killed before new pod starts on the same node.	Most DaemonSets — log agents, exporters.
OnDelete	Pod is only updated when you manually delete the old one.	CNI plugins and storage drivers where you want to control exactly when each node is updated.

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1    # update 1 node at a time (default)

DaemonSet vs Deployment

Question	Use DaemonSet	Use Deployment
Do you need one pod per node?	Yes	No
Do you need the pod on every node automatically?	Yes	No
Does the pod need host filesystem/network access?	Usually yes	Rarely
Do you need a fixed replica count independent of nodes?	No	Yes
Is this a user-facing service or API?	No	Yes

If you find yourself setting replicas to the number of nodes and manually managing pod-to-node assignment in a Deployment, you want a DaemonSet.

kubectl Commands

# Apply a DaemonSet
kubectl apply -f daemonset.yaml

# Check status — DESIRED should match node count
kubectl get daemonset node-exporter -n monitoring

# List the pods it created
kubectl get pods -l app=node-exporter -n monitoring -o wide

# Check which node each pod is on
kubectl get pods -l app=node-exporter -n monitoring \
  -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName'

# Describe for events and conditions
kubectl describe daemonset node-exporter -n monitoring

# Trigger a rolling update (after editing the image)
kubectl rollout status daemonset/node-exporter -n monitoring

# Roll back
kubectl rollout undo daemonset/node-exporter -n monitoring