Upgrading a Helm chart sounds simple until you are three environments deep, maintaining a vendored copy of a 2000-line values.yaml, and trying to remember whether the image tag format uses a v prefix or not. I have been doing chart upgrades like this for years. They are never truly hard, but they are reliably fiddly, and “reliably fiddly” is exactly where human error lives.

Recently I wrote a Claude Code skill to handle this class of work. This post is about what that looks like in practice: the actual skill file, the plan the agent produced for a real upgrade, and the broader idea behind encoding operational knowledge as reusable AI instructions.

The Problem with Routine Upgrades

Our Kubernetes monitoring setup uses kube-prometheus-stack deployed via Flux CD across three environments: experimental, dev, and prod. Each environment has its own values.yaml that contains the full upstream chart values, not a sparse overlay, but the complete file with our ACR image overrides baked in. When a new chart version drops, the upgrade checklist looks something like this:

1. Pull the new upstream chart and replace the local directory
2. Check whether CRDs changed (if so, handle migration before the upgrade lands)
3. Diff the upstream values.yaml — find every new key, every removed key, every image tag that changed
4. Apply those structural changes to all three environment values files, preserving our ACR paths and custom settings
5. Update the three containerImagesToMirror blocks in azure-pipelines.yml (one per environment — miss one and prod runs a stale image). Pipeline mirrors images from upstream to our ACR, so the tags there must match the env values files.
6. Update the README.md shields.io badges
7. Add a CHANGELOG.md entry
8. Lint and template-render all three environments to verify nothing is broken
9. Grep for stale version references
10. Commit

None of these steps are individually hard. But do them manually at 4pm on a Friday and you will probably miss something: a tag update in one pipeline block, a new upstream schema key that needs propagating, an old version reference hiding in a comment. I have seen all of these before.

The Skill

Claude Code supports custom skills, markdown files that describe a workflow and live under .claude/skills/. When you invoke a skill, the agent follows the instructions in the file as if they were part of its operating context.

The key insight is that a skill is not a script. It is not a Bash file that blindly executes a sequence of commands. It is a set of instructions that the agent interprets, bringing its own judgment to ambiguous situations, like figuring out whether a new upstream key was intentionally omitted from an env file, or whether a changed image tag format means the v prefix convention changed.

Here is the actual helm-chart-upgrade skill file, unedited:

---
name: helm-chart-upgrade
description:
  Guides a full Helm chart upgrade for repositories that bundle an upstream
  chart with environment values files and an Azure DevOps pipeline.

# Helm Chart Upgrade

This skill guides a complete, safe Helm chart upgrade for repos that follow this pattern:
  - A local chart directory wrapping an upstream Helm chart (e.g. `kyverno/`)
  - Three environment values files (`environments/{experimental,dev,prod}/values.yaml`)
    containing the **full** upstream chart values
  - An Azure DevOps pipeline (`azure-pipelines.yml`) with `containerImagesToMirror`
    lists per environment
  - A `README.md` with shield.io version badges
  - A `CHANGELOG.md`
---

## Step 0 — Collect inputs

Before touching anything, confirm these five values. Extract from the user's message
first; only ask about what is missing.

| Input                  | Example           |
| ---------------------- | ----------------- |
| Local chart directory  | `kyverno`         |
| Helm repo + chart name | `kyverno/kyverno` |
| Old app version tag    | `v1.17.1`         |
| New app version tag    | `v1.17.2`         |
| New Helm chart version | `3.7.2`           |

The app version tag and Helm chart version often differ (e.g. app `v1.17.2`,
chart `3.7.2`). Confirm both.

**`v` prefix consistency** — before doing any version replacement, check the existing
`containerImagesToMirror` entries to determine whether images are tagged `v1.2.3` or
`1.2.3`. Match that format exactly in every replacement. Never assume the `v` prefix
is present or absent.

---

## Step 1 — Plan first, change nothing

Enter plan mode. Before any file is touched:

1. Find every file referencing the old version:
   ```bash
   grep -r "<old-version>" --include="*.yaml" --include="*.yml" --include="*.md" . \
     | grep -v ".git/"
   ```
2. Write a checklist of all files and what changes each needs.
3. Identify any potential breaking changes in the new chart version (schema changes,
   removed keys, etc.) and how they will be reconciled in env values files.
4. Inform the user of the full scope of changes that will be made, including any manual
   steps they must take (e.g. applying CRDs if they changed).
5. Get explicit user approval before proceeding.

---

## Step 1.5 — Read the upstream release notes

Before touching any file, fetch the chart's changelog or GitHub release notes for the
version range being crossed. Even patch bumps occasionally include a "migration required"
note. Summarise any breaking changes (or note "none found") in the plan so the user is
informed before implementation begins.

---

## Step 2 — Pull the upstream chart

```bash
helm repo update
helm pull <helm-repo/chart> --version <new-chart-version> --untar --untardir /tmp/chart-upgrade
```

Replace the local chart directory:

```bash
rm -rf ./<chart-dir>/
cp -r /tmp/chart-upgrade/<chart-name> ./<chart-dir>/
```

Verify the pull succeeded before continuing.

**CRD check** — Helm does not upgrade CRDs by default. After pulling, diff the `crds/`
directory:

```bash
git diff HEAD <chart-dir>/crds/
```

If CRDs changed, flag it to the user — they must apply CRDs manually with
`kubectl apply -f <chart-dir>/crds/` before the Helm release is upgraded, or the
cluster will have schema drift.

---

## Step 3 — Identify schema changes in values.yaml

```bash
git diff HEAD <chart-dir>/values.yaml
```

Focus on **additions** — new top-level keys or sections that were not present in the
old chart. These need to be propagated to the environment values files.

**Also check for removed or renamed keys** — a key that exists in env values files but
no longer exists in the new upstream `values.yaml` will be silently ignored by Helm,
masking misconfiguration. Identify removals explicitly:

```bash
comm -23 <(grep -E "^[a-z]" environments/experimental/values.yaml | sort) \
         <(grep -E "^[a-z]" <chart-dir>/values.yaml | sort)
```

Report any removed top-level keys to the user before proceeding.

**Check sub-chart dependency versions** — if `<chart-dir>/Chart.yaml` has a
`dependencies:` block, diff it:

```bash
git diff HEAD <chart-dir>/Chart.yaml
```

Sub-charts (e.g. `common`, `postgresql`) may have their own image tags that appear in
`containerImagesToMirror` and need separate version bumps.

---

## Step 4 — Reconcile environment values files

The env values files contain the **complete upstream chart values**, not a sparse
override layer. They must be kept in full structural sync with the upstream
`<chart-dir>/values.yaml` after every upgrade — including key additions, key removals,
comments, blank lines, and ordering. The only exception is the _value_ of any key that
has been deliberately customised.

**Sync rules — apply in order:**

1. **Added keys** — if a key/block exists in the new upstream `values.yaml` but not in
   the env file, insert it at the same relative position as upstream (between the same
   neighbouring keys). Use the upstream default value.

2. **Removed keys** — if a key/block no longer exists in the new upstream `values.yaml`
   but is still present in the env file, remove it from the env file. Exception: if the
   key is known to be a local-only addition (not derived from upstream), leave it in
   place and flag it to the user.

3. **Preserve customised values** — for keys that exist in both the env file and
   upstream, keep the env file's value unchanged if it differs from the upstream default
   (ACR image repository paths, resource limits, Azure Workload Identity annotations,
   replica counts, etc.). Do not overwrite customised values with upstream defaults.

4. **Sync structure** — comments, blank lines, ordering, and indentation should follow
   the upstream `values.yaml`. Do not invent or retain env-file-specific formatting that
   diverges from upstream.

**Apply to all three environments** — experimental, dev, and prod must all receive the
same structural reconciliation.

**How to find the insertion point for new blocks:**

1. In the new `<chart-dir>/values.yaml`, identify the key immediately _before_ and
   immediately _after_ the new section.
2. Grep for the preceding key in each env values file to locate the exact insertion line.
3. Insert the new block between those keys.

**Images that inherit from appVersion** — component images that have `tag: ~` or no
explicit tag set automatically pick up `appVersion` from `Chart.yaml`. Do not add
explicit version tags for them; leave them as-is.

---

## Step 5 — Update azure-pipelines.yml

The pipeline has **three separate `containerImagesToMirror` blocks** — one each for
experimental, dev, and prod. All three must be updated. Missing even one leaves a stale
tag in production.

For each block: replace every image entry containing `:<old-version>` with
`:<new-version>`.

**Do not change utility image versions** (`readiness-checker`, `kubectl`, `busybox`,
`curl`, etc.) unless the user explicitly asks. These are pinned independently of the
chart release.

**Re-read the file before each subsequent edit.** A Prettier post-edit hook reformats
YAML files in place. If you edit twice without re-reading, your `old_string` won't match
the reformatted file and the edit will fail silently.

---

## Step 6 — Update README.md

Find shield.io badge lines that reference the chart being upgraded and update the
version strings:

- App version badge: `<ChartName>-v<old>` → `<ChartName>-v<new>`
- Helm chart version badge: `<ChartName>_Helm_Chart-v<old-chart>` →
  `<ChartName>_Helm_Chart-v<new-chart>`
- Any inline version references in documentation text (e.g. example ACR mirror paths)

Re-read the file before each subsequent edit if Prettier is active.

---

## Step 7 — Update CHANGELOG.md

Prepend a new dated entry at the top of the file (immediately after the file header):

```markdown
## YYYY-MM-DD — <Chart Name> v<new-version>

### Changed

- Upgraded <Chart Name> to v<new-version> (Helm chart <new-chart-version>)
- Updated all container image tags in `azure-pipelines.yml` to v<new-version>
  across all three environments
```

If new schema sections were added to env values files, add a bullet for each:

```markdown
- New `<section-name>` section added to upstream chart; inserted into all environment
  values files with upstream defaults
```

---

## Step 8 — Verify

```bash
# Lint the chart against every environment
helm lint ./<chart-dir> -f environments/experimental/values.yaml
helm lint ./<chart-dir> -f environments/dev/values.yaml
helm lint ./<chart-dir> -f environments/prod/values.yaml
```

`helm lint` validates structure but passes even with some broken references. Follow with
a full template render, which catches missing keys, bad references, and schema violations
that lint misses:

```bash
helm template <release-name> ./<chart-dir> -f environments/experimental/values.yaml > /dev/null
helm template <release-name> ./<chart-dir> -f environments/dev/values.yaml > /dev/null
helm template <release-name> ./<chart-dir> -f environments/prod/values.yaml > /dev/null
```

```bash
# Confirm no stale old-version references remain
# (CHANGELOG historical entries are expected — exclude them)
grep -r "<old-version>" --include="*.yaml" --include="*.yml" --include="*.md" . \
  | grep -v "CHANGELOG.md" | grep -v ".git/"
```

The grep must return empty output. Fix any remaining references before committing.

---

## Step 9 — Commit

Use the `gitflow` skill to stage and commit all changed files atomically. Craft the
commit message to explain _why_ (the version bump) and _what_ changed (images, schema):

```
chore(<chart-name>): upgrade to v<new-version> (Helm chart <new-chart-version>)

Bump <Chart Name> from v<old-version> to v<new-version>.
- Updated containerImagesToMirror in azure-pipelines.yml across all three environments
- Reconciled new <section-name> schema section into all environment values files
```

Omit sections that did not apply to this upgrade.

A few things worth noting about how this is written:

It documents the traps inline. Step 5 explicitly warns about the Prettier post-edit hook and instructs the agent to re-read the file before each subsequent edit. That warning exists because I was burned by it once: the hook reformats YAML on every save, which means a second edit without re-reading produces an old_string that no longer matches the file. Writing it into the skill means it is handled correctly on every subsequent run.

The invariant in Step 4 is the most important line in the file. “Never overwrite customised values with upstream defaults.” Without this constraint, the agent has no way to know which values are intentionally different from upstream and which are just stale. Stating it explicitly means it is applied consistently every time.

Steps are numbered but the agent can reason across them. Step 1.5 exists because I added it after an upgrade where I missed a breaking change note that was buried in the release notes. It is out of numeric order but the agent reads it correctly in context.

A gitflow is a another custom skill I wrote to handle git operations with a structured commit message format. The helm-chart-upgrade skill calls it for the commit step, but it can also be invoked independently for other commits that follow the same format.

The Plan the Agent Produced

When I invoked /helm-chart-upgrade for the kube-prometheus-stack 83.5.1 → 85.1.3 upgrade, the agent entered plan mode and produced the following plan before touching a single file. This is the actual output, not a summary:

Context

Routine Helm chart version bump from 83.5.1 → 85.1.3. The prometheus-operator appVersion stays at v0.90.1, so no CRD changes are required. However, two meaningful changes land in this range:

1. Grafana jumps to v13 (13.0.1-security-01) — a major version upgrade from the current 12.4.3.
2. Distroless images become the default for Prometheus (explicit -distroless tag) and Node Exporter (distroless: true flag in sub-chart values). These are lean, shell-free variants — they work fine in production but kubectl exec shell access into those pods will not be available.

Step 0 — Confirmed Inputs

Release Notes Summary (83.5.1 → 85.1.3)

Image Changes (all 3 pipeline blocks + env values files)

Files to Change

Execution Steps

• Helm repo update + pull chart 85.1.3
• Replace chart directory
• CRD diff (expected: no changes since appVersion unchanged)
• Values diff — identify all structural additions/removals
• Sync environments/dev/values.yaml
• Sync environments/prod/values.yaml
• Sync environments/experimental/values.yaml
• Update azure-pipelines.yml — replace image tags in all 3 containerImagesToMirror blocks
• Update README.md — chart badge 83.5.1 → 85.1.3
• Update CHANGELOG.md — prepend entry dated 2026-05-19
• Lint + template render all 3 environments
• Stale-reference check — grep must return empty
• Commit via gitflow skill

Verification

# Lint
helm lint kube-prometheus-stack -f environments/experimental/values.yaml
helm lint kube-prometheus-stack -f environments/dev/values.yaml
helm lint kube-prometheus-stack -f environments/prod/values.yaml

# Full template render (catches missing keys and bad references)
helm template kube-prometheus-stack kube-prometheus-stack \
  -f environments/experimental/values.yaml > /dev/null
helm template kube-prometheus-stack kube-prometheus-stack \
  -f environments/dev/values.yaml > /dev/null
helm template kube-prometheus-stack kube-prometheus-stack \
  -f environments/prod/values.yaml > /dev/null

# No stale references (CHANGELOG historical entries expected — excluded)
grep -r "83\.5\.1" --include="*.yaml" --include="*.yml" --include="*.md" . \
  | grep -v "CHANGELOG.md" | grep -v ".git/"

Manual Action Required Before Deployment

None — appVersion unchanged (v0.90.1), CRDs are identical. Upgrade can proceed via normal pipeline after this PR lands.

Grafana 13 note — test dashboard rendering and any custom plugins in experimental environment before promoting to dev/prod.

I approved the plan, and the agent executed every step. The commit landed clean on the first attempt.

What the Agent Caught

Two things in this upgrade would have been easy to miss manually:

The distroless: true placement. The flag needed to go inside the existing image: block in each env values file, not at a sibling level. Placing it at the wrong indentation level would have produced a duplicate image: key and silent YAML parsing behaviour. The agent correctly identified the existing block structure and inserted the flag at the right depth.

The shields.io hyphen encoding. The badge URL for Grafana references 13.0.1-security-01. In a shields.io URL, hyphens inside the badge value must be doubled: 13.0.1--security--01. An obscure encoding rule the skill carries from when I first got it wrong. The agent applied it correctly without being prompted.

What Makes a Good Skill

A skill is essentially a runbook for an AI operator. A few things I learned from writing this one:

Start with the process you actually follow, not an idealized one. The Prettier hook warning in Step 5 exists because I hit that bug in a real session. The double-hyphen badge encoding in Step 6 exists because I published a broken badge once. Skills work best when they encode real institutional knowledge including the failure modes, not just the happy path.

Identify and state the invariants explicitly. “Never overwrite customised values with upstream defaults” is a constraint that the agent cannot infer from context alone. Without it, the agent has no way to know which values are intentionally different from upstream and which are just stale. Stating invariants as named rules means they are applied consistently without requiring per-upgrade review.

Plan-first is non-negotiable for destructive work. The requirement to get explicit approval before any file is touched might feel like friction. It is not. The plan step consistently surfaces things I would not have noticed until after the fact (like the Grafana major version crossing in this upgrade) when they are cheap to act on. After approval, execution is fast because all the thinking is already done.

Skills compose. The helm-chart-upgrade skill calls the gitflow skill for the commit step. Skills can depend on other skills, share conventions, and be invoked independently when needed.

The Broader Pattern

Helm chart upgrades are one example of a class of work that is: well-defined, multi-step, error-prone when done manually, and repeated often enough that the accumulated cost of mistakes is real. Infrastructure work is full of this class of work.

The traditional automation answer is a Bash script. Scripts are great for deterministic sequences but brittle at the edges: they cannot handle ambiguity, cannot ask clarifying questions, and cannot adapt when upstream schema structure changes in ways you did not anticipate. A script that worked for v83 might silently corrupt an env file on v85 because a new key appeared in a position the script was not expecting.

An AI agent with encoded domain knowledge handles these edge cases differently. It can inspect what it finds, reason about whether a structural change is a breaking rename or a new addition, and surface the ambiguity for human review rather than silently doing the wrong thing. The skill file is the mechanism for transferring that knowledge from your head to the agent, written once and applied consistently.

The agent is not infallible. The failure mode is better: when the agent encounters something unexpected, it surfaces it rather than proceeding. That is a much more useful behaviour than a script that silently produces a broken output.

Getting Started with Claude Code Skills

Skills live in .claude/skills/<skill-name>/SKILL.md. The format is plain markdown, no special syntax required. The agent reads the file as part of its context when you invoke /skill-name.

A few practical tips:

• Write down the manual process you follow today, including the parts where you have been burned before.
• Identify the invariants, the things that must never change regardless of what upstream does. Write them as explicit constraints.
• Add a verification section. The agent should always prove the change is correct before committing, not just report that it thinks it is.
• Keep skills focused. One workflow per skill file is easier to reason about and compose than one large file trying to cover everything.

If you are running Kubernetes in production and doing this kind of maintenance work manually, skills are worth the hour it takes to write one.

The Broader Toolbox: Loops, Hooks, MCP, and Subagents

Skills are one piece of what Claude Code offers. The toolbox goes further, and the other parts matter for infrastructure work specifically.

Loops let the agent run a workflow on a schedule or iterate until a condition is met. The same helm-chart-upgrade skill could be invoked in a loop that checks for new chart versions every morning, drafts the plan, and notifies you when there is something to approve. The agent does the reconnaissance; you do the approval. For teams managing a large number of charts across multiple clusters, this shifts chart maintenance from reactive (“someone noticed the chart is three versions behind”) to proactive (“I get a plan every time upstream ships”).

Hooks are shell commands that fire automatically before or after agent tool calls. The Prettier hook that reformats YAML files after every edit is itself a hook, one the blog site uses to keep files formatted consistently. But hooks can do much more in an operations context: run helm lint after every values file edit to catch schema errors immediately, create a git checkpoint before any destructive operation, or send a notification when the agent completes a phase. Hooks let you wrap the agent’s actions with your own guardrails without modifying the skill itself; the skill defines the workflow, hooks enforce the environment constraints.

MCP (Model Context Protocol) is the integration layer that connects agents to external systems as native tool calls. An agent with an MCP server for Azure can query Key Vault for secret metadata, check managed identity assignments, or verify federated credentials, without shell scripts or manual credential setup. For Kubernetes operations this opens up live cluster integrations: the agent can check whether a CRD version already exists before recommending a manual apply, verify that a Flux HelmRelease reached a Ready state after the pipeline ran, or inspect current resource limits before proposing changes. MCP turns the agent from a file editor into an operator with real-time visibility into the systems it is working on.

Subagents let a parent agent delegate work to parallel child instances, each with their own context window and tool access. For the upgrade workflow, that means reconciling three environment values files concurrently rather than sequentially, or researching breaking changes across a range of chart versions without loading a dozen changelogs into a single context. The parent agent orchestrates; the subagents do the reading and return structured results. In wider infrastructure use, the pattern scales further: verify CRD state across multiple clusters in parallel, run helm lint against a matrix of chart and environment combinations, or fan out a secret rotation across namespaces and collect the results in one place.

Together these capabilities describe a continuum. At one end: a human types /helm-chart-upgrade and approves a plan. At the other: a scheduled loop detects a new upstream release, generates the plan, runs the pipeline, monitors Flux reconciliation via MCP, and pages you only when something needs a decision. The same skill file powers both ends of that continuum; the difference is just how much of the surrounding workflow you have also encoded.

Conclusion

The pace at which AI agents are changing the way we work is real and it is accelerating. A year ago, the idea of writing a markdown file and having an AI reliably execute a 13-step infrastructure workflow (pulling a chart, diffing schemas, syncing three environment files, updating pipeline configs, verifying renders, and committing) would have sounded optimistic. Today it is a Tuesday afternoon.

The important shift is not that the work gets done faster, although it does. The shift that matters is that the work gets done consistently. The agent does not skip the stale-reference grep because it is late. It does not forget the third pipeline block because the first two looked fine. It does not miss the Grafana major version crossing in the release notes because it was already familiar with the chart. Every execution follows the full process, including the parts that humans tend to abbreviate under time pressure.

What I want to emphasise is that AI agents like Claude are not primarily a tool for software developers writing application code. The example in this post has no application code in it at all; it is entirely infrastructure: YAML files, Helm charts, pipeline configurations, badge URLs. The agent performed exactly as well here as it would on a code refactoring task, because the underlying capability is not “write code” but “follow a complex process with context-dependent judgment.” That capability applies equally well to:

• Rotating secrets across multiple Kubernetes namespaces following an audit trail
• Upgrading a Flux HelmRelease including drift detection and valuesFrom reconciliation
• Reviewing a pull request against a checklist of infrastructure security controls
• Onboarding a new cluster environment by replicating configuration from an existing one
• Auditing alert rule coverage against a list of SLOs

These are not software engineering tasks in the traditional sense. They are IT operations tasks, the kind that platform engineers, SREs, and cloud infrastructure engineers do every day. They are also exactly the kind of tasks where consistency matters most and where the cost of human error is highest.

None of this means you stop paying attention. The plan step exists precisely because a human needs to review what the agent intends to do before it does it. Approving the plan for the kube-prometheus-stack upgrade took me about two minutes: scanning the image changes table, confirming the Grafana 13 note was flagged, checking that CRDs were correctly identified as unchanged. That two-minute review is far more valuable than the hour of manual execution it replaces, because it happens at the right moment: before anything is touched, when the cost of catching a mistake is zero. You are not removing the human from the loop; you are moving the human to the part of the loop where their judgment actually matters.

The question worth asking is not “can I use an AI agent for this?” but “what does my current process look like written down?” If the answer is a mental checklist you carry in your head, or a Confluence page that is three versions out of date, or a script that breaks when anything changes, that is a skill waiting to be written.

The world is not waiting for this to happen. It is already happening. The only variable is whether the institutional knowledge that makes your operations reliable ends up encoded somewhere useful, or stays locked in the heads of the people who do the work today.

Over the past few months I have been building and operating a GitOps platform for Kubernetes clusters running on Azure. The driving motivation was a shift away from push-based CI/CD pipelines — where pipelines hold cluster credentials and kubectl apply changes directly — toward a pull-based model where an in-cluster operator continuously reconciles the live state against what Git says it should be. This post is a practical walkthrough of how I implemented that shift using Flux CD, the Flux Operator, Helm, and Azure DevOps.

If you are already familiar with basic GitOps concepts and want to see how things connect in a production-grade setup — including image scanning, signing, private registry mirroring, and a multi-environment onboarding model — this is the post for you.

Why GitOps?

GitOps is a way of managing infrastructure and application delivery where Git is the single source of truth. Any desired state change is made via a pull request, and an in-cluster operator continuously reconciles the live state against what Git says it should be. The key properties, as defined by the OpenGitOps principles, are:

• Declarative — the entire desired state is described declaratively
• Versioned and immutable — Git history is the canonical audit trail
• Pulled automatically — an agent in the cluster pulls state from Git (no external push into the cluster)
• Continuously reconciled — if live state drifts from desired state, the agent corrects it

For Kubernetes this model fits perfectly. Every resource is a YAML manifest, Git provides versioning and review workflows via pull requests, and controllers like Flux watch the repository and apply changes automatically. The alternative — pushing manifests from a CI/CD pipeline with kubectl apply — requires the pipeline to have cluster credentials, creates a push-based model that bypasses drift detection, and gives you no built-in reconciliation if someone applies something directly to the cluster.

With the “why” clear, let me walk through how these principles are put into practice: what the repository looks like, how the pipeline fits in, and where Flux takes over.

Architecture Overview

Flux Architecture

Flux CD is a set of Kubernetes controllers, each responsible for a specific piece of the GitOps loop. Understanding what each controller does makes the rest of the setup much easier to reason about:

• source-controller — watches Git repositories, Helm repositories, OCI registries, and S3-compatible buckets. It fetches content and makes it available as an in-cluster artifact that other controllers consume. This is the only controller that talks to external sources.
• kustomize-controller — watches Kustomization CRDs and applies the manifests they point to (plain YAML, Kustomize overlays, or files that happen to contain HelmRelease objects). It handles decryption, dependency ordering, health checks, and pruning of removed resources.
• helm-controller — watches HelmRelease CRDs and manages the full Helm lifecycle: install, upgrade, rollback, and uninstall. It reads chart artifacts from source-controller and renders them server-side.
• notification-controller — handles inbound webhooks (to trigger reconciliation) and outbound event notifications (Slack, Teams, GitHub commit status, etc.).
• image-reflector-controller and image-automation-controller — together they implement image update automation: scanning a registry for new tags and writing the updated tag back to Git.

These controllers communicate through the Kubernetes API only; they never talk directly to each other. source-controller produces Artifact objects; kustomize-controller and helm-controller consume them. This loose coupling means you can enable only the controllers you need.

Platform Layout

The platform manages multiple Kubernetes environments. The branch-to-environment mapping is straightforward:

Any push to these branches triggers an Azure DevOps pipeline that handles the infrastructure layer: scanning and mirroring Flux controller images to a private Azure Container Registry (ACR), deploying the Flux Operator via Helm, and applying the FluxInstance CRD that tells Flux how to configure itself. Once Flux is running, it takes over and continuously reconciles everything under clusters/<env>/systems/. No pipeline trigger required for day-to-day system changes.

Git Push / PR
     │
     ├── clusters/*/flux-instance.yaml           clusters/*/systems/** (PR only)
     │   clusters/*/flux-operator-values.yaml              │
     │   flux-operator/**                                  │
     ▼                                                     ▼
azure-pipelines.yaml                        azure-pipelines-systems.yaml
     │                                                     │
     ├── [On PR] Validate stage                [On PR] Validate stage
     │     ├── Scan Flux images (Trivy)            └── kubectl apply --dry-run=server
     │     ├── Helm dry-run of Flux Operator              clusters/<env>/systems/
     │     └── kubectl dry-run of FluxInstance
     │
     └── [On push] Apply stage
           ├── Scan images (Trivy)
           ├── Mirror images to private ACR (crane)
           ├── Sign images (Cosign)
           ├── Helm upgrade --install Flux Operator
           └── kubectl apply FluxInstance
                     │
                     ▼
              Flux Operator reconciles FluxInstance
                     │
                     ▼
              Flux controllers deployed in cluster
                     │
                     ▼
              Flux syncs clusters/<env>/systems/ from Git
                     │
                     ├── Reconciles GitRepository sources (system repos)
                     └── Reconciles Kustomization → HelmRelease or plain YAML

Repository Structure

The k8s-gitops repository is the single source of truth for the platform layer. Its layout reflects the branch-per-environment model directly:

k8s-gitops/
├── azure-pipelines.yaml               # Main pipeline — triggers and environment wiring
├── azure-pipelines-systems.yaml       # Systems manifest validation pipeline — PR-only
├── ci-cd-templates/                   # Reusable Azure DevOps pipeline templates
│   ├── flux.yaml                      # Top-level: validate (on PR) + apply (on push)
│   ├── flux-operator.yaml             # Helm deploy/upgrade of Flux Operator
│   ├── flux-instance.yaml             # kubectl apply of FluxInstance CRD
│   ├── flux-images.yaml               # Mirror → scan → sign each Flux image
│   ├── scan-image.yaml                # Trivy vulnerability scan step
│   ├── sign-image.yaml                # Cosign image signing step
│   └── systems-validate.yaml          # kubectl dry-run validation of systems/ manifests
├── clusters/
│   ├── experimental/
│   │   ├── flux-instance.yaml         # FluxInstance spec for this environment
│   │   ├── flux-operator-values.yaml  # Helm values for Flux Operator
│   │   └── systems/                   # Flux-managed resources (reconciled by Flux, not the pipeline)
│   │       └── <system>.yaml          # One file per system — GitRepository + Kustomization
│   ├── dev/
│   └── prod/
└── flux-operator/                     # Helm chart for the Flux Operator (vendored)

The key separation is between clusters/<env>/flux-instance.yaml and clusters/<env>/systems/:

• flux-instance.yaml and flux-operator-values.yaml are managed by the Azure DevOps pipeline — they define Flux itself.
• Everything under systems/ is managed by Flux directly — the pipeline never touches it.

Flux Operator and FluxInstance CRD

One of the first architectural decisions was how to install and lifecycle-manage Flux itself. The traditional approach — flux bootstrap — generates YAML and commits it to your repo. It works, but upgrading Flux means re-running bootstrap or hand-editing generated files. There is no declarative upgrade path.

The Flux Operator solves this. It introduces a FluxInstance CRD that lets you declare Flux’s desired state the same way you declare anything else in Kubernetes. The operator reads the spec and takes care of deploying, configuring, and upgrading the Flux controllers. You never run flux bootstrap again.

Installing the Flux Operator is a single Helm command:

helm upgrade --install flux-operator ./flux-operator \
  --namespace flux-system \
  --create-namespace \
  -f ./clusters/<env>/flux-operator-values.yaml

Once the operator is running, you apply the FluxInstance manifest:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
  name: flux
  namespace: flux-system
spec:
  distribution:
    version: "2.8.3" # Flux version to deploy
    registry: "myacr.azurecr.io/fluxcd" # Private ACR where images are mirrored
    variant: "upstream-alpine" # Required when using a private registry
  components:
    - source-controller
    - kustomize-controller
    - helm-controller
    - notification-controller
    - image-reflector-controller
    - image-automation-controller
  sync:
    kind: GitRepository
    provider: azure # Azure DevOps authentication
    url: "https://dev.azure.com/my-org/my-project/_git/k8s-gitops"
    ref: "refs/heads/experimental"
    path: "clusters/experimental/systems" # Flux watches this path
  kustomize:
    patches: [...] # Workload identity patches (see below)

The spec.distribution block is the key part. When you increment the version number and apply the manifest, the Flux Operator reconciles the change and upgrades all controllers automatically. No manual rollout, no bootstrap re-run. The operator also handles the case where the controllers drift from the declared state and corrects them.

All six Flux controllers are deployed: source-controller, kustomize-controller, helm-controller, notification-controller, image-reflector-controller, and image-automation-controller.

Azure Workload Identity — Secretless Git Authentication

Flux’s source-controller needs to pull from Azure DevOps Git repositories. The naive approach is to create a Personal Access Token (PAT), store it as a Kubernetes secret, and reference it in the GitRepository spec. This works but PATs expire, need rotation, and storing credentials in the cluster creates a credential management burden.

The better approach is Azure Workload Identity. The idea is to federate a Kubernetes service account with an Azure Managed Identity. When the source-controller pod authenticates to Azure DevOps, it uses the federated credential, with no secrets stored anywhere.

The setup involves three steps:

1. Create a Managed Identity with a federated credential:

# Create the managed identity
az identity create \
  --name aks-dev-flux-source-controller \
  --resource-group my-cluster-rg \
  --subscription "my-subscription"

# Create the federated credential linking it to the source-controller service account
az identity federated-credential create \
  --name aks-dev-flux-federated-credential \
  --identity-name aks-dev-flux-source-controller \
  --resource-group my-cluster-rg \
  --issuer <AKS_OIDC_ISSUER_URL> \
  --subject system:serviceaccount:flux-system:source-controller

2. Grant the identity read access to your Azure DevOps project — a project administrator adds it to the Readers group in Project Settings → Permissions.

3. Patch the source-controller service account and deployment via FluxInstance.spec.kustomize.patches:

kustomize:
  patches:
    - patch: |-
        apiVersion: v1
        kind: ServiceAccount
        metadata:
          name: source-controller
          annotations:
            azure.workload.identity/client-id: <AZURE_CLIENT_ID>
            azure.workload.identity/tenant-id: <AZURE_TENANT_ID>
      target:
        kind: ServiceAccount
        name: source-controller
    - patch: |-
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: source-controller
        spec:
          template:
            metadata:
              labels:
                azure.workload.identity/use: "true"
      target:
        kind: Deployment
        name: source-controller

The Flux Operator applies these patches when it deploys the controllers, so the workload identity configuration is fully declarative and version-controlled alongside everything else.

CI/CD Pipeline: Images, Scanning, Signing

The entry point for the whole Flux infrastructure pipeline is azure-pipelines.yaml. It triggers on pushes to the three environment branches (or on PRs touching those paths) and calls the flux.yaml template once per environment, passing the image list and environment-specific parameters:

# azure-pipelines.yaml
name: k8s-gitops-flux

trigger:
  branches:
    include:
      - experimental
      - master
      - dev
  paths:
    include:
      - clusters/*/flux-instance.yaml
      - clusters/*/flux-operator-values.yaml
      - flux-operator/*

pool: my-linux-agents

parameters:
  - name: helmReleaseName
    default: "flux-operator"
  - name: helmReleaseNamespace
    default: "flux-system"
  - name: helmChartPath
    default: "./flux-operator"
  - name: helmCreateNamespace
    type: boolean
    default: true

stages:
  # Experimental cluster CI/CD
  - template: ci-cd-templates/flux.yaml
    parameters:
      environment: "experimental"
      branch: "experimental"
      serviceConnection: "k8s-rbac-experimental-ado-sc"
      helmReleaseName: $
      helmReleaseNamespace: $
      helmChartPath: $
      helmValuesFilePath: "./clusters/experimental/flux-operator-values.yaml"
      helmCreateNamespace: $
      containerImages:
        - ghcr.io/controlplaneio-fluxcd/flux-operator:v0.45.1
        - ghcr.io/fluxcd/helm-controller:v1.5.3
        - ghcr.io/fluxcd/image-automation-controller:v1.1.1
        - ghcr.io/fluxcd/image-reflector-controller:v1.1.1
        - ghcr.io/fluxcd/kustomize-controller:v1.8.2
        - ghcr.io/fluxcd/notification-controller:v1.8.2
        - ghcr.io/fluxcd/source-controller:v1.8.1

  # Dev cluster CI/CD
  - template: ci-cd-templates/flux.yaml
    parameters:
      environment: "dev"
      branch: "dev"
      serviceConnection: "k8s-rbac-dev-ado-sc"
      helmReleaseName: $
      helmReleaseNamespace: $
      helmChartPath: $
      helmValuesFilePath: "./clusters/dev/flux-operator-values.yaml"
      helmCreateNamespace: $
      containerImages:
        - ghcr.io/controlplaneio-fluxcd/flux-operator:v0.45.1
        - ghcr.io/fluxcd/helm-controller:v1.5.3
        - ghcr.io/fluxcd/image-automation-controller:v1.1.1
        - ghcr.io/fluxcd/image-reflector-controller:v1.1.1
        - ghcr.io/fluxcd/kustomize-controller:v1.8.2
        - ghcr.io/fluxcd/notification-controller:v1.8.2
        - ghcr.io/fluxcd/source-controller:v1.8.1

  # Production cluster CI/CD
  - template: ci-cd-templates/flux.yaml
    parameters:
      environment: "prod"
      branch: "master"
      serviceConnection: "k8s-rbac-prod-ado-sc"
      helmReleaseName: $
      helmReleaseNamespace: $
      helmChartPath: $
      helmValuesFilePath: "./clusters/prod/flux-operator-values.yaml"
      helmCreateNamespace: $
      containerImages:
        - ghcr.io/controlplaneio-fluxcd/flux-operator:v0.45.1
        - ghcr.io/fluxcd/helm-controller:v1.5.3
        - ghcr.io/fluxcd/image-automation-controller:v1.1.1
        - ghcr.io/fluxcd/image-reflector-controller:v1.1.1
        - ghcr.io/fluxcd/kustomize-controller:v1.8.2
        - ghcr.io/fluxcd/notification-controller:v1.8.2
        - ghcr.io/fluxcd/source-controller:v1.8.1

The containerImages list is what the flux-images.sh script generates — on every upgrade you replace this block and nothing else.

One of the things that took the most thought was the image management story. Flux controller images come from ghcr.io/fluxcd (upstream) or ghcr.io/controlplaneio-fluxcd (enterprise). Pulling images directly from public registries in production clusters is a risk: rate limits, availability dependencies, no supply chain verification. The solution is to mirror all images to a private ACR on every release.

The pipeline handles this for each image:

1. Scan with Trivy — before copying anything, the source image is scanned for vulnerabilities. In audit mode the pipeline logs findings without failing; switching to enforce mode fails the pipeline on HIGH or CRITICAL CVEs. The Trivy database itself is hosted in private ACR to avoid GHCR rate limiting (I wrote about this in a previous post).

2. Mirror with crane — crane copy copies the image directly between registries without pulling it locally. Critically, it preserves the full OCI manifest index, so the digest in ACR is identical to the upstream digest. This matters for Cosign signature verification — the signature is tied to the digest. The Flux Operator uses the digest for flux-controller images by default, so the mirrored image must have the same digest for the signature to be valid. If signature verification fails, the pipeline fails. The same signature verification is enforced in the cluster before allowing the workload to run, thanks to Kyverno.

crane copy \
  ghcr.io/fluxcd/source-controller:v1.4.3 \
  myacr.azurecr.io/fluxcd/source-controller:v1.4.3

3. Sign with Cosign — after copying, the image is signed using a key stored in Azure Key Vault. The signature is pushed to ACR alongside the image. Signing only runs on push (not on PRs), and after signing the pipeline verifies the signature before proceeding.

All of this runs for every image in the containerImages list defined in azure-pipelines.yaml. When upgrading Flux, a helper script generates the updated image list:

./scripts/flux-images.sh v2.8.3 v0.45.1

The script uses the flux CLI and yq to produce a ready-to-paste containerImages block. You replace the list in azure-pipelines.yaml, update spec.distribution.version in each FluxInstance, open a PR to experimental, validate, then promote to dev and master.

The pipeline templates are split into focused reusable files:

Every tool installation in common-tools.yaml downloads from the official release URL and verifies the SHA256 checksum before installing. If the checksum does not match, the step fails immediately. A small but important supply chain control.

Here is how the three core image pipeline templates look in practice.

ci-cd-templates/flux-images.yaml — iterates over the image list and calls scan → copy → sign for each one:

parameters:
  - name: containerImages
    type: object
    default: []
  - name: containerRegistryName
    default: ""
  - name: containerRegistryServiceConnection
    default: ""
  - name: scanMode
    default: "audit" # switch to 'enforce' to fail on HIGH/CRITICAL
  - name: scanImages
    type: boolean
    default: false
  - name: signImages
    type: boolean
    default: false
  - name: environment
    default: ""
  - name: imageSignerServiceConnection
    default: ""

steps:
  - task: Docker@2
    displayName: "Login to Container Registry"
    inputs:
      command: login
      containerRegistry: $

  - $:
      - bash: |
          IMAGE_PATH=$(echo "$" | cut -d'/' -f2-)
          TARGET_IMAGE="$/$IMAGE_PATH"
          echo "##vso[task.setvariable variable=TARGET_IMAGE;]$TARGET_IMAGE"
        displayName: "Prepare target image name $"

      - $:
          - template: scan-image.yaml
            parameters:
              image: $
              scanMode: $

      - bash: |
          crane copy $ $(TARGET_IMAGE)
        displayName: "Copy $"
        condition: and(succeeded(), not(startsWith(variables['Build.SourceBranch'], 'refs/pull')))

      - $:
          - template: sign-image.yaml
            parameters:
              image: $(TARGET_IMAGE)
              serviceConnection: $
              environment: $

  - task: Docker@2
    displayName: "Logout of Container Registry"
    condition: always()
    inputs:
      command: logout
      containerRegistry: $

ci-cd-templates/scan-image.yaml — runs Trivy with two modes: audit (log only) and enforce (fail on HIGH/CRITICAL):

parameters:
  - name: image
    type: string
    default: ""
  - name: containerRegistryName
    default: ""
  - name: scanMode
    type: string
    default: "audit"

steps:
  - bash: |
      trivy image \
        --scanners vuln \
        --ignore-unfixed \
        --pkg-types os \
        --exit-code 0 \
        --db-repository=$/trivy/trivy-db:2 \
        --java-db-repository=$/trivy/trivy-java-db:1 \
        $
    condition: eq('$', 'audit')
    displayName: "Scan $ (audit)"

  - bash: |
      # LOW/MEDIUM — informational only
      trivy image --exit-code 0 --severity LOW,MEDIUM \
        --ignore-unfixed --pkg-types os \
        --db-repository=$/trivy/trivy-db:2 \
        $

      # HIGH/CRITICAL — fail the pipeline
      trivy image --exit-code 1 --severity HIGH,CRITICAL \
        --ignore-unfixed --pkg-types os \
        --db-repository=$/trivy/trivy-db:2 \
        $
    condition: eq('$', 'enforce')
    displayName: "Scan $ (enforce)"

ci-cd-templates/sign-image.yaml — resolves the digest via az acr, signs with cosign using an Azure Key Vault key, then immediately verifies the signature:

parameters:
  - name: image
    type: string
    default: ""
  - name: serviceConnection
    type: string
    default: ""
  - name: environment
    type: string
    default: ""

steps:
  - task: AzureCLI@2
    displayName: "Sign $"
    inputs:
      azureSubscription: "$"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        REGISTRY=$(echo "$" | cut -d'/' -f1)
        REPOSITORY=$(echo "$" | cut -d'/' -f2- | cut -d':' -f1)
        TAG=$(echo "$" | grep -o ':[^@]*$' | cut -d':' -f2)

        IMAGE_DIGEST=$(az acr repository show \
          --name $REGISTRY \
          --image $REPOSITORY:$TAG \
          --query "digest" -o tsv)

        if [ "$" == "prod" ]; then
          KV_NAME="my-prod-keyvault"
          KV_KEY="my-prod-cosign-key"
        else
          KV_NAME="my-dev-keyvault"
          KV_KEY="my-dev-cosign-key"
        fi

        cosign sign \
          --key azurekms://$KV_NAME.vault.azure.net/$KV_KEY \
          $REGISTRY/$REPOSITORY@$IMAGE_DIGEST \
          --upload=true --yes=true --tlog-upload=false

  - task: AzureCLI@2
    displayName: "Verify $ signature"
    inputs:
      azureSubscription: "$"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        if [ "$" == "prod" ]; then
          SECRET_NAME="cosign-public-key-prod"
          KV_NAME="my-prod-keyvault"
        else
          SECRET_NAME="cosign-public-key-dev"
          KV_NAME="my-dev-keyvault"
        fi

        az keyvault secret show \
          --vault-name $KV_NAME --name $SECRET_NAME \
          --query value -o tsv > cosign.pub

        cosign verify \
          --key cosign.pub \
          $ \
          --private-infrastructure=true

ci-cd-templates/flux.yaml — top-level orchestrator: routes to validate on PRs and to apply on push:

parameters:
  - name: branch
    default: ""
  - name: environment
    default: ""
  - name: serviceConnection
    default: ""
  - name: containerImages
    type: object
    default: []
  - name: scanMode
    default: "audit"
    values: [audit, enforce]
  - name: helmReleaseName
    default: ""
  - name: helmReleaseNamespace
    default: ""
  - name: helmChartPath
    default: ""
  - name: helmValuesFilePath
    default: ""
  - name: helmArgs
    type: object
    default: []
  - name: helmCreateNamespace
    type: boolean
    default: true

stages:
  - stage: "ValidateFluxConfiguration_in_$_cluster"
    displayName: Validate Flux configuration in $ cluster
    condition: startsWith(variables['Build.SourceBranch'], 'refs/pull')
    jobs:
      - template: flux-validate.yaml
        parameters:
          environment: $
          serviceConnection: $
          containerImages: $
          scanImages: true
          scanMode: $
          signImages: false
          helmReleaseName: $
          helmReleaseNamespace: $
          helmChartPath: $
          helmValuesFilePath: $
          helmArgs: $
          helmCreateNamespace: $

  - stage: "ApplyFluxConfiguration_to_$_cluster"
    displayName: Apply Flux configuration to $ cluster
    condition: eq(variables['Build.SourceBranchName'], '$')
    jobs:
      - template: flux-apply.yaml
        parameters:
          environment: $
          serviceConnection: $
          containerImages: $
          scanImages: true
          scanMode: $
          signImages: true
          helmReleaseName: $
          helmReleaseNamespace: $
          helmChartPath: $
          helmValuesFilePath: $
          helmArgs: $
          helmCreateNamespace: $

ci-cd-templates/flux-validate.yaml — PR validation job: scan images, Helm dry-run, FluxInstance dry-run. Note the per-environment ACR and service connection variables resolved at job level:

jobs:
  - job: ValidateFluxConfiguration
    displayName: Validate Flux configuration in $ cluster
    variables:
      - name: containerRegistryName
        $:
          value: myacr-prod.azurecr.io
        $:
          value: myacr-dev.azurecr.io
      - name: containerRegistryServiceConnection
        $:
          value: acr-prod-sc
        $:
          value: acr-dev-sc
      - name: imageSignerServiceConnection
        $:
          value: image-signing-prod-sc
        $:
          value: image-signing-dev-sc
    steps:
      - template: common-tools.yaml

      - template: flux-images.yaml
        parameters:
          environment: $
          containerRegistryName: $
          containerRegistryServiceConnection: $
          imageSignerServiceConnection: $
          containerImages: $
          scanImages: $
          signImages: $ # false on PR
          scanMode: $

      - template: flux-operator.yaml
        parameters:
          environment: $
          serviceConnection: $
          helmReleaseName: $
          helmReleaseNamespace: $
          helmChartPath: $
          helmValuesFilePath: $
          helmDryRun: true # dry-run only on PR

      - template: flux-instance.yaml
        parameters:
          environment: $
          serviceConnection: $
          waitTimeout: "3m"

ci-cd-templates/flux-apply.yaml — push apply job: same structure as validate but signImages: true and helmDryRun: false:

jobs:
  - job: ApplyFluxConfiguration
    displayName: Apply Flux configuration to $ cluster
    variables:
      - name: containerRegistryName
        $:
          value: myacr-prod.azurecr.io
        $:
          value: myacr-dev.azurecr.io
      - name: containerRegistryServiceConnection
        $:
          value: acr-prod-sc
        $:
          value: acr-dev-sc
      - name: imageSignerServiceConnection
        $:
          value: image-signing-prod-sc
        $:
          value: image-signing-dev-sc
    steps:
      - template: common-tools.yaml

      - template: flux-images.yaml
        parameters:
          environment: $
          containerRegistryName: $
          containerRegistryServiceConnection: $
          imageSignerServiceConnection: $
          containerImages: $
          scanImages: $
          signImages: $ # true on push
          scanMode: $

      - template: flux-operator.yaml
        parameters:
          environment: $
          serviceConnection: $
          helmReleaseName: $
          helmReleaseNamespace: $
          helmChartPath: $
          helmValuesFilePath: $
          helmDryRun: false # real install on push

      - template: flux-instance.yaml
        parameters:
          environment: $
          serviceConnection: $
          waitTimeout: "3m"

ci-cd-templates/flux-operator.yaml — Helm template, dry-run, and upgrade of the Flux Operator; credentials are fetched and cleaned up around the Helm steps:

parameters:
  - name: environment
    default: ""
  - name: serviceConnection
    default: ""
  - name: helmReleaseName
    default: ""
  - name: helmReleaseNamespace
    default: ""
  - name: helmChartPath
    default: ""
  - name: helmValuesFilePath
    default: ""
  - name: helmDryRun
    type: boolean
    default: true
  - name: helmCreateNamespace
    type: boolean
    default: true

steps:
  - task: AzureCLI@2
    displayName: "Get AKS credentials"
    inputs:
      azureSubscription: "$"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        kubelogin remove-cache-dir
        az aks get-credentials \
          --name aks-$ \
          --resource-group aks-rg-$ \
          --overwrite-existing
        kubelogin convert-kubeconfig -l azurecli

  - task: AzureCLI@2
    displayName: "Helm Template $"
    inputs:
      azureSubscription: "$"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        helm template $ \
          ./$ \
          -f ./$

  - task: AzureCLI@2
    displayName: "Helm Dry-Run $"
    condition: eq('$', 'true')
    inputs:
      azureSubscription: "$"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        helm upgrade $ \
          ./$ \
          -f ./$ \
          --namespace $ \
          --create-namespace --dry-run --install

  - task: AzureCLI@2
    displayName: "Helm Install $"
    condition: and(succeeded(), not(startsWith(variables['Build.SourceBranch'], 'refs/pull')))
    inputs:
      azureSubscription: "$"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        helm upgrade $ \
          ./$ \
          -f ./$ \
          --namespace $ \
          --create-namespace --install --wait

  - task: AzureCLI@2
    displayName: "Cleanup AKS credentials"
    condition: always()
    inputs:
      azureSubscription: "$"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        kubelogin remove-cache-dir
        kubectl config delete-context aks-$ 2>/dev/null || true
        kubectl config delete-cluster aks-$ 2>/dev/null || true
        echo "Credentials cleaned up ✓"

ci-cd-templates/flux-instance.yaml — validates the FluxInstance spec using the flux-operator CLI, applies it, and waits for the Ready condition. Skips the apply step on PRs:

parameters:
  - name: environment
    default: ""
  - name: serviceConnection
    default: ""
  - name: waitTimeout
    default: "3m"

steps:
  - task: AzureCLI@2
    displayName: "Get AKS credentials"
    inputs:
      azureSubscription: "$"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        kubelogin remove-cache-dir
        az aks get-credentials \
          --name aks-$ \
          --resource-group aks-rg-$ \
          --overwrite-existing
        kubelogin convert-kubeconfig -l azurecli

  - task: AzureCLI@2
    displayName: "Install flux-operator CLI"
    inputs:
      azureSubscription: "$"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        VERSION=$(yq '.image.tag' ./clusters/$/flux-operator-values.yaml)
        VERSION_NO_V="${VERSION#v}"
        curl -s -O -L "https://github.com/controlplaneio-fluxcd/flux-operator/releases/download/${VERSION}/flux-operator_${VERSION_NO_V}_linux_amd64.tar.gz"
        curl -s -O -L "https://github.com/controlplaneio-fluxcd/flux-operator/releases/download/${VERSION}/flux-operator_${VERSION_NO_V}_checksums.txt"
        grep "flux-operator_${VERSION_NO_V}_linux_amd64.tar.gz" flux-operator_${VERSION_NO_V}_checksums.txt | sha256sum --check
        tar -xzf flux-operator_${VERSION_NO_V}_linux_amd64.tar.gz -C /tmp flux-operator
        sudo mv /tmp/flux-operator /usr/local/bin/flux-operator
        flux-operator version --client

  - task: AzureCLI@2
    displayName: "Validate FluxInstance on $"
    inputs:
      azureSubscription: "$"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        INSTANCE_FILE="./clusters/$/flux-instance.yaml"
        flux-operator build instance -f $INSTANCE_FILE
        kubectl apply -f $INSTANCE_FILE --dry-run=server

  - task: AzureCLI@2
    displayName: "Apply FluxInstance on $"
    condition: and(succeeded(), not(startsWith(variables['Build.SourceBranch'], 'refs/pull')))
    inputs:
      azureSubscription: "$"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        INSTANCE_FILE="./clusters/$/flux-instance.yaml"
        APPLY_OUTPUT=$(kubectl apply -f $INSTANCE_FILE)
        echo "$APPLY_OUTPUT"

        if echo "$APPLY_OUTPUT" | grep -q "unchanged"; then
          READY=$(kubectl get fluxinstance flux -n flux-system \
            -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
          if [ "$READY" != "True" ]; then
            kubectl wait fluxinstance/flux --for=condition=Ready \
              --namespace flux-system --timeout=$
          else
            echo "FluxInstance already Ready ✓"
          fi
        else
          kubectl wait fluxinstance/flux --for=condition=Ready \
            --namespace flux-system --timeout=$
        fi

        kubectl -n flux-system get fluxinstance flux
        kubectl -n flux-system get pods

  - task: AzureCLI@2
    displayName: "Cleanup AKS credentials"
    condition: always()
    inputs:
      azureSubscription: "$"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        kubelogin remove-cache-dir
        kubectl config delete-context aks-$ 2>/dev/null || true
        kubectl config delete-cluster aks-$ 2>/dev/null || true
        echo "Credentials cleaned up ✓"

ci-cd-templates/common-tools.yaml — installs every tool the pipeline needs with SHA256 checksum verification on each download:

parameters:
  - name: kubectlVersion
    default: "v1.32.9"
  - name: kubeLoginVersion
    default: "v0.2.10"
  - name: yqVersion
    default: "v4.52.4"
  - name: trivyVersion
    default: "0.69.3"
  - name: cosignVersion
    default: "v2.6.1"
  - name: craneVersion
    default: "v0.20.3"
  - name: helmVersion
    default: "v3.18.4"

steps:
  - task: KubectlInstaller@0
    displayName: Install kubectl $
    inputs:
      kubectlVersion: $

  - bash: |
      set -euo pipefail
      curl -fsSL "https://github.com/Azure/kubelogin/releases/download/$/kubelogin-linux-amd64.zip" -o kubelogin-linux-amd64.zip
      curl -fsSL "https://github.com/Azure/kubelogin/releases/download/$/kubelogin-linux-amd64.zip.sha256" -o kubelogin-linux-amd64.zip.sha256
      sha256sum --check kubelogin-linux-amd64.zip.sha256
      unzip kubelogin-linux-amd64.zip && sudo mv bin/linux_amd64/kubelogin /usr/local/bin
      kubelogin --version
    displayName: Install kubelogin $

  - bash: |
      set -euo pipefail
      curl -fsSL "https://github.com/aquasecurity/trivy/releases/download/v$/trivy_$_Linux-64bit.tar.gz" -o trivy.tar.gz
      curl -fsSL "https://github.com/aquasecurity/trivy/releases/download/v$/trivy_$_checksums.txt" -o trivy_checksums.txt
      grep "trivy_$_Linux-64bit.tar.gz" trivy_checksums.txt | sha256sum --check
      tar -xzf trivy.tar.gz trivy && sudo mv trivy /usr/local/bin/trivy
      trivy -v
    displayName: Install Trivy $

  - bash: |
      set -euo pipefail
      curl -fsSL "https://github.com/mikefarah/yq/releases/download/$/yq_linux_amd64" -o yq_linux_amd64
      curl -fsSL "https://github.com/mikefarah/yq/releases/download/$/checksums-bsd" -o yq_checksums-bsd
      grep "^SHA256 (yq_linux_amd64)" yq_checksums-bsd | awk '{print $NF "  yq_linux_amd64"}' | sha256sum --check
      sudo mv yq_linux_amd64 /usr/local/bin/yq && sudo chmod +x /usr/local/bin/yq
      yq --version
    displayName: Install yq $

  - bash: |
      set -euo pipefail
      curl -fsSL "https://github.com/google/go-containerregistry/releases/download/$/go-containerregistry_Linux_x86_64.tar.gz" -o crane.tar.gz
      curl -fsSL "https://github.com/google/go-containerregistry/releases/download/$/checksums.txt" -o crane_checksums.txt
      grep "go-containerregistry_Linux_x86_64.tar.gz" crane_checksums.txt | sha256sum --check
      tar -xzf crane.tar.gz crane && sudo mv crane /usr/local/bin/crane
      crane version
    displayName: Install crane $

  - bash: |
      set -euo pipefail
      curl -fsSL "https://get.helm.sh/helm-$-linux-amd64.tar.gz" -o helm.tar.gz
      curl -fsSL "https://get.helm.sh/helm-$-linux-amd64.tar.gz.sha256sum" -o helm.tar.gz.sha256sum
      sha256sum --check helm.tar.gz.sha256sum
      tar -xzf helm.tar.gz linux-amd64/helm && sudo mv linux-amd64/helm /usr/local/bin/helm
      helm version
    displayName: Install Helm $

  - bash: |
      set -euo pipefail
      curl -fsSL "https://github.com/sigstore/cosign/releases/download/$/cosign-linux-amd64" -o cosign-linux-amd64
      curl -fsSL "https://github.com/sigstore/cosign/releases/download/$/cosign_checksums.txt" -o cosign_checksums.txt
      grep "cosign-linux-amd64$" cosign_checksums.txt | sha256sum --check
      sudo mv cosign-linux-amd64 /usr/local/bin/cosign && sudo chmod +x /usr/local/bin/cosign
      cosign version
    displayName: Install Cosign $

Systems Validation Pipeline

There are actually two pipelines. The main azure-pipelines.yaml handles Flux infrastructure (operator, FluxInstance, images). A separate azure-pipelines-systems.yaml validates system manifests on PRs.

When someone opens a PR that touches clusters/<env>/systems/, this pipeline runs kubectl apply --dry-run=server against the live cluster for the target environment. Server-side dry-run is important — it catches errors that client-side validation misses, like referencing a CRD that does not exist in the cluster or a resource that would conflict with an existing one.

Only the stage matching the PR target branch runs. A PR targeting dev validates clusters/dev/systems/ only — not experimental or prod.

# azure-pipelines-systems.yaml
name: k8s-gitops-systems-validate

trigger: none # no push trigger — PR only via branch policy

pool: my-linux-agents

stages:
  - stage: ValidateExperimental
    displayName: Validate systems manifests in experimental
    condition: eq(variables['System.PullRequest.TargetBranch'], 'refs/heads/experimental')
    jobs:
      - template: ci-cd-templates/systems-validate.yaml
        parameters:
          environment: "experimental"
          serviceConnection: "k8s-rbac-experimental-ado-sc"

  - stage: ValidateDev
    displayName: Validate systems manifests in dev
    condition: eq(variables['System.PullRequest.TargetBranch'], 'refs/heads/dev')
    jobs:
      - template: ci-cd-templates/systems-validate.yaml
        parameters:
          environment: "dev"
          serviceConnection: "k8s-rbac-dev-ado-sc"

  - stage: ValidateProd
    displayName: Validate systems manifests in prod
    condition: eq(variables['System.PullRequest.TargetBranch'], 'refs/heads/master')
    jobs:
      - template: ci-cd-templates/systems-validate.yaml
        parameters:
          environment: "prod"
          serviceConnection: "k8s-rbac-prod-ado-sc"

The trigger: none is intentional — this pipeline is attached as a branch policy in Azure DevOps, not triggered by a push. It only runs when a PR is opened or updated.

Flux does not trigger from changes to clusters/*/systems/. Those are reconciled directly by Flux on its own interval (every 1–5 minutes depending on the resource). The systems pipeline only validates on PRs — it has no push trigger.

ci-cd-templates/systems-validate.yaml — fetches cluster credentials, runs kubectl apply --dry-run=server against all files in clusters/<env>/systems/, then cleans up credentials:

parameters:
  - name: environment
    default: ""
  - name: serviceConnection
    default: ""

jobs:
  - job: ValidateSystemsManifests
    displayName: Validate systems manifests in $ cluster
    steps:
      - template: common-tools.yaml

      - task: AzureCLI@2
        displayName: "Get AKS credentials"
        inputs:
          azureSubscription: "$"
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            kubelogin remove-cache-dir
            az aks get-credentials \
              --name aks-$ \
              --resource-group aks-rg-$ \
              --overwrite-existing
            kubelogin convert-kubeconfig -l azurecli

      - task: AzureCLI@2
        displayName: "Dry-run systems manifests on $"
        inputs:
          azureSubscription: "$"
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            set -euo pipefail
            SYSTEMS_PATH="./clusters/$/systems"
            FILES=$(find "$SYSTEMS_PATH" -name "*.yaml" | sort)

            if [ -z "$FILES" ]; then
              echo "No system files found — nothing to validate"
              exit 0
            fi

            echo "Files to validate:"
            echo "$FILES" | while read -r f; do echo "  → $f"; done

            kubectl apply -f "$SYSTEMS_PATH" --dry-run=server
            echo "Validation passed ✓"

      - task: AzureCLI@2
        displayName: "Cleanup AKS credentials"
        condition: always()
        inputs:
          azureSubscription: "$"
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            kubelogin remove-cache-dir
            kubectl config delete-context aks-$ 2>/dev/null || true
            kubectl config delete-cluster aks-$ 2>/dev/null || true
            echo "Credentials cleaned up ✓"

Onboarding Systems — Three Patterns

The platform supports three onboarding patterns. The platform team’s registration step in k8s-gitops is identical for all three. The difference is entirely in what the system team places in their own repository.

Platform team registration — one file per system, per environment:

# clusters/<env>/systems/<system>.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: k8s-<system>
  namespace: flux-system
spec:
  interval: 1m
  url: https://dev.azure.com/my-org/my-project/_git/k8s-<system>
  ref:
    branch: <env>
  provider: azure
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: <system>
  namespace: flux-system
spec:
  interval: 5m
  path: ./environments/<env>/flux
  prune: true
  sourceRef:
    kind: GitRepository
    name: k8s-<system>

Flux watches clusters/<env>/systems/ in the k8s-gitops repo. When this file is merged, Flux picks it up on its next reconciliation interval, starts watching the system repo at the specified branch and path, and applies whatever it finds there.

Pattern A — System Owns Its Own Chart and HelmRelease

Used for platform components (nginx, cert-manager, kured, etc.) and application systems with custom charts. Each system repo co-locates its Helm chart and HelmRelease.

k8s-<system>/
├── <chart-dir>/                   # Helm chart co-located in the same repo
│   ├── Chart.yaml
│   ├── templates/
│   └── values.yaml                # base values
└── environments/
    ├── experimental/
    │   ├── values.yaml            # env-specific overrides
    │   └── flux/
    │       └── helmrelease.yaml   # Flux reconciles this path
    ├── dev/
    │   ├── values.yaml
    │   └── flux/
    │       └── helmrelease.yaml
    └── prod/
        ├── values.yaml
        └── flux/
            └── helmrelease.yaml

The HelmRelease in each environment references the chart by relative path in the same GitRepository:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: <system>
  namespace: flux-system
spec:
  interval: 10m
  targetNamespace: <system>
  install:
    createNamespace: true
  upgrade:
    cleanupOnFail: true
  chart:
    spec:
      chart: ./<chart-dir>
      reconcileStrategy: Revision
      sourceRef:
        kind: GitRepository
        name: k8s-<system>
        namespace: flux-system
      valuesFiles:
        - ./<chart-dir>/values.yaml
        - ./environments/<env>/values.yaml

The HelmRelease always lives in flux-system. The targetNamespace tells helm-controller where to actually deploy the chart’s resources. Flux auto-creates an internal HelmChart object in flux-system — you never manage that directly.

Pattern B — Central Shared Chart, System Owns Only Values

Pattern B is the standard pattern for business applications. The platform team maintains a single central Helm chart repo that encapsulates everything a well-behaved application deployment needs: namespace creation, RBAC, NetworkPolicy, ResourceQuota, LimitRange, and default ServiceAccount annotations. Application teams supply only a HelmRelease pointing at that central chart with their own values — they never write RBAC YAML, never define quotas, and never touch chart internals.

The platform team registers the central chart repo once in k8s-gitops as a shared GitRepository, then each business application gets its own GitRepository + Kustomization registration as usual. In the application repo the HelmRelease references the central chart source:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: <system>
  namespace: flux-system
spec:
  interval: 10m
  targetNamespace: <system>
  chart:
    spec:
      chart: ./platform-app-chart # chart from the central repo
      sourceRef:
        kind: GitRepository
        name: k8s-platform-charts # central chart GitRepository, registered once
        namespace: flux-system
  values:
    replicaCount: 2
    image:
      repository: myacr.azurecr.io/my-org/<system>
      tag: "1.0.0"
    aadGroupId: "<team-aad-group-id>" # drives RoleBinding generation inside the chart

The aadGroupId value is the key pattern — the central chart uses it to generate a RoleBinding for the team’s Azure AD group, scoped to their namespace. Teams get access to their own namespace without ever writing a RBAC manifest. Baseline security posture (NetworkPolicy, LimitRange, default quotas) is inherited from the chart automatically. It enforces baseline security posture across all application teams without manual steps or platform team involvement per namespace.

Platform SKUs — Right-Sized Resource Quotas

One extension of the central chart model is platform SKUs — a set of pre-defined resource profiles that teams choose from when onboarding. Instead of negotiating ResourceQuota and LimitRange values per team, the platform exposes a small menu of named tiers:

Each SKU is a separate Helm chart (or a named values preset inside the central chart). A team references their chosen SKU alongside their app HelmRelease:

values:
  replicaCount: 2
  image:
    repository: myacr.azurecr.io/my-org/<system>
    tag: "1.0.0"
  aadGroupId: "<team-aad-group-id>"
  sku: sku-medium # injects ResourceQuota + LimitRange for this tier

The platform team owns the SKU definitions centrally — if a tier needs tuning, one chart change propagates to every team using that SKU on the next reconciliation. Teams never touch quota YAML and can request a larger SKU via a PR comment rather than a platform ticket.

Pattern C — Plain YAML or Kustomize (Exception Only)

Pattern C is not a standard onboarding path — it is a deliberate escape hatch for the small category of resources that genuinely cannot fit inside a Helm chart: CRDs, ClusterRoles, cross-namespace objects, or anything that must exist before any chart can be installed. The Kustomization points at a path containing plain YAML files or a kustomization.yaml overlay, and kustomize-controller applies them directly.

If you find yourself reaching for Pattern C for a regular application workload, that is a signal something is missing from the central chart — fix the chart rather than bypassing it. Pattern C should be rare, reviewed carefully, and always additive. It cannot override what a Pattern B chart already manages, and any Pattern C registration requires explicit platform team approval before merging to k8s-gitops.

Choosing a Pattern

Business Application CI Pipeline

The k8s-gitops repo and the Flux infrastructure pipeline are entirely separate from how business applications build and deliver their images. Each application team owns their own Azure DevOps pipeline in their own repo — the platform only defines what must happen before an image is allowed into the cluster.

The contract is simple: every image that lands in production must be scanned, pushed to the private ACR, and signed. How the pipeline is structured internally is up to the team, but these three steps are non-negotiable:

1. Build or mirror, then scan with Trivy — application images are typically built from source in the team’s own pipeline. Third-party images the application depends on (databases, sidecars, off-the-shelf tools) are mirrored from public registries using crane copy. In both cases the image is scanned for HIGH and CRITICAL vulnerabilities before it is pushed to ACR. The pipeline fails if any are found.
2. Push to private ACR — all images go to the private registry. HelmRelease values and manifests always reference myacr.azurecr.io/..., never a public registry directly.
3. Sign with Cosign — the image is signed using the shared Azure Key Vault key after pushing. Admission controls in the cluster can verify the signature before allowing the workload to run.

This is where Pattern B closes the loop neatly. The team updates the image.tag value in their HelmRelease after a successful pipeline run, merges to the environment branch, and Flux picks up the change — no pipeline needs cluster credentials, no kubectl is run from CI. The pipeline’s only job is to produce a trusted, signed image in ACR. Flux handles the rest.

Switching from Push Pipelines to GitOps

Before GitOps the deployment model looked like most traditional CI/CD setups: pipelines authenticated to the cluster with a service principal and pushed changes in with kubectl apply. It worked, but over time the friction became hard to ignore:

• Credentials in the pipeline — every environment required a service connection with cluster-admin or broad RBAC permissions stored in the CI/CD platform. Rotation was manual, scope was wider than needed.
• No drift detection — if someone applied a change directly to the cluster, the pipeline had no idea. The next pipeline run would overwrite it, or not, depending on whether that path was touched. There was no continuous reconciliation.
• Deployment only on trigger — the cluster state was only as fresh as the last pipeline run. A failed pipeline meant nothing was deployed, with no retry or self-healing.
• No supply chain controls on images — images were mirrored or built in the pipeline but without vulnerability scanning or signing. There was no enforcement gate preventing an unscanned or unsigned image from reaching the cluster.
• Pipeline as the gatekeeper — every change had to flow through a pipeline, but the pipeline could not tell you whether the cluster had drifted from what it last applied.

The GitOps model inverts this. The cluster reaches out to Git, not the other way around. Credentials stay inside the cluster (and in this setup are federated via Workload Identity — no secrets at all). The operator reconciles continuously, so a drift is corrected within minutes without any human or pipeline intervention. Because everything is declared in Git and reviewed via pull requests, you get a full audit trail for free.

The shift is not just operational; it changes how teams think about deployments. Instead of “run the pipeline to deploy”, the mental model becomes “merge to the environment branch and Flux will pick it up”. The pipeline still exists, but its job is narrower: validate the infrastructure layer (Flux Operator, FluxInstance), mirror and sign images, and ensure the config is correct before it lands in Git. Day-to-day system changes bypass the pipeline entirely — Flux handles them on its own interval.

Upgrading Flux

Upgrading Flux in this setup is a four-step process:

1. Generate the updated image list using the helper script:

   ./scripts/flux-images.sh v2.9.0 v0.46.0 ./clusters/experimental/flux-operator-values.yaml
   

   The script uses the flux CLI to resolve controller images for the requested version and reads the operator image repository from the values file. Output looks like this:

   ======================================================================
   Add to azure-pipelines.yaml containerImages:
   ----------------------------------------------------------------------
     - myacr.azurecr.io/controlplaneio-fluxcd/flux-operator:v0.46.0
     - ghcr.io/fluxcd/helm-controller:v1.2.0
     - ghcr.io/fluxcd/image-automation-controller:v0.40.0
     - ghcr.io/fluxcd/image-reflector-controller:v0.34.0
     - ghcr.io/fluxcd/kustomize-controller:v1.5.0
     - ghcr.io/fluxcd/notification-controller:v1.5.0
     - ghcr.io/fluxcd/source-controller:v1.5.0
   ======================================================================
   

   Here is the full script (scripts/flux-images.sh):

   #!/bin/bash
   # Generates the containerImages list for azure-pipelines.yaml when upgrading
   # Flux controllers or the Flux Operator.
   #
   # Usage:
   #   ./flux-images.sh <FLUX_VERSION> <OPERATOR_VERSION> [VALUES_FILE]
   #
   # Prerequisites: flux CLI, yq
   set -euo pipefail
   
   FLUX_VERSION=${1:-""}
   OPERATOR_VERSION=${2:-""}
   DEFAULT_VALUES=${3:-"values.yaml"}
   
   [[ -z "$FLUX_VERSION" ]]     && { echo "ERROR: FLUX_VERSION required (e.g. v2.9.0)" >&2; exit 1; }
   [[ -z "$OPERATOR_VERSION" ]] && { echo "ERROR: OPERATOR_VERSION required (e.g. v0.46.0)" >&2; exit 1; }
   [[ ! -f "$DEFAULT_VALUES" ]] && { echo "ERROR: Values file '$DEFAULT_VALUES' not found." >&2; exit 1; }
   command -v flux &>/dev/null   || { echo "ERROR: flux CLI not installed." >&2; exit 1; }
   command -v yq   &>/dev/null   || { echo "ERROR: yq not installed." >&2; exit 1; }
   
   OPERATOR_REPO=$(yq '.image.repository' "$DEFAULT_VALUES")
   OPERATOR_IMAGE="$OPERATOR_REPO:$OPERATOR_VERSION"
   
   FLUX_IMAGES=$(flux install \
     --version="$FLUX_VERSION" \
     --components=source-controller,helm-controller,kustomize-controller,notification-controller \
     --components-extra=image-reflector-controller,image-automation-controller \
     --export \
     | grep 'image: ghcr.io/fluxcd/' \
     | awk '{print $2}' \
     | sort -u)
   
   echo "======================================================================"
   echo "Add to azure-pipelines.yaml containerImages:"
   echo "----------------------------------------------------------------------"
   echo "  - $OPERATOR_IMAGE"
   echo "$FLUX_IMAGES" | while IFS= read -r IMAGE; do echo "  - $IMAGE"; done
   echo "======================================================================"
   echo ""
   echo "Next steps:"
   echo "  1. Copy the image list into azure-pipelines.yaml"
   echo "  2. Update spec.distribution.version in clusters/*/flux-instance.yaml to $FLUX_VERSION"
   echo "  3. Update the operator image tag in clusters/*/flux-operator-values.yaml to $OPERATOR_VERSION"
   echo "  4. git commit + push + open PR to experimental first"
   

   Requires the flux CLI and yq to be installed locally.

2. Update azure-pipelines.yaml — replace the containerImages list for all three environments with the output above.

3. Update spec.distribution.version in each clusters/<env>/flux-instance.yaml.

4. Open a PR to experimental — validate, merge, then promote to dev and finally master.

The Flux Operator handles the rollout. You do not interact with the cluster directly. The pipeline validates the configuration before it touches anything, and the FluxInstance apply step waits for the Ready condition before the pipeline marks success.

Key Lessons

Use the Flux Operator. flux bootstrap was the right tool for getting started quickly, but the Flux Operator is the right tool for operating Flux long-term. Declarative upgrades, version-controlled configuration, and no generated YAML to maintain.

Mirror images to private ACR. Public registries have rate limits and availability dependencies. Mirroring with crane copy preserves digests, which means Cosign signatures remain valid. Trivy scanning before the copy gives you a vulnerability gate. All three together — mirror, scan, sign — is a practical and automatable supply chain control.

Server-side dry-run on PRs is worth it. Client-side validation misses schema errors for custom resources and referential integrity issues. Server-side dry-run catches these before anything reaches the cluster.

prune: true on Kustomizations. This ensures that when you remove a manifest from Git, the corresponding resource is deleted from the cluster. Without it, removed resources silently linger and you lose the “Git is the source of truth” property.

Workload Identity over credentials. Setting up federated credentials takes a bit more upfront effort than creating a PAT, but you eliminate a class of credential management problems entirely. No rotation, no expiry, no secrets stored anywhere.

Conclusion

Building a GitOps platform that teams actually want to use comes down to making the happy path easy and the unsafe path hard. In this setup the platform team maintains the Flux infrastructure layer and provides the system registration pattern. System teams own their repositories, their charts, their release cadence, and their values. They do not need platform team approval to deploy a new version of their application.

The pipeline handles the trust model: images are scanned before they enter the private registry, signed after they arrive, and verified before they are deployed. Pull requests are validated with server-side dry-run before merge. Flux handles reconciliation continuously, not just on pipeline triggers.

If you are looking to build a similar setup or are in the middle of migrating from a more manual approach, I hope this walkthrough is useful. Although the examples here use Azure DevOps and AKS, the approach is broadly portable — Flux supports GitHub, GitLab, and Gitea natively via the provider field in GitRepository, and the pipeline layer is standard YAML that maps directly to GitHub Actions or GitLab CI. The Flux side of the setup does not change at all. Feel free to reach out if you have questions — I’m happy to dig into any of the details.

Thank you for reading!

Hello everyone!

I’m incredibly excited to share that I have just achieved the Golden Kubestronaut status — the highest recognition in the CNCF certification landscape! This is a milestone I’ve been working towards for quite some time and I’m happy to share the full journey with you.

As you may remember from my previous post about Kubestronaut, I became a Kubestronaut in September 2024 after completing all five Kubernetes certifications (CKA, CKAD, CKS, KCNA, KCSA). This time around the journey was even more ambitious — requiring me to pass all remaining CNCF certifications plus the Linux Foundation Certified System Administrator (LFCS) exam on top of what I already had.

What is Golden Kubestronaut?

The Golden Kubestronaut program was announced by CNCF at KubeCon + CloudNativeCon Europe in London on April 1, 2025. It is the highest tier of recognition in the CNCF ecosystem, awarded to professionals who have successfully completed every CNCF certification plus the LFCS exam. What makes this program special compared to the regular Kubestronaut status is that once you earn it, the title is yours for life — you don’t need to recertify to keep it.

The program requires the following certifications on top of the five Kubestronaut ones (CKA, CKAD, CKS, KCNA, KCSA):

• LFCS - Linux Foundation Certified System Administrator
• PCA - Prometheus Certified Associate
• ICA - Istio Certified Associate
• CCA - Cilium Certified Associate
• CAPA - Certified Argo Project Associate
• CGOA - Certified GitOps Associate
• CBA - Certified Backstage Associate
• OTCA - OpenTelemetry Certified Associate
• KCA - Kyverno Certified Associate
• CNPA - Certified Cloud Native Platform Engineering Associate

In total that is 15 certifications to achieve Golden Kubestronaut status. Quite a list, but every single one of them is worth it!

KodeKloud — Your Best Friend for This Journey

Before diving into the individual exams I want to give a special shout-out to KodeKloud. If I had to recommend a single learning platform for the Golden Kubestronaut path, it would be KodeKloud without hesitation. They have dedicated courses for every CNCF certification covered here — complete with structured theory, hands-on labs, and mock exams. The quality is consistently high across all courses and the hands-on lab environments are particularly valuable because they let you practice in real clusters without having to set up your own infrastructure. Whether you are a complete beginner to a specific technology or just need to fill in a few gaps before an exam, KodeKloud will get you there. I used it extensively throughout my Golden Kubestronaut journey and can recommend it wholeheartedly.

The Additional Certifications

If you have already completed your Kubestronaut journey like I did, you just need to knock out the remaining certifications. Here are my thoughts on each of them.

LFCS — Linux Foundation Certified System Administrator

LFCS is a hands-on, performance-based exam testing your Linux system administration skills. You can expect tasks covering Linux file system management, user and group administration, service configuration, networking, and basic security hardening. The exam is 2 hours long and has a similar feel to the CKA exam in terms of format — a terminal-based environment where you need to perform real tasks.

Do not underestimate this exam. Even with years of Linux experience and having passed CKA and CKS, I found LFCS to be genuinely challenging. The breadth of topics is wide and some tasks can catch you off guard if you have gaps in areas you don’t use day-to-day. Make sure to thoroughly review systemd service management, networking configuration, storage and LVM management, user/group administration, simple apache or nginx configs as well as essential commands like find, mount etc. Dedicated preparation is strongly recommended regardless of your Linux background — the KodeKloud LFCS course is a great structured option here.

PCA — Prometheus Certified Associate

PCA is a multiple choice exam (60 questions, 90 minutes) covering the fundamentals of Prometheus — the de facto standard for monitoring in the cloud native world. The exam is split across five domains: observability concepts, Prometheus architecture and configuration, PromQL, alerting, and exporters.

If you use Prometheus in your daily work this exam will feel very familiar. The most important topic to master is PromQL — make sure you understand how to write queries, use functions like rate(), increase(), histogram_quantile(, and how to apply label selectors and aggregation operators correctly. Also pay attention to the observability concepts domain as it covers broader topics like SLOs, SLIs, and SLAs which you need to understand well. For learning resources I recommend the Prometheus official documentation and the KodeKloud PCA course.

ICA — Istio Certified Associate

ICA is a hands-on, performance-based exam similar in format to CKA and CKAD. You will be working in a real Kubernetes environment with Istio pre-installed, solving 15-20 practical tasks in 2 hours. The exam covers traffic management (35%), securing workloads (25%), troubleshooting (20%), and installation/upgrades/configuration (20%).

This is one of the most challenging of the additional certifications given its hands-on nature. Traffic management is the dominant topic so make sure you have solid hands-on experience with VirtualService, DestinationRule, and Gateway resources. For security, focus on mTLS and AuthorizationPolicy. You are allowed to use Istio documentation during the exam which is a big help. Make sure you are comfortable in navigating the documentation and you know how to quickly find the resources example you need. As with CKA/CKAD I highly recommend practicing in a real cluster before attempting this one — reading alone is not enough.

CCA — Cilium Certified Associate

CCA is a multiple choice exam focused on Cilium — the eBPF-based networking, observability, and security solution for Kubernetes. The exam covers Cilium architecture, network policies, Hubble observability, and cluster mesh.

Cilium has been gaining a lot of traction recently especially since it became a CNCF graduation project and was added as a topic to the updated CKS exam. If you have been keeping up with the Kubernetes ecosystem Cilium should not be a stranger to you. The official Cilium documentation is your main resource for preparation. Yes it questions/answers multiple choice exam, but I can assure that it was most difficult multiple choice exam I have taken in my entire IT career. The questions are very detailed and require a deep understanding of Cilium concepts and features to answer correctly. Make sure to read the documentation carefully and understand how Cilium works under the hood — this is not an exam you can pass with just surface-level knowledge.

CAPA — Certified Argo Project Associate

CAPA is a multiple choice exam covering the Argo project suite — Argo Workflows, Argo CD, Argo Rollouts, and Argo Events. It is not easy, but if you have experience with Argo CD and Argo Workflows it should be manageable without to much extra preparation. The exam covers Argo architecture, core concepts, and common use cases for each of the four projects. For Argo CD focus on understanding application manifests, sync policies, and how to troubleshoot common issues. For Argo Workflows make sure you understand how to define workflows, use templates, and manage workflow execution. The official Argo documentation is comprehensive and the KodeKloud CAPA course provides a great structured introduction to the entire Argo ecosystem.

CGOA — Certified GitOps Associate

CGOA is a beginner-level multiple choice exam focused on GitOps principles and practices. It is one of the easiest exams in the entire Golden Kubestronaut set — especially if you have done CAPA first.

The exam covers GitOps principles as defined by the OpenGitOps project, GitOps patterns, and common tooling. Understanding the four core GitOps principles and the difference between push-based and pull-based deployment models is the bulk of what you need. If you work with Argo CD or Flux regularly you may be able to pass this one with minimal extra preparation.

CBA — Certified Backstage Associate

CBA is a multiple choice exam covering Backstage — the open source developer portal framework originally created by Spotify and now a CNCF project. The exam covers Backstage architecture, the software catalog, TechDocs, software templates (scaffolding), and the plugin ecosystem.

Backstage was probably the technology I was least familiar with before starting the Golden Kubestronaut journey. But the concepts are not overly complex and with some reading and hands-on practice you can get up to speed relatively quickly. The official Backstage documentation is your best friend here and the KodeKloud CBA course provides a great structured introduction to the platform.

OTCA — OpenTelemetry Certified Associate

OTCA is a multiple choice exam covering OpenTelemetry — the CNCF observability framework for traces, metrics, and logs. The exam covers OTel architecture, instrumentation, the collector (receivers, processors, exporters), and how to work with all three telemetry signals. Exam may fill challenging due to the a lot of different terminology like spans, metrics types, context propagation, bagage, etc.

OpenTelemetry is increasingly becoming the standard for observability in cloud native environments so this is a very practical and relevant certification. Focus on understanding the three pillars of observability (traces, metrics, logs), the OTel collector architecture, and how to instrument applications with OTel SDKs. The official OpenTelemetry documentation is comprehensive and the KodeKloud OTCA course is a solid structured option.

KCA — Kyverno Certified Associate

KCA is a multiple choice exam focused on Kyverno — a policy engine designed specifically for Kubernetes. The exam covers Kyverno architecture, policy writing (validate, mutate, generate policies), admission webhooks, and policy exceptions.

Kyverno is a fantastic tool for enforcing security and compliance policies in Kubernetes clusters and I’ve been using it in production for a while now, so this one felt very natural to me. If you have experience writing Kyverno policies you will do well here. For those new to Kyverno the official Kyverno documentation is excellent.

CNPA — Certified Cloud Native Platform Engineering Associate

CNPA is a multiple choice easy exam covering platform engineering concepts and practices in a cloud native context. The exam focuses on Internal Developer Platforms (IDPs), the CNCF platform engineering landscape, and developer experience. Platform engineering has become one of the hottest topics in the industry so this is a timely and relevant certification. The CNCF Platform Engineering whitepaper is essential reading before taking this exam. Unlike all other multiple choice exams in the Golden Kubestronaut journey, CNPA has ~ 90 questions instead of 60, but the exam time is prolonged accordingly.

Tips for the Golden Kubestronaut Journey

Having gone through the entire journey from zero to Golden Kubestronaut here are some tips that might help you:

1. Don’t rush. The journey to Golden Kubestronaut is a marathon, not a sprint. Take your time to properly learn each technology rather than just cramming for the exam. The knowledge you gain along the way is far more valuable than the badge itself.

2. Leverage your existing knowledge. Many of the CNCF associate certifications build on each other. Your Kubernetes knowledge from CKA/CKAD/CKS will help significantly with ICA, CCA, KCA and others. Don’t underestimate how much you already know.

3. Use KodeKloud. As mentioned above, KodeKloud is the single best resource for this entire journey. Their courses cover virtually every certification you need, and the combination of structured theory and real hands-on labs is exactly the right formula for passing both multiple choice and performance-based exams.

4. Practice with real tools. Set up a local Kubernetes cluster (kind or minikube) and actually install and use each technology. Hands-on experience is invaluable even for multiple choice exams because it gives you the intuition to eliminate wrong answers confidently. For ICA this is absolutely mandatory — you cannot pass it by reading alone.

5. Watch for CNCF discounts. CNCF and Linux Foundation regularly offer discounts especially around Black Friday / Cyber Monday (up to 60% off). Buy exam bundles when discounts are available to save significant money — with 15 certifications to collect this adds up quickly.

New Exam Requirement

Since 2026 March 1st one more exam has been added to the Golden Kubestronaut requirements — the Certified Cloud Native Platform Engineer (CNPE) which is an advanced hands-on exam focused on platform engineering concepts and practices. Here you need to demonstrate your ability to use a variety of CNCF tools tools like Kubernetes, Argo CD, Backstage, Kyverno, OpenTelemetry, Crossplane, etc to build and operate a cloud native platform. I have not taken this one yet but as far as I heard it is quite interesting, but at the same time very challenging.

Conclusion

Achieving Golden Kubestronaut has been challenging and rewarding certification journey. It required learning and demonstrating proficiency across the broadest possible slice of the CNCF ecosystem — from core Kubernetes to service mesh, observability, security policies, GitOps, developer portals and platform engineering.

What I love most about this program is that it pushes you to explore technologies you might not encounter in your day-to-day work. Even if you never work with Backstage or Argo Events professionally, understanding these tools makes you a much more well-rounded cloud native engineer and opens your eyes to solutions for problems you didn’t know could be solved this elegantly.

If you are currently on your Kubestronaut journey and wondering whether to go for Golden Kubestronaut as well — I say absolutely go for it! It is a lot of work, but the CNCF ecosystem is rich and fascinating and there is something genuinely satisfying about seeing how all the pieces fit together into a coherent cloud native platform story.

And remember: once a Golden Kubestronaut, always a Golden Kubestronaut!

If you have any questions about the journey or any of the individual certifications feel free to reach out. I will be happy to help!

Thank you for reading and let’s keep learning!!!

Those of you who are using Trivy in your CI/CD pipelines to scan container images for vulnerabilities, you might have noticed that already for some time trivy image command may fail with the following error:

2024-10-29T09:07:18.631+0200	FATAL	init error: DB error: failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred:
	* GET https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2: TOOMANYREQUESTS: retry-after: 825.911µs, allowed: 44000/minute

This issue is caused by the rate limiting of the GitHub Container Registry (GHCR) and for the last few months, it started to appear very frequently. For more details you can check the GitHub Issue. My solution for this problem is to host the Trivy DB in a private container registry in my case in Azure Container Registry (ACR) which actually an OCI registry. The DB can be updated on a schedule or on-demand using a Azure DevOps pipelines or just a simple bash script and cron jobs. If you wonder how we can keep the DB in container registry the answer is simple using the OCI Artifacts. Actually OCI registry is a storage for OCI artifacts and this can be anything like images, Helm charts or even videos. So here is the azure-pipelines.yml file which I use to update the Trivy DB in ACR:

pr: none
trigger: none

schedules:
- cron: '0 2 * * *' # Run every day at 2:00 UTC
  displayName: Daily Trivy DB Update and Upload for Dev Azure Container Registry

parameters:      
- name: orasVersion
  default: '1.2.0'
- name: jqVersion
  default: '1.7.1'
- name: containerRegistryServiceConnection
  default: 'Azure DevOps Service Connection Name for ACR'
- name: containerRegistryName
  default: 'youracr.azurecr.io'

stages:
- stage: 'Update_Trivy_DB'
  displayName: Update Trivy DB
  jobs:
    - job: DownloadAndUpdateTrivyDB
      displayName: Download and Update for $ environment
      parameters:
      - name: containerRegistryName
        value: 'youracr.azurecr.io'
      steps:
        - task: CmdLine@2
          displayName: 'Download and Install ORAS'
          inputs:
            script: |
              curl -sLO "https://github.com/oras-project/oras/releases/download/v$/oras_$_linux_amd64.tar.gz"
              mkdir -p oras-install/
              tar -zxf oras_$_*.tar.gz -C oras-install/
              sudo mv oras-install/oras /usr/local/bin/
              rm -rf oras_$_*.tar.gz oras-install/
              oras version
        - task: CmdLine@2
          displayName: 'Download and Install jq'
          inputs:
            script: |
              curl -sL -o jq "https://github.com/jqlang/jq/releases/download/jq-$/jq-linux-amd64"
              chmod +x jq
              sudo mv jq /usr/local/bin/
              jq --version    
        - task: Docker@2
          displayName: Login to Container Registry
          inputs:
            command: login
            containerRegistry: $
        - task: Bash@3
          displayName: Download and Update Trivy databases
          inputs:
            workingDirectory: '$(System.DefaultWorkingDirectory)'
            targetType: 'inline'
            script: |  
              # Function to retry a command if it fails
              retry() {
                local n=1
                local max=5
                local delay=5
                while true; do
                  "$@" && break || {
                    if [[ $n -lt $max ]]; then
                      ((n++))
                      echo "Command failed. Attempt $n/$max:"
                      sleep $delay;
                    else
                      echo "The command has failed after $n attempts."
                      return 1
                    fi
                  }
                done
              }
              
              # Pull the trivy-db image from ghcr.io with retry logic
              retry oras pull ghcr.io/aquasecurity/trivy-db:2

              # Push the trivy-db image to registry
              oras push --export-manifest manifest.json $/trivy/trivy-db:2 db.tar.gz
              
              # Update the mediaType of the manifest 
              jq '.layers[0].mediaType="application/vnd.aquasec.trivy.db.layer.v1.tar+gzip"' manifest.json > trivy_db_manifest.json
              
              # Push the updated manifest to registry
              oras manifest push $/trivy/trivy-db:2 trivy_db_manifest.json

              # Clean up
              rm -f db.tar.gz manifest.json trivy_db_manifest.json

              # Pull the trivy-java-db image from ghcr.io with retry logic
              retry oras pull ghcr.io/aquasecurity/trivy-java-db:1

              # Push the trivy-java-db image to registry and export the manifest
              oras push --export-manifest manifest.json $/trivy/trivy-java-db:1 javadb.tar.gz
              
              # Update the mediaType of the manifest
              jq '.layers[0].mediaType="application/vnd.aquasec.trivy.db.layer.v1.tar+gzip"' manifest.json > trivy_javadb_manifest.json
              
              # Push the updated manifest to registry
              oras manifest push $/trivy/trivy-java-db:1 trivy_javadb_manifest.json

              # Clean up
              rm -f javadb.tar.gz manifest.json trivy_javadb_manifest.json

        - task: Bash@3
          displayName: Logout of Container Registry
          inputs:
            targetType: 'inline'
            script: |
              docker logout $

In simple words pipeline does the following:

1. Downloads the ORAS and jq binaries which are needed for the script
2. Logs in to the Azure Container Registry
3. Pulls the Trivy DB images from GHCR
4. Pushes the images to ACR and exports the manifest
5. Updates the mediaType of the manifest
6. Pushes the updated manifest to ACR
7. Repeats the same procedure twice once for the regular Trivy DB and once for the Java Trivy DB
8. Logs out of the Azure Container Registry

If you will use the above pipeline make sure to replace the parameters with your values. The pipeline is scheduled to run every day at 2:00 UTC. Make sure to use correct Azure DevOps Service Connection of type Docker Registry for the ACR also associated Azure entraID service principal should have the correct PUSH permissions on the ACR.

If you need a more simple bash script version of the above pipeline you can use the following script:

#!/bin/bash

# Make sure oras and jq are installed on the client

# Login to Azure Container Registry
az acr login --name youracr.azurecr.io

# Pull the trivy-db image from ghcr.io
oras pull mirror.gcr.io/aquasecurity/trivy-db:2
# Push the trivy-db image to registry
oras push --export-manifest manifest.json youracr.azurecr.io/trivy/trivy-db:2 db.tar.gz
# Update the mediaType of the manifest 
jq '.layers[0].mediaType="application/vnd.aquasec.trivy.db.layer.v1.tar+gzip"' manifest.json > trivy_db_manifest.json
# Push the updated manifest to registry
oras manifest push youracr.azurecr.io/trivy/trivy-db:2 trivy_db_manifest.json
# Clean up
rm -f db.tar.gz manifest.json trivy_db_manifest.json

# Pull the trivy-java-db image from ghcr.io
oras pull ghcr.io/aquasecurity/trivy-java-db:1
# Push the trivy-java-db image to registry and export the manifest
oras push --export-manifest manifest.json youracr.azurecr.io/trivy/trivy-java-db:1 db.tar.gz
# Update the mediaType of the manifest
jq '.layers[0].mediaType="application/vnd.aquasec.trivy.db.layer.v1.tar+gzip"' manifest.json > trivy_java_db_manifest.json
# Push the updated manifest to registry
oras manifest push youracr.azurecr.io/trivy/trivy-java-db:1 trivy_java_db_manifest.json
# Clean up
rm -f db.tar.gz manifest.json trivy_java_db_manifest.json

Once you have the Trivy DB in your ACR you can use your regular trivy image command but with additional --db-repository=youracr.azurecr.io/trivy/trivy-db:2 and --java-db-repository=youracr.azurecr.io/trivy/trivy-java-db:1

Important to note that in one of the latest PR’s this issue is fixed and mirror.gcr.io is now used as a default DB repository, but your will still benefit from hosting the DB in your private registry as you will have more control over the DB updates and you will not be affected by any potential rate limiting. DB download process should be faster as less concurrent downloads will be made compared to the public registry. And of course keeping and using data in/from private registry is more secure.

Also worth mentioning that the above pipeline can be easily converted to GitHub Actions or any other CI/CD tool you are using. Same approach can be used to host the Trivy DB in any other OCI registry like Docker Hub, Quay, AWS ECR etc.

I hope this post was useful for you and you will find it helpful. If you have any questions or suggestions feel free to contact me and I will be happy to discuss it with you.

Hello everyone!

Few days ago I have finished my Kubernetes certification journey and after renewing my CKA, CKAD, CKS certification and additionally passing KCNA and KCSA exams I can officially and proudly call myself a Kubestronaut.

What is a Kubestronaut? Kubestronaut is relatively new program announced by CNCF at Kubecon 2024 in Paris. It is a program for Kubernetes professionals who have passed all Kubernetes certifications and have proven their knowledge and skills in Kubernetes ecosystem. Every one who has passed all 5 Kubernetes certifications (CKA, CKAD, CKS, KCNA, KCSA) will be recognized as a Kubestronaut. Worth to mention that Kubestronaut is not only a title, but in addition to that every Kubestronaut will get a special badge and an exclusive Kubestronaut jacket. Also Kubestronauts will get discount for all CNCF events and certifications.

If you ask is it hard to become a Kubestronaut I would say that it is not easy, but it is definitely worth it not only because of the title, but also because of the knowledge and experience you will gain during the learning process and this is probably the most important thing. I hold some k8s certifications like CKA, CKAD, CKS from 2020/21 and I worked with k8s even before that, but still I can say that I have learned some new things and improved my skills during the preparation for re-certification and new exams. Kubernetes is a fast evolving technology and it is important to keep up with the latest updates and new features.

Now let’s talk about exams a little bit.

CKA or Certified Kubernetes Administrator - this one is the most popular Kubernetes certification and it’s my favorite one. In my opinion it is a must have certification for every Kubernetes administrator/cluster operator. It is a hands-on exam where you need to solve multiple tasks in a real Kubernetes clusters. You maybe asked to create and configure different types of resources, troubleshoot issues with cluster, perform kubernetes version updates in other words all you can expect in a real world scenario. Also some task may require a knowledge of linux administration i.e. checking logs, editing files enabling/disabling services, viewing processes, networking etc. The exam is 2 hours long and you may need to solve 15-20 tasks. Keep in mind that each task may have some so called sub-tasks so you need to act fast. From my experience I can say that this exam is not easy, but not hardest one. If you have experience with Kubernetes and have linux administration skills with few weeks of honest preparation you can pass it. Make sure to take a practice exercises and mock exams before the real exam they really well shows how good you are prepared. Then you register for the exam you will get a free retake as well as a 2 free practice exams for killer.sh platform. For training I can recommend Kodekloud’s CKA course + Mock exam labs. To be honest for re-certification I have used only killer.sh exam simulator and it was enough for me to pass the exam with 90% score.

killer.sh provides a real exam experience and it’s tasks in many cases are even harder than the real exam tasks. They have practice exams for CKA, CKAD and CKS certifications and with which exam you will register you will get 2 free practice exams for that certification. Each practice exam is valid for 36 hours and you also have possibility to reset the lab if you want to start from scratch. I highly recommend killer.sh platform for training and preparation for CKA, CKAD and CKS certifications.

CKAD or Certified Kubernetes Application Developer - this exam is more oriented for developers who deploy, maintain and troubleshoot applications in Kubernetes. Here you can expect tasks like creating and configuring different types of resources, running and configuring application, login, debugging app issues etc. Main difference from CKA is that you can’t expect tasks related to cluster administration like troubleshooting, upgrading clusters or configuring kubelet. Same as CKA this exam is hands-on and you need to solve multiple tasks in a real Kubernetes clusters. The exam is also 2 hours long and you may need to solve 15-20 tasks. From my experience I can say that this exam is easier than CKA in terms of tasks complexity and difficulty, but it is may require more typing (i.e. creating and configuring yaml manifest for k8s resources) and due to that you may need to act fast what could consequently lead to more mistakes. My general advice would be to do not start your Kubernetes certification journey with CKAD, but start with CKA and then go for CKAD in this case with your CKA knowledge you should be able to pass CKAD without any special preparation (just additionally take a look on things like cronjobs, jobs, init containers, liveness and readiness probes etc.). Same as for CKA for re-certification I have used only killer.sh platform and it was enough for me to pass the exam with 89% score. As I mentioned any CKA training course with small addons should be enough for CKAD as well, but you can also take a look on Kodekloud’s CKAD course + Mock exam labs.

CKS or Certified Kubernetes Security Specialist - this exam is around Kubernetes security and it’s covers various security aspects of Kubernetes clusters. Here you can expect tasks like securing cluster components i.e. api-server, etcd or kubelet, configuring audit logging, network policies, pod security standards, apparmor profiles, runtime classes etc. Searching for vulnerabilities, analyzing runtime security or performing static analysis of Dockerfiles or Kubernetes manifests and applying best practices also could be in exam. The exam is 2 hours long and you may need to solve 15-20 tasks. From my experience I can say that this exam is the hardest one from all Kubernetes certifications. It requires a deep knowledge of Kubernetes security and a lot of practice. I have used killer.sh platform for training and was able to solve practice exams with 100% score in less than 2 hours, but to be honest the real exam was more challenging for me and I failed it with 60% (required to pass is 67%)on my first attempt during re-certification (Initially in 2021 I passed exam on my first attempt). Worth to mention that during preparation for CKS re-certification I not only used killer.sh platform, but also watched complete CKS course from killer.sh on Youtube. I can say that I failed the exam not because of lack of knowledge, but because of lack of time. Some of the tasks are quite complex and may require some time to solve them and double check what you did it right and related components are working as expected after your changes. Once I failed the exam I have used my free retake and scheduled retry right after 3 days and this time I was able to pass it with 69% score. I can’t say what I’m satisfied with this score as I’m confident that I know the topic well and can perform better. I’m just need bit more time to solve and double check all tasks. So my advice for CKS would be to have as much practice as possible and to be fast in solving tasks and if you are really don’t know how to solve some task just mark it for later review and go to the next one. I also think that more complex tasks have more weight in terms of score so make sure to not skip and solve them at least partially as I think that even partially solved task will give you some points.

CKA, CKAD and CKS exams are proctored exams and you need to take them online with a proctor watching you via webcam. You need to have a good internet connection, a quiet private room and a computer with a webcam. You can’t use any additional monitors, mobile phones, books or notes during the exam. During these exams you are allowed to use kubernetes.io/docs + some other additional resources depending on the exam. Personally I never had any technical issues with the exam environment or proctoring, just make sure that your PC and environment meets all the requirements. I also recommend to use at least 27 inch monitor as it will be much more comfortable compared to 14-15 inch laptop screen. You exam score will be available after 24 hours after the exam. And if you will not pass you will get a top 3 domains where you need to improve your knowledge.

My recommended learning resources for CKA, CKAD and CKS certifications:

• killer.sh - practice exams for CKA, CKAD and CKS certifications. Highly recommended!
• Kodekloud - CKA,CKAD course and Mock exams. Highly recommended!
• CKS course from killer.sh on Youtube - Is a must watch for CKS exam.
• Killercoda - Practice tasks for CKA exam.
• CKA Mock exams
• CKAD Mock exams
• CKS Mock exams

Also keep in mind that Kubernetes is a fast evolving technology and exam domains may change from time to time. Make sure to check the official exam page for the latest updates and exam domains. For example soon there will be a new version of CKA and CKS exams with new domains and tasks. In CKA exam we will have some tasks related to Helm and in CKS exam we will have Cilium related tasks.

KCNA or Kubernetes and Cloud Native Associate - this exam is an entry level certification for Kubernetes and Cloud Native technologies. It is a multiple choice exam and you need to answer 60 questions in 90 minutes. The exam covers various topics from Kubernetes basics + plus some additional questions on other CNCF projects like Prometheus. From my experience I can say that this exam is not hard and if you already have passed exams like CKA or CKAD you should be able to pass it without any special preparation. If you want to be super confident you can take a look on KCNA course from kodekloud and also take a Mock exam provided by same course. Personally I didn’t take any special preparation and only tried to take the Mock exam which I passed with no issues during first attempt. I passed the real exam with 89% score and it took me less than 30 minutes to answer all questions.

KCSA or Kubernetes and Cloud Native Security Associate - this exam is also considered as an entry level certification for Kubernetes and Cloud Native technologies, but it is focused on security aspects. It is a multiple choice exam and you need to answer 60 questions in 90 minutes. The exam covers various topics from Kubernetes security basics + plus some general cybersecurity questions for example be familiar with STRIDE model or 4C’s. Comparing to KCNA this exam is a bit harder and requires some readings, but in general you can pass it if you have CKS certification even without any special preparation. Unlike KCNA at current moment there is no official training courses for KCSA, but I can recommend to take a look on this blog post where you can find some useful notes for KCSA exam. I passed the real exam with 91% I was not expecting to get such a high score as I was not sure about some questions, but I’m was very happy to finish my Kubestronaut journey on such a good note.

During KCNA and KCSA exams you are not allowed to use any additional resources like it is allowed for CKA, CKAD and CKS exams. For KCNA and KCSA in my case score was emailed to me immediately after the exam.

CNCF exams are not cheap and it hurts especially if you need to take multiple exams and you are not sponsored by your employer. But good thing is that CNCF provides a free retake for all exams and with some exams (like CKA, CKAD, CKS) you also get a free practice tests on killer.sh platform. Very often there are some discounts available especially on days like cyber Monday there usually you can get a 60% discount for all exams. Also worth to mention that CNCF certification are valid for 2 years and after that you need to re-certify. Keep in mind that when you buy an exam you will have 1 year to take it including free retake.

In conclusion I would like to say that Kubernetes is a great technology and it is worth to learn it as it opens a lot of opportunities in IT career. Kubernetes is constantly changing so learning and passing Kubernetes certifications is a great way to stay up to date with. If you are interested in Kubernetes certification and Cloud Native technologies I would recommend to start with CKA and CKAD certifications and then go for CKS, KCNA and KCSA. It is not easy, but it is definitely worth it. And remember like in many other live aspect it not about the destination, but more about the journey so it’s not only about certifications, but also about the knowledge and experience you will gain during the learning process.

If you have any questions or need some help with Kubernetes certifications feel free to contact me. I will be happy to help you of course if it will not violate the CNCF NDA.

Thank you for reading and good luck and let’s do some Kubernetes!!!

Hello everyone!

It’s has been a while since my last post, but now I’m back on track and hopefully will be able to post more frequently. Today I want to share with you a tool that I have been working on for the past few weeks. It’s called Helm Release Viewer. Helm Release Viewer is a web-based UI that allows you to view the status of your Helm releases in a Kubernetes cluster. It provides a simple and intuitive interface that allows you to see the status of all your releases at a glance. You can filter the releases by namespace, release name, or status, and you can also view information about each release like release revision history, values and manifest files. Helm release viewer is built using Python, Flask, Jinja2 and Gunicorn it can be easily packaged as a Docker image or you can run it locally. Packaging it as a Docker image and having a Helm chart allows you to deploy it in your Kubernetes cluster in a matter of minutes. Helm Release Viewer is open source and available on GitHub at Helm Release Viewer repository.

Now let’s quickly deploy Helm Release Viewer in our Kubernetes cluster. The easiest way to deploy a Helm Release Viewer is to use the Helm chart that I have already created. Raw chart templates are available in the Helm Release Viewer repository and it also automatically being packaged and published to github pages of the repository. So to install Helm Release Viewer in your Kubernetes cluster you can use the following commands:

# Add Helm Release Viewer Helm repository to your Helm
helm repo add helm-release-viewer https://sysadminas.eu/helm-release-viewer
# Update Helm repositories
helm repo update
# List and Test Helm Release Viewer Helm chart
helm template helm-release-viewer/helm-release-viewer
helm upgrade --install helm-release-viewer helm-release-viewer/helm-release-viewer --namespace helm-release-viewer --create-namespace --dry-run
# Install Helm Release Viewer Helm chart
helm upgrade --install helm-release-viewer helm-release-viewer/helm-release-viewer --namespace helm-release-viewer --create-namespace 

Please keep in mind that you can also customize the Helm Release Viewer deployment by providing your own values file. You can find possible configuration options in the Helm Release Viewer helm chart configuration section. You can pass your custom values by providing a values file with the --values flag. For example:

helm upgrade --install helm-release-viewer helm-release-viewer/helm-release-viewer --namespace helm-release-viewer --create-namespace --values my-values.yaml

or you can also pass the values directly in the command line using the --set flag. For example:

helm upgrade --install helm-release-viewer helm-release-viewer/helm-release-viewer --namespace helm-release-viewer --create-namespace --set deploymwnt.image.tag=latest

Successful installation of Helm Release Viewer Helm chart will output message similar to the following:

Thank you for installing Helm Release Viewer!
UI normally can be accessed:
 * Locally by running: 
   kubectl port-forward --namespace default svc/helm-release-viewer 8080:80
   and then opening 
   http://localhost:8080 in your web browser
 * Via the internet using the following ingress resource:
   - Hostname: https://helm-release-viewer.mydomain.com/

⚠️ Important: Usage of ingress is optional however it is highly recommended if you are planning to use Helm Release Viewer in a production environment and you will need more than one replica of the Helm Release Viewer pod. Ingress allows most easiest way to configure sticky sessions which in current version of Helm Release Viewer is required to make sure that filtering and sorting works correctly in multi-replica environment.

⚠️ Important: Also by default helm chart is configured to create ClusterRole and ClusterRoleBinding which gives Helm Release Viewer service account almost full access to the cluster. However you can easily switch to read only access by changing appropriate values in the values.yaml file. Also instead of use predefined read only or full access cluster roles you can specify your own cluster role permissions in the values.yaml file. All these option can be defined in the values.yaml rbac section.

⚠️ Important: Helm Release Viewer selects and analyzes all namespaces in the cluster, but you can also limit it to the some specific namespaces by passing the label selector to the environment variable SELECTORS in the deployment. For example, to limit the namespaces to only those that have the label app: myapp you add to deployment.additionalEnvs new env SELECTORS and set the value to app=myapp. You can also pass multiple selectors separated by a comma. For example, to limit the namespaces to only those that have the label app: myapp and env: production you set deployment.additionalEnv SELECTORS to app=myapp,env=production.

For complete details on how to configure Helm Release Viewer Helm chart please refer to the Helm Release Viewer Helm chart configuration section.

As deployment notes suggest depending on your configuration you can access Helm Release Viewer UI either by port-forwarding the service to your local machine or by using the ingress resource. Of course if your service is exposed via NodePort or LoadBalancer you can also access it using the external IP address of the service.

In my case I deployed Helm Release Viewer in my local kind cluster and I will access it by port-forwarding. In my case service is configured to listen on port 80 which is default value in your case it might be different. To access Helm Release Viewer UI by port-forwarding run the following command:

kubectl port-forward --namespace default svc/helm-release-viewer 8080:80

and then open http://localhost:8080 in your web browser Helm Release Viewer UI welcomes you with the following screen:

As you can see in my environment I have only one release and it’s helm-release-viewer itself. If needed you can filter/search data by namespace, release name or status also sorting by various columns available. Additionally it is possible to view the release revision history, values and manifest files by clicking on appropriate column in the table:

Revision History:

Manifest:

Values (As I did not provide any values file to the Helm Release Viewer deployment it uses default values which are defined in the values.yaml file of the Helm chart. As consequence helm get values command will not return any values so this is why you see empty values in the screenshot above however if you provide your own values file you will see the values that you have provided):

You also can run application locally in this case your current kubectl context will be used to fetch the data from the cluster. You can pass applications parameters by setting appropriate environment variables.

To run the application locally you need to perform the following steps:

1. Clone the repository

git clone https://github.com/andriktr/helm-release-viewer.git

1. Change to the app directory

cd helm-release-viewer/app

1. Install the required dependencies by running pip install -r requirements.txt

pip install -r requirements.txt

1. Set the IN_CLUSTER environment variable to false and if needed set other environment variables

export IN_CLUSTER=false

1. Run the application

python helm_release_viewer.py

1. Access the application by opening http://localhost:8080 in your web browser

To make the application faster and more responsive I have implemented a simple caching mechanism that caches helm releases for specified time. You can configure the cache time and cache size by passing CACHE_TTL and CACHE_SIZE environment variables to the deployment. As my testing show this significantly improves the performance of the application in large clusters with hundreds of namespaces and releases. Additionally to the caching data from namespace is being fetched in parallel which also improves the performance of the application.

Additionally a warm-up mechanism is implemented which fetches all the data from the cluster on the application start-up and does it regularly in the background. This allows to have the data ready when the user opens the application. It also useful in large clusters where fetching all the data can take some time.

So this is the first version of Helm Release Viewer and I have some ideas on how to improve it. I’m planning to add more features like uninstalling releases or rolling back them to desired revision. If you have any ideas or suggestions on how to improve Helm Release Viewer please let me know by creating an issue in the Helm Release Viewer GitHub repository. Also if you would like to contribute to the project you are more than welcome to do so. I’m open to any suggestions and contributions.

I really hope that you will find Helm Release Viewer useful especially if you prefer UI over CLI. I’m looking forward to hearing your feedback and suggestions.

Thank you for reading and stay tuned for more posts!

Hi All, In one of my previous posts I wrote about how to write and deploy your first Golang app to K8S. In this post we will write a Python app and deploy it to Kubernetes cluster. Even if you not a developer (I’m also not) you for 100% have heard about Python. Python is probably one of the most popular programming languages in the world. It is used in many different areas, from web development to data science. Python is a very powerful language, and it is very easy to learn especially when Chat GPT comes to the rescue. Knowing Python for you as a DevOps engineer, K8S operator or SysAdmin is a huge advantage which can make your life easier and your work more efficient. In this post we will write a simple beginner oriented Python application which will go through the list of the website addresses and check a expire date of the SSL certificates. If the certificate is about to expire in less than X days, the application will send an email to the defined responsible person. The application will be deployed to Kubernetes cluster and will be run as a Cronjob.

Prerequisites

In order to develop and test python application you need to have Python installed on your machine. If you are on macOS you can install Python with Homebrew:

# Install Python with Homebrew
brew install python
# Confirm Python version
python --version

For all other platforms please follow the official installation guide

Python application code

When you have Python installed on your machine you can start writing your first Python application. I’m positioning myself as a Python beginner, so code might not be the super optimized and there might be some areas for improvement, but in general it does the job it’s easy to understand and results are satisfying. So here is the code (let’s name app cert-checker.py):

import datetime
import ssl
import socket
import logging
import sys
import time
import os
import subprocess
import yaml
import smtplib
import warnings

from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.mime.application import MIMEApplication

logging.basicConfig(level=logging.INFO , stream=sys.stdout, format='%(asctime)s %(levelname)s %(message)s', datefmt='%d/%m/%Y %I:%M:%S %p')
# Suppress the DeprecationWarning
warnings.filterwarnings("ignore", category=DeprecationWarning)

# Comment out the following lines if you want to run the app locally and uncomment the lines below them if you want to run the app in a container
email_alerts_enabled = os.environ["EMAIL_ALERTS_ENABLED"]
from_email = os.environ["FROM_EMAIL"]
smtp_server = os.environ["SMTP_SERVER"]
config_file_path = os.environ["CONFIG_FILE_PATH"]

# Uncomment the following lines if you want to run the app locally and comment out the lines above them if you want to run the app in a container
# email_alerts_enabled = "true"
# from_email = "from_email"
# smtp_server = "smtp_server"
# config_file_path = "config_file_path"


logging.info(f"Starting sites cert checker")
def email_alert(to_email, from_email, smtp_server, host, port, site_status, expiration_date):
    msg = MIMEMultipart()
    msg['To'] = to_email
    msg['From'] = from_email
    msg['Subject'] = f"CERTIFICATE EXPIRATION ALERT For {host}:{port}"
    if site_status == "near_expiration":
          body_text_html_part_1 = f"<p>Hello,<br>The certificate for <span style=\"color:yellow\"><b>{host}:{port}</b></span> will expire on <span style=\"color:yellow\"><b>{expiration_date}</b></span>.</p>"
          body_text_html_part_2 = "<p>Please take appropriate action to renew the certificate before it expires.</p>"
          body_html = MIMEText("<html><body><p>" + body_text_html_part_1 + "</p><p>" + body_text_html_part_2 + "</p></body></html>", "html")
    if site_status == "expired":
          body_text_html_part_1 = f"<p>Hello,<br>The certificate for <span style=\"color:red\"><b>{host}:{port}</b></span> is expired on {expiration_date}</p>"
          body_text_html_part_2 = "<p>Please take immediate appropriate action to renew the certificate.</p>"
          body_html = MIMEText("<html><body><p>" + body_text_html_part_1 + "</p><p>" + body_text_html_part_2 + "</p></body></html>", "html")
    if site_status == "error" or site_status == "unreachable" or site_status == "non_ssl_site":
          body_text_html_part_1 = f"<p>Hello,<br>There was an error (status code: <span style=\"color:red\"><b>{site_status}</b></span>) checking the certificate for <span style=\"color:red\"><b>{host}:{port}</b></span> please double check if site is reachable and certificate is readable.</p>"
          body_text_html_part_2 = "<p>Please take immediate appropriate action to fix the issue.</p>"
          body_html = MIMEText("<html><body><p>" + body_text_html_part_1 + "</p><p>" + body_text_html_part_2 + "</p></body></html>", "html")
    msg.attach(body_html)
    logging.info(f"----- Sending alert to {to_email}-----")
    server = smtplib.SMTP(smtp_server)
    try:
        server.sendmail(from_email, to_email, msg.as_string())
        server.quit()
    except Exception as e:
        logging.error(f"Error sending email: {e}")
        raise Exception(f"Error sending email") from e

def check_site_availability(host, port):
    try:
        # First, try to establish an SSL connection
        sock = socket.create_connection((host, port), timeout=10)
        context = ssl.SSLContext(ssl.PROTOCOL_TLS)
        context.verify_mode = ssl.CERT_REQUIRED
        context.check_hostname = True
        context.load_default_certs()
        conn = context.wrap_socket(sock, server_hostname=host)
        conn.close()
        return "available"
    except ssl.SSLError as e:
        # If the SSL connection fails with an SSLError, check the message of the exception
        if "certificate verify failed" in str(e):
            # This means the site is an SSL site but has an untrusted certificate
            return "untrusted_certificate"
        else:
            # This means the site is not an SSL site, try to establish a non-SSL connection
            try:
                sock = socket.create_connection((host, port), timeout=10)
                sock.close()
                return "non_ssl_site"
            except Exception:
                # If the non-SSL connection also fails, the site is unreachable
                return "unreachable"
    except Exception:
        # If the SSL connection fails with a different exception, the site is unreachable
        return "unreachable"
            
def site_cert_checker(host, port, validity_threshold):
     try:
          cmd = f"echo | openssl s_client -servername {host} -connect {host}:{port} 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null"
          result = subprocess.check_output(cmd, shell=True).decode().strip()
          expiration_date_str = result.split("=")[1]
          expiration_date = datetime.datetime.strptime(expiration_date_str, '%b %d %H:%M:%S %Y %Z')           
          #Calculate the date 30 days from now
          threshold_date = datetime.datetime.utcnow() + datetime.timedelta(days=validity_threshold)
          current_date = datetime.datetime.utcnow()
          # Check if the certificate will expire in less than 30 days
          if expiration_date < threshold_date:
               logging.warning(f"The certificate for {host} will expire on {expiration_date}")
               site_status = "near_expiration"
          elif expiration_date < current_date:
               logging.error(f"The certificate for {host} is expired!")
               site_status = "expired"                
          else:
               logging.info(f"The certificate for {host} is still valid until {expiration_date} so it's more than defined {validity_threshold} days threshold")
               site_status = "valid"
     except subprocess.CalledProcessError as e:
          logging.exception(f"Error checking site {host}: {e}")
          site_status = "error"
     return site_status, expiration_date

def main():
     # In case you want to inspect the container, uncomment the following line this will give you 20 minutes to inspect the container then 
     # main code will start running. It's useful for debugging purposes for example if you want to check if the config file is mounted correctly.
     #time.sleep(1200)
     
     with open(config_file_path, 'r') as f:
        data = yaml.safe_load(f)
        sites = data.get('sites', [])
        logging.info(f"Total amount of sites to be checked: {len(sites)}")
        for site in sites:
            host, port, validity_threshold, to_email = site['name'], site['port'], site['threshold'], site['email']
            logging.info(f"Checking site: {host}")
            site_availability = check_site_availability(host, port)
            if site_availability != "unreachable" and site_availability != "non_ssl_site":
               site_status, expiration_date = site_cert_checker(host, port, validity_threshold)
               if email_alerts_enabled == "true":
                    for email in to_email:
                         if site_status == "near_expiration":
                              email_alert(email, from_email, smtp_server, host, port, site_status, expiration_date)
                         if site_status == "expired":
                              email_alert(email, from_email, smtp_server, host, port, site_status, expiration_date)
                         if site_status == "error":
                              email_alert(email, from_email, smtp_server, host, port, site_status, "N/A")             
            else:
                if site_availability == "unreachable":
                    logging.error(f"Site {host} is unreachable!")
                    site_status = site_availability
                    if email_alerts_enabled == "true":
                         for email in to_email:
                              email_alert(email, from_email, smtp_server, host, port, site_status, "N/A")
                    continue
                if site_availability == "non_ssl_site":
                    logging.error(f"Site {host} is not an SSL site!")
                    site_status = site_availability
                    if email_alerts_enabled == "true":
                         for email in to_email:
                              email_alert(email, from_email, smtp_server, host, port, site_status, "N/A")
                    continue
if __name__ == "__main__":
     main()

Run the application locally

As you can see the code is pretty simple and should be easy to understand if you have some basic Python knowledge. Any way let’s shortly describe what the code does:

• First of all we import all the necessary modules. Then we define some variables which will be used later in the code. Then we define a function which will send an email alert to the responsible person(-s).

• Then we define a function which will check if the site is available.

• Next we define a function which will check the certificate expiration date.

• Then we define a main function which will be the main entry point of our application. In the main function we read the config file which is a sitse.yaml file containing the list of the sites to be checked. The structure of the sites.yaml file is the following:

sites:
  - name: site1.com
    port: 443
    threshold: 30
    email:
      - email1@domain.com
      - email2@domain.com

Then we iterate through the list of the sites and check the certificate expiration date. If the certificate is expire or about to expire in less than 30 days, the application will send an email to the responsible email address(-es) defined in the sites.yaml file. Also if the site is unreachable or not an SSL site the application will send appropriate email alert to the responsible person(-s) email address(-es) defined in the sites.yaml file. Optionally you can enable or disable email alerts by setting the EMAIL_ALERTS_ENABLED environment variable to true or false. If you set it to false the application will not send any email alerts and will output all the logs to the console only.

We now have to steps completed. We have Python installed on our machine and we have a Python application code ready. Now let’s specify the requirements for our application and create a requirements.txt file which will contain all the necessary modules which our application needs to run. The requirements.txt file will look like this:

PyYAML

In our case we need only one module which is PyYAML which is a YAML parser and emitter for Python. Also we do not specify the version of the module, so the latest version will be installed. In case you want to specify the version of the module you can do it like this:

PyYAML==5.4.1

To install the module specified in the requirements.txt file you can run the following command:

pip install -r requirements.txt

Now we have all the necessary prerequisites to run our application locally. Application is meant to be run in a container, but with simple comment/uncomment of the lines in the code you can run it locally. To run the application locally adjusting the following lines in the code:

# Comment out the following lines if you want to run the app locally and uncomment the lines below them if you want to run the app in a container
email_alerts_enabled = os.environ["EMAIL_ALERTS_ENABLED"]
from_email = os.environ["FROM_EMAIL"]
smtp_server = os.environ["SMTP_SERVER"]
config_file_path = os.environ["CONFIG_FILE_PATH"]

# Uncomment the following lines if you want to run the app locally and comment out the lines above them if you want to run the app in a container
# email_alerts_enabled = "true"
# from_email = "from_email"
# smtp_server = "smtp_server"
# config_file_path = "config_file_path"

Then create a sites.yaml and put it in the CONFIG_FILE_PATH directory. The sites.yaml file should be formatted as follows:

sites:
  - name: sysadminas.eu
    port: 443
    threshold: 30
    email:
      - andriktr@gmail.com

Then run the application with the following command:

python cert-checker.py

If everything is configured correctly application will process and output the logs to the console and send email alert if conditions are met. The output in the console will look like this (here we are checking sysadminas.eu site certificate):

Deploy the application to Kubernetes cluster

OK we now tested application locally and it works as expected. Our aim is to deploy the application to Kubernetes cluster. So let’s do it. First let’s wrap our application in a Docker container. To do that we need to create a Dockerfile which will look like this:

FROM python:3.12-slim

RUN apt-get update && apt-get install -y curl openssl

RUN mkdir /app

WORKDIR /app

COPY requirements.txt .

RUN python -m pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt && rm requirements.txt

USER 1000

COPY cert-checker.py .

CMD ["python", "cert-checker.py"]

In the Dockerfile we are using the official Python 3.12 slim image as a base image. Then we install curl (having curl might be useful for debugging purposes) and openssl (openssl is required for site_cert_checker function where it used to check certificate) packages. Then we create a directory for our application and set it as a working directory. Next we copy the requirements.txt file to the working directory and install all the necessary modules. Also we set USER to 1000 as per best security practices. Then we copy the cert-checker.py file to the working directory and set the entry point of the container to run the cert-checker.py file. Now we can build our Docker image with the following command:

# Build Docker image
docker build -t andriktr/cert-checker:0.1.0 --no-cache --pull --platform linux/amd64 .
# Push Docker image to Docker Hub
docker push andriktr/cert-checker:0.1.0

Best way to deploy our application to Kubernetes cluster is to use Helm chart. I have already written a Helm chart for this application which you can find here. Easiest way would be to clone the cert-checker repository, adjust the values in the values.yaml file and run helm command for deployment. So let’s do it.

# Clone the repository
git clone https://github.com/andriktr/cert-checker.git

Now let’s adjust the values in the values.yaml file. You can find the description of the values in the README.md file.

Once ready we can test and deploy the Helm chart with the following commands:

# Test cert-checker Helm chart
helm upgrade --install cert-checker cert-checker --namespace cert-checker --create-namespace --dry-run --debug

# Deploy cert-checker Helm chart
helm upgrade --install cert-checker cert-checker --namespace cert-checker --create-namespace

Once you run the above commands you should see a cronjob created in the cert-checker namespace which will spin up a pod according the schedule specified in the values.yaml file.

Repository cert-checker contains all the necessary files and instructions to test and deploy the application either locally or to Kubernetes cluster. Feel free to use it and adjust it according to your needs.

I hope you will find this post useful and it will help you in your day to day work.

See you in the next post 🤜 🤛, bye!

Hi All,

This post will be a references to my recently created github repository Consul-OIDC-Azure-AD which contains a Terraform configuration which will help you to configure OIDC authentication with Azure AD in your Consul cluster.

If you like me using Consul as a Service Mesh and your cloud provider is Azure it will be a really good idea to use Azure AD as your OIDC provider. In this case you will be able to assign Consul roles to your Azure AD users and groups and your users will be able to authenticate to Consul using their Azure AD credentials. This will simplify your life as you will no longer need to issue separate Consul tokens for your users.

So I encourage you to check out Consul-OIDC-Azure-AD repository and try it out. There is a detailed README which will guide you through the process of deploying the configuration.

If you have any questions or suggestions please feel free to contact me.

See you soon in the next post 🤜 🤛, bye!

Hi All,

Kubernetes has become the de facto standard for container orchestration. It is a powerful tool that can help you manage your containerized applications. However, it is not a silver bullet. It is a complex system that requires a lot of attention and care. In this post I try to overview some tools and general best practices which IMO are must have for successful Kubernetes (AKS) journey.

Infrastructure as Code

In current DevOps era infrastructure as code is a must have for any serious IT project. It allows you to automate the deployment of your infrastructure and to manage it in a consistent and repeatable way. It also allows you to version control your infrastructure and to track changes. If we talk about AKS specifically there are a bunch of different tools which you can use for IaC accomplishment. You can use Azure CLI, Terraform, ARM templates, Ansible, Azure Bicep, Powershell etc. Each of these tools has its own pros and cons, but personally I prefer to use Hashicorp’s Terraform which is one of the most popular and easy to use IaC languages in todays DevOps market. Terraform covers all the major cloud providers and much more. You can find more information about Terraform and AKS here. Having your infrastructure as code is a great decision, but keep in mind that cloud providers constantly evolving and changing their API’s. So the code which you wrote X days ago may not work today. So you need to have a constant automated way to test your code on a regular basis. In case of AKS infrastructure from my experience I can recommend to have some kind of experimental environment which should be deployed using exact same code as was used for dev/prod environments. Then you should have an automated process for example a CI/CD pipeline which will regularly redeploy this experimental environment from scratch and will run some smoke tests. It’s quite easy to achieve with Terraform and Azure DevOps. By having such setup you will be sure that your IaC code is working and you will be able to redeploy/update other business critical environments. Also such approach may help you to test new features and new versions of your infrastructure before rolling them out to more important environments.

Cluster maintenance and management

When we have our Kubernetes cluster up and running we need to have right tools to manage it. There are allot of tools which can help you with that and here are some of them which I use and recommend.

Kubectl

kubectl - is a number 1 command-line tool for controlling Kubernetes clusters. It allows you to run commands against Kubernetes clusters. You can use it to deploy applications, inspect and manage cluster resources, and view logs. It is a must have tool for any Kubernetes cluster.

Lens

Lens - It is a desktop application that allows you to manage your Kubernetes clusters. It is a great tool for developers and DevOps engineers. It has a lot of useful features like cluster management, monitoring, debugging, and more. It is available for Windows, Mac, and Linux.

Kube-node_shell

kube-node_shell - is a simple tool that allows you to run commands on your Kubernetes nodes. It is a great tool for troubleshooting and debugging. It is a must have tool for any Kubernetes cluster.

K9S

k9s - provides a terminal UI to interact with your Kubernetes clusters. The aim of this project is to make it easier to navigate, observe and manage your applications in the wild. K9s continually watches Kubernetes for changes and offers subsequent commands to interact with your observed resources.

Kured

kured/kubereboot - is a Kubernetes DaemonSet that performs safe automatic node reboots when required. It is a great tool for cluster maintenance and management. In terms of AKS it is a must have tool because node updates are managed automatically by Azure however you as a cluster administrator is responsible for the nodes reboot and kured/kubereboot can help you with that.

Networking

Another very important and probably most dificult part of Kubernetes ecosystem is networking. It is crucial to have a secure, reliable, easy configurable connections from and to applications running in your cluster.

Ingress Controller

If you plan to expose your applications outside of your cluster you may need to have an Ingress controller. There are a lot of Ingress controllers like Nginx, Traefik, HAProxy, Kong etc. Personally I use open source version of ingress for official k8s repo ingress-nginx. I choose it because it is highly supported by community easy to use and configure. As alternative you may consider to use Azure Application Gateway Ingress Controller which is more native for AKS. Also there other NGINX ingress version officially distributed by NGINX/F5 F5 NGINX Ingress Controller for Kubernetes whey offer free and paid versions. Here you can find some comparison between top ingress controllers.

Service Mesh

A service mesh is a dedicated infrastructure layer that adds features to a network between services. It allows to control traffic and gain insights throughout the system. Observability, traffic shifting (for canary releasing), resiliency features (such as circuit breaking and retry/timeout) and automatic mutual TLS can be configured once and enforced in a decentralized fashion. In contrast to libraries, which are used for similar functionality, a service mesh does not require code changes. Instead, it adds a layer of additional containers that implement the features reliably and agnostic to technology or programming language. There are a lot of service meshes like Istio, Linkerd, Consul, Open Service Mesh etc. Currently I’m using Consul which is quite is very powerful service mesh with a lot of different features. Main benefit of Consul is that it can be used not only for K8S service but with any other service outside the cluster. However be ready to spend some time in Consul documentation in order to understand how everything works. In terms of AKS the only one officially supported service mesh is Open Service Mesh(OSM). OSM is driven by Microsoft and can be enabled as add-on. If you want to get more info about service meshes including detailed comparison of various solutions I encourage you to check the following source.

High Availability

Autoscaling

In all production environments high availability is a critical requirement. First things which is needed to achieve high availability in AKS and make sure that your pods will be scheduled is to have autoscaling enabled and properly configured. You can find more information about AKS autoscaling here. Autoscaling is a great tool for high availability it will make sure that you have enough compute resource to run your applications. Down scaling will ensure that you will not pay for unused resources.

Horizontal Pod Autoscaler

Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics). You can find more information about HPA here. HPA is a great tool for high availability it will make sure that you have enough compute resource to run your applications.

Descheduler

Scheduling in Kubernetes is the process of binding pending pods to nodes, and is performed by a component of Kubernetes called kube-scheduler. The scheduler’s decisions, whether or where a pod can or can not be scheduled, are guided by its configurable policy which comprises of set of rules, called predicates and priorities. The scheduler’s decisions are influenced by its view of a Kubernetes cluster at that point of time when a new pod appears for scheduling. As Kubernetes clusters are very dynamic and their state changes over time, there may be desire to move already running pods to some other nodes for various reasons:

Some nodes are under or over utilized. The original scheduling decision does not hold true any more, as taints or labels are added to or removed from nodes, pod/node affinity requirements are not satisfied any more. Some nodes failed and their pods moved to other nodes. New nodes are added to clusters. Consequently, there might be several pods scheduled on less desired nodes in a cluster. Descheduler, based on its policy, finds pods that can be moved and evicts them. Please note, in current implementation, descheduler does not schedule replacement of evicted pods but relies on the default scheduler for that.

KEDA

KEDA allows for fine-grained autoscaling (including to/from zero) for event driven Kubernetes workloads. KEDA serves as a Kubernetes Metrics Server and allows users to define autoscaling rules using a dedicated Kubernetes custom resource definition. For AKS KEDA can be enabled as add-on.

Monitoring and Logging

Kube-capacity

kube-capacity - kube-capacity is a simple CLI tool (it’s like a kubectl top, but more advanced version) that provides an overview of the resource requests, limits, and utilization in your cluster. It is a great tool for monitoring your cluster and to see how much resources you have available and utilized.

Kube-prometheus-stack

kube-prometheus-stack - kube-prometheus-stack is a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

Filebeat and Fluentd

filebeat - Filebeat is a lightweight shipper for forwarding and centralizing log data. It is a great tool for collecting the logs from your pods and sending them to the central location such as ElasticSearch. As alternative you can consider to use Fluentd.

Azure Monitor

For AKS monitoring you can combine existing tools with built-in Azure monitoring solutions such as Azure Monitor and Azure Monitor for containers. You can also use Azure Workbooks to create custom dashboards and Azure Alerts to create alerts and be notified when something goes wrong.

CI/CD

Azure DevOps

Azure DevOps is a great tool for CI/CD. You can use it to build, test and deploy your application image to AKS according best Git practices.

Helm

There are couple of different template languages for Kubernetes such as Helm, [Kustomize](https://kustomize.io/, Kapitan. However Helm is the most popular one. Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application. Helm can be easily integrated in any CI/CD solution like Azure DevOps Pipelines. Also almost all serious application providers offer Helm charts for their applications.

ArgoCD

Both ArgoCD continuous delivery. Comparing to Azure DevOps Pipelines ArgoCD works in pull mode. It means that ArgoCD will pull the latest version of your application from the repository and deploy it to the cluster. ArgoCD syncs the cluster state with the desired state defined in the Git repository. Also ArgoCD supports not only plain yaml files but Helm charts. Competitor of ArgoCD is Flux it has less UI and it is more CLI oriented.

Security

Kyverno

Kyverno is a policy engine designed for Kubernetes. Using Kyverno, you can enforce policies that are impossible using native Kubernetes admission controls. Using kyverno you can control your cluster easily and effectively. All pod security best practices can be implemented using Kyverno policies. Even more you can write policies which will generate new resources like quotas, network policies, limitranges, etc. this help automate your cluster management. Comparing tools like OPA Gatekeeper Kyverno is a winner IMO, because writing policies in kyverno is much easier and do not require familiarity with Rego language also Kyverno allows to generate new resources based on policies.

Azure Policy

In case if Open Source tools are not allowed by your company when your choice for AKS cluster protection should be Azure Policy for Kubernetes which are based on OPA Gatekeeper engine.

In my case I merge Kyverno and Azure Policy together. I use Kyverno for in all modes (enforce, audit, generate) and Azure Policy for audit only.

Azure Workload Identity

In order to securely access Azure resources such as Azure Key Vault or Storage Accounts from your AKS cluster using Azure Workload Identity is a must. It allows you to securely access Azure resources from your AKS cluster pods without using any credentials.

Trivy-operator

The Trivy-Operator leverages trivy security tools by incorporating their outputs into Kubernetes CRDs (Custom Resource Definitions) and from there, making security reports accessible through the Kubernetes API. This way users can find and view the risks that relate to different resources in what we call a Kubernetes-native way. The Trivy operator automatically updates security reports in response to workload and other changes on a Kubernetes cluster, generating the following reports:

• Vulnerability Scans: Automated vulnerability scanning for Kubernetes workloads.
• ConfigAudit Scans: Automated configuration audits for Kubernetes resources with predefined rules or custom Open Policy Agent (OPA) policies.
• Exposed Secret Scans: Automated secret scans which find and detail the location of exposed Secrets within your cluster.
• RBAC scans: Role Based Access Control scans provide detailed information on the access rights of the different resources installed.
• K8s core component infra assessment scan Kubernetes infra core components (etcd,apiserver,scheduler,controller-manager and etc) setting and configuration.
• Compliance reports NSA, CISA Kubernetes Hardening Guidance v1.1 cybersecurity technical report is produced.

Private container registry

If you really aware about your cluster security and container images you should use a private container registries instead of public ones. In terms of Azure and AKS you can use Azure Container Registry which can be easily attached to your AKS cluster. Please note that even if you use private container registry you should still use image scanning tools like trivy and integrate this process into your CI/CD pipeline. This will help you to find and prevent vulnerabilities in your container images before you push them to the registry and deploy to the cluster.

Microsoft Defender for Cloud

Microsoft Defender for Cloud also has some features which can help you to secure your cluster and prevent misconfigurations. The following features are available:

• Protects Kubernetes clusters running on Azure Kubernetes Service and Kubernetes on-premises/IaaS
• Identifies misconfigurations in Kubernetes clusters
• Vulnerability assessment for images stored in ACR registries
• Vulnerability assessment for images running in Azure Kubernetes Service
• Run-time threat protection for nodes and clusters
• Provides guidelines to help investigate and mitigate identified threats

Azure defender for cloud support most of major cloud providers as well as on-premises Kubernetes clusters. You can find more details here.

Falco

Falco is a runtime security tool that helps you to monitor your Kubernetes cluster. Falco is a Cloud Native Computing Foundation (CNCF) project. Falco is a behavioral activity monitor designed to detect anomalous activity in your applications. You can use Falco to monitor run-time security of your Kubernetes applications and internal components. Falco allows you to continuously monitor and detect container, application, and host-level anomalous activity. Falco leverages the rules engine and Lua scripting engine to define and execute rules. Falco is a great tool for detecting threats and vulnerabilities in your cluster.

Disaster Recovery

Having a disaster recovery plan is a critical peace of infrastructure puzzle in any organization. First thing first we already have our infrastructure defined in the code this will help us to easily rebuild everything in case of disaster, misconfigurations, malicious actions, etc. When in terms of AKS or Kubernetes we simply need to redeploy our applications to the fresh cluster as probably most of microservices running in k8s are typically stateless. However I’m more than sure that you will have some stateful apps as well. For such cases we need to have a regular backups which will save us in case of disaster or if for some reason our application data will become inconsistent. For this I can highly recommend Kasten or K10. I strongly believe that K10 currently is Nr.1 backup solution for Kubernetes environments. If your clusters are not large enough (up to 5 nodes) you can use Kasten for free. For larger infrastructures be ready to pay some $$$. Also as alternative you can consider to use Velero which is also a great tool for backup and restore your Kubernetes workloads.

Costs

Last but not least we need to be able to evaluate our infrastructure costs and split them by different teams, projects, departments, etc. One of the tools which can be helpful is Kubecost. This tools helps you monitor and manage cost and capacity in Kubernetes environments and has integration with major cloud providers.

Conclusion

Kubernetes is a great tool for building and managing containerized applications. However it’s not enough to just deploy a cluster and run your applications. You need to add some work in order to have well managed, secure and cost-effective ecosystem.

I hope this post will be helpful for you and would like to thank you for reading it.

See you 🤜 🤛, bye!

Hi All,

In todays post I would like to talk about trivy-operator. Trivy-operator is a Kubernetes operator which can provide:

• Vulnerability reports for container images running in your cluster

• Configuration Audit reports for your cluster

• Exposed secret reports for your cluster

• RBAC Assessment reports for your cluster

• Cluster RBAC Assessment reports for your cluster

It is based on Trivy (I wrote about trivy in one of my previous post already), a simple and comprehensive vulnerability scanner for containers and other artifacts. Trivy is a static analysis tool that detects vulnerabilities of OS packages (Alpine, RHEL, CentOS, etc.) and application dependencies (Bundler, Composer, npm, yarn, etc.) in container images. Both trivy and trivy-operator are open source projects and are developed by Aqua Security.

In terms of Kubernetes trivy-operator is a deployment which runs in your cluster which watches your namespaces and scans target workloads like pod, replicasets, statefulsets, daemonsets, jobs, cronjobs, secrets, roles, rolebindings, clusterroles, clusterrolebindings.

Trivy-operator deployment process is pretty straightforward. However in terms of AKS if we use Azure Container Registry (ACR) as private registry for images used in our cluster we need to use Azure Pod Identity in order to make trivy-operator able to scan these images and access ACR. If you are not familiar with Azure Pod Identity I would recommend to read my previous post about it.

So let’s get started and jump into our demo. First thing first let’s define a common variables which we will use in our setup.

# Azure Tenant ID
export AZ_TENANT_ID="<Your Azure Tenant ID goes here>"
# Azure Subscription Name
export AZ_SUBSCRIPTION_NAME="Visual Studio Premium with MSDN"
# Location
export AZ_LOCATION="westeurope"
# AKS Cluster Name
export AKS_RESOURCE_GROUP="sysadminas-aks"
# AKS Cluster Name 
export AKS_CLUSTER_NAME="sysadminas"
# AKS Cluster Node Resource Group Name
export AKS_NODE_RESOURCE_GROUP="sysadminas-aks-nodes"
# Resource Group for ACR
export ACR_RESOURCE_GROUP="sysadminas-aks"
# Name of the ACR used in AKS cluster
export ACR_NAME="sysadminas"
# Name of the Azure managed identity which will be used by trivy-operator
export AZ_IDENTITY_NAME="sysadminas-trivy-operator-identity"
# Name of K8S namespace where deployment 
export NAMESPACE="trivy-operator"

Next ensure that our AKS cluster has Azure Pod Identity addon enabled. To do that we need to run following command:

# Enable Azure Pod Identity (it safe to run it even if it is already enabled)
az aks update --enable-pod-identity --subscription $AZ_SUBSCRIPTION_NAME --resource-group $AKS_RESOURCE_GROUP --name $AKS_CLUSTER_NAME

Now we need to create a new Azure managed identity which will be used by trivy-operator to access ACR. Also we will assign identity to the AKS cluster scale set. To do that we need to run following command:

# Create Azure managed identity
az identity create --name $AZ_IDENTITY_NAME --subscription $AZ_SUBSCRIPTION_NAME --resource-group $AKS_RESOURCE_GROUP  --location $AZ_LOCATION
# Get Azure managed identity resource ID and save it as a variable
export IDENTITY_ID=$(az identity show --name $AZ_IDENTITY_NAME --resource-group $AKS_RESOURCE_GROUP --subscription "$AZ_SUBSCRIPTION_NAME" --query id -o tsv)
# Get Azure managed identity client ID and save it as a variable
export IDENTITY_CLIENT_ID=$(az identity show --name $AZ_IDENTITY_NAME --resource-group $AKS_RESOURCE_GROUP --subscription "$AZ_SUBSCRIPTION_NAME"  --query clientId -o tsv)
# Get AKS cluster scale set name and save it as a variable
export AKS_SCALE_SET=$(az vmss list --resource-group $AKS_NODE_RESOURCE_GROUP --subscription "$AZ_SUBSCRIPTION_NAME" --query "[].name" -o tsv)
# Assign Azure managed identity to the AKS cluster scale set
az vmss identity assign --resource-group $AKS_NODE_RESOURCE_GROUP -n $AKS_SCALE_SET --identities $IDENTITY_ID --subscription "$AZ_SUBSCRIPTION_NAME"
# Get ACR resource ID and save it as a variable
export ACR_RESOURCE_ID=$(az acr show --name $ACR_NAME --resource-group $ACR_RESOURCE_GROUP --subscription "$AZ_SUBSCRIPTION_NAME" --query id -o tsv)
# Assign ACR pull role to the Azure managed identity
az role assignment create --role acrpull --assignee $IDENTITY_CLIENT_ID --scope $ACR_RESOURCE_ID --subscription "$AZ_SUBSCRIPTION_NAME"

OK, we seems finished with Azure side of things and now we can switch to Kubernetes side. Let’s create a namespace for trivy-operator as well as azure identity and binding for it:

# create namespace for trivy-operator
kubectl create namespace $NAMESPACE

# create azure identity for trivy-operator
cat <<EOF | kubectl apply -f -
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: $AZ_IDENTITY_NAME
  namespace: $NAMESPACE
spec:
    clientID: $IDENTITY_CLIENT_ID
    resourceID: $IDENTITY_ID
    type: 0 
EOF

# create azure identity binding for trivy-operator
cat <<EOF | kubectl apply -f -
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: $AZ_IDENTITY_NAME
  namespace: $NAMESPACE 
spec:
  azureIdentity: $AZ_IDENTITY_NAME  
  selector: $AZ_IDENTITY_NAME
EOF

Now we can configure and deploy trivy-operator. First let’s add Helm repository which contains trivy-operator chart and pull it to our local machine:

# add Helm repository
helm repo add aqua https://aquasecurity.github.io/helm-charts/
# update Helm repositories
helm repo update
# pull trivy-operator chart to local machine
helm pull aqua/trivy-operator --untar

Next let’s configure trivy-operator chart by creating our own values.yaml file to override some of defaults. We will save our settings into the separate file:

touch trivy-operator-values.yaml
cat <<EOF >> trivy-operator-values.yaml
excludeNamespaces: "kube-system"

operator:
  # scanJobsConcurrentLimit the maximum number of scan jobs create by the operator
  scanJobsConcurrentLimit: 2  
  # scanJobsRetryDelay the duration to wait before retrying a failed scan job
  scanJobsRetryDelay: 60s
  # infraAssessmentScannerEnabled the flag to enable infra assessment scanner
  infraAssessmentScannerEnabled: false
  # metricsVulnIdEnabled the flag to enable metrics about cve vulns id
  # be aware of metrics cardinality is significantly increased with this feature enabled.
  metricsVulnIdEnabled: true

# Prometheus ServiceMonitor configuration (enable this if you use Prometheus Operator and want to scrape metrics from trivy-operator)
serviceMonitor:
  # enabled determines whether a serviceMonitor should be deployed
  enabled: true
  # The namespace where Prometheus expects to find service monitors
  namespace: "kube-prometheus-stack"
  interval: "60s"
  # Additional labels for the serviceMonitor
  labels:
    release: kube-prometheus-stack

trivyOperator:
  # Our AAD Pod Identity selector which we set in AzureIdentityBinding
  scanJobPodTemplateLabels: "aadpodidbinding=sysadminas-trivy-operator-identity"

trivy:
  # resources resource limits (During my tests I have noticed that default values may be not enough for trivy to scan images)) 
  resources:
    limits:
      cpu: 2000m
      memory: 2000M
EOF

Now we are ready to deploy trivy-operator:

# Run helm template to generate Kubernetes manifests
helm template trivy-operator trivy-operator --namespace $NAMESPACE --values trivy-operator-values.yaml
# Run helm install/upgrade --dry-run --debug to see what will be deployed
helm upgrade -i --dry-run --debug trivy-operator trivy-operator --namespace $NAMESPACE --values trivy-operator-values.yaml
# Run helm install/upgrade to deploy trivy-operator
helm upgrade -i trivy-operator trivy-operator --namespace $NAMESPACE --values trivy-operator-values.yaml

Once we have deployed trivy-operator it immediately starts scanning all selected workloads in the cluster. Also once you spin up new pod, replicasets, deployment, etc. it will be scanned automatically. Trivy-operator scan jobs in kubernetes are represented as jobs.batch objects. Kubernetes jobs.batch creates a pod and runs it until it completes. Once the job is completed, the pod is deleted.

So we have deployed trivy-operator and it is scanning our cluster. But how can we see the results of the scan? Currently I see three options:

1. Use built in trivy-operator reports which which in kubernetes are represented as custom resources:

For example to view vulnerabilities report from all namespaces we can run:

kubectl get vulnerabilityreports.aquasecurity.github.io -A -owide

This results in to the following output:

To view more details we can use:

kubectl describe vulnerabilityreports.aquasecurity.github.io -n <<namespace>>

1. Second approach is to install Lens which is super cool Kubernetes IDE. Lens is a desktop application which allows you to manage Kubernetes clusters. It is available for Windows, MacOS and Linux. Once you install Lens you can connect to your cluster and see and manage all the resources in the cluster. Lens has an extensions for tryvi-operator which allows you to see the results of the scan in it’s UI.

You can see the results of the scan as a combined report:

Or if you click on the pod you can see the vulnerabilities for the specific pod:

If you have not tried Lens yet, I highly recommend it. It is awesome and very useful tool.

1. Third method is to use Prometheus to scrape metrics from trivy-operator and visualize them in Grafana.

To use this option make sure you have enabled serviceMonitor and operator.metricsVulnIdEnabled in trivy-operator-values.yaml. After this save and import the following json dashboard into Grafana.

As result you will able to see the following dashboard:

That’s it! We have deployed trivy-operator and it is scanning our cluster. We also can see results of the scan using different tools. Definitely trivy-operator is a great tool to scan your cluster and find vulnerabilities in your workloads. I hope this post was helpful and you will find it useful.

See you 🤜 🤛, bye!