KubeDriveDownloadDownloadDownload
Blog

Nothing is removed in Kubernetes 1.38. Your upgrade can still fail.

Kubernetes 1.38 is due on December 16. Open the deprecated API migration guide and the newest entry is 1.32: flowcontrol.apiserver.k8s.io/v1beta3, replaced by v1. Nothing for 1.33, 1.34, 1.35, 1.36, 1.37 or 1.38. The big waves, extensions/v1beta1 in 1.16, Ingress and CRD v1beta1 in 1.22, PodSecurityPolicy in 1.25, are years behind us.

And yet the Monday after an upgrade someone runs helm upgrade and gets unable to build kubernetes objects from current release manifest. Here is why, and what to look for the week before.

Objects survive. Manifests do not.

An upgrade does not delete anything. Every object already in the cluster was converted by the API server when it was stored, and it keeps being served at whatever version the new release supports. A Deployment created in 2019 as extensions/v1beta1 is an apps/v1 Deployment today and nobody noticed.

What breaks is whatever still writes the old version:

  • A Helm release’s stored manifest. Helm keeps the rendered manifest of each revision in a Secret and reads it on every upgrade and rollback to compute the diff. A release last upgraded in 2021 still says networking.k8s.io/v1beta1 in that Secret. The cluster upgrade does not touch it; the next helm upgrade fails. The release itself keeps running, which is why nobody finds this until they need to change it.
  • kubectl’s last-applied annotation. kubectl apply writes the manifest it was given into last-applied-configuration. The next apply of the same file, with the same old apiVersion, fails once that version is gone.
  • Clients. An old controller, an operator pinned two years ago, a CI job with a vendored client: anything still calling a removed version gets 404 after the upgrade. The API server counts these calls in its own metrics, apiserver_requested_deprecated_apis, so the evidence exists before the upgrade, if someone reads it.
  • Kinds removed with no successor. PodSecurityPolicy did not get a v1. On 1.25 the objects stopped existing, and what they enforced stopped being enforced.
  • Nodes. The control plane supports kubelets up to three minor versions older. Move the control plane to 1.38 and a node still on 1.34 falls out of the supported skew, with the cluster otherwise fine.

On a cluster going from 1.37 to 1.38, the last item is the only thing that is new. Everything else is debris from earlier waves that already fails today and has been waiting for the day someone touches it. The upgrade is when they do.

What the check does

Cluster → Upgrade check in KubeDrive takes a target version, the next one by default, and reads the cluster without changing anything:

  1. The latest revision of every Helm release, its stored manifest parsed for API versions the target no longer serves. The finding names the release, the kind, the version it should be, and what else changes in that kind beyond the apiVersion line: an Ingress’s serviceName and servicePort become service.name and service.port, and pathType is required; an HPA’s targetAverageUtilization becomes target.averageUtilization.
  2. Every object of a kind that had a removed version, metadata only, for the apiVersion kubectl last applied.
  3. The API server’s metrics, for clients that called deprecated versions since it started, with the request counts.
  4. Objects of kinds removed with no successor.
  5. The kubelet version of each node against the target.

Versions removed in the two releases after the target are listed too, so a cluster going to 1.38 also hears about what 1.39 and 1.40 would take, when there is anything. Right now there is not.

The overview shows Not ready for 1.38 when anything breaks, and the sidebar entry carries the count. A version removed before the current release counts as a break too: that Helm release fails today, not in December.

Fixing it from the same screen

A finding says what to write instead. With Pro, selecting it shows the manifest converted for the target version, fields moved and marked, ready to paste into the chart or the repository it came from. That is the right fix while the cluster still serves both versions: update the chart, upgrade the release, and the stored manifest is new.

If the cluster is already upgraded and helm upgrade is already failing, Rewrite stored manifest writes a new revision of the release with the API versions mapped, the way the helm mapkubeapis plugin does, without touching the cluster’s objects. It can be undone, and the details show the plugin command for people who would rather run it themselves.

Run the check on the week before the upgrade, not the Monday after. It takes a few seconds and touches nothing, and the guide has the details.