How KubeDrive opens 50,000 pods in 3 seconds
A Kubernetes client is easy to make fast on a laptop cluster with 30 pods. The cluster that matters is the one at work: hundreds of nodes, tens of thousands of pods, and an incident in progress. So we built a test cluster of that size and timed KubeDrive next to the clients people actually use.
The test
The cluster is kind with kwok fake nodes: kwok plays the kubelet, so 50,000 pods reach Running without a single container. We ran three shapes:
- 500 pods on 4 nodes, an ordinary cluster;
- 50,000 pods on 4 nodes, the most pods a small cluster can hold;
- 50,000 pods on 470 nodes of 110 pods each, the size of a large production cluster.
Each client starts cold, with a kubeconfig that holds only the test cluster, and opens pods in all namespaces. A small probe records only the client’s windows, reads every frame with the macOS text recognizer and notes when the first pod row appears and when the client’s own count reaches 50,000. Clients that need clicks to get to the pod list get them as soon as the item shows up. A second probe samples the memory of the whole process tree: Electron’s helpers and WebKit’s content processes count too. Every number below is the median of five runs on a 2018 MacBook Pro.
The results
On 470 nodes:
| Client | First rows | Full list | Memory |
|---|---|---|---|
| KubeDrive | 2.98 s | 10.4 s | 428 MB |
| Headlamp | 6.4 s | no count | 590 MB |
| Freelens | none in 150 s | — | 530 MB |
| k9s | 26.4 s | 26.4 s | 2.6 GB |
kubectl get pods -A |
25.5 s | 27.0 s | — |
With the same 50,000 pods on 4 nodes, KubeDrive shows the first rows in 1.53 s and all of them in 5.3 s, against 16.9 s for Aptakube and 34 s for k9s. Aptakube’s free plan opens clusters of up to 5 nodes, so it is not in the large run. The performance page has every profile and metric, and the scripts and raw output of each run are on GitHub.
Notice kubectl: it needs 27 s for a list KubeDrive shows in 10. Both talk to the same API
server; the difference is in what they ask for and when they show it.
Ask for tables, not objects
A pod as the API server stores it is around 13 KB of JSON once decoded: spec, status, managed
fields, annotations. A row in a pod list needs a few hundred bytes. So KubeDrive does not list
objects at all. It asks the API server for the
Table
representation, the one kubectl get prints, with only the metadata of each object. The server
renders the columns itself, which also gives custom resources their own columns from
additionalPrinterColumns without any code on our side. The full object is fetched only when you
open it.
Show the first page first
A list request with resourceVersion=0, which is what client-go’s informers send first, is served
from the API server’s watch cache in one response, whatever the page size. On our cluster that
means 50,000 pods arriving all at once after 13 s, with nothing on screen until then. KubeDrive
lists at the current version instead, which the server pages: the first page of 500 rows comes
back on its own, the rest follow in pages of 5,000, and the watch picks up from there.
The benchmark caught a mistake of ours here too. The interface redrew the table after every page, and with tens of thousands of rows those redraws delayed the next pages: the full list took 20 to 40 s, while the Go side had it in 4. Now the first page is shown at once and the rest at most every 400 ms while the list loads, and the full list takes 5 s.
Draw only what is on screen
The table renders only the rows in view, so scrolling 50,000 pods costs the same as scrolling 50. Sorting and filtering compute each row’s key once per change rather than in every comparison, which keeps a re-sort of 20,000 rows at around 30 ms.
Run it yourself
These are our numbers, on our machine. If your cluster or laptop tells a different story, we want to hear it: the benchmark scripts build a test cluster of any size and time any client the same way. Or just download KubeDrive and open your largest namespace.