[Spencer Libbing (stl327)]

Summary

Publishes an AKS Engineering Blog article about Node Hardening, an AKS preview feature that enforces kube-reserved and system-reserved capacity with cgroups and adds soft eviction on Linux nodes.

This change:

• Adds controlled benchmarks that compare otherwise identical standard and hardened nodes when memory is held near capacity, during a runaway memory leak, and under CPU saturation, with a short methodology summary and limitations.
• Shows measured allocatable-capacity changes across six node sizes and maxPods settings.
• Compares memory withheld before Kubernetes 1.29, on 1.29 and later, and with Node Hardening, and shows NotReady results with kube-reserved memory lowered to 1,077 MiB and to the 750 MiB minimum.
• Notes how hardened nodes behave when pods fully saturate CPU, as an observation rather than a tuning recommendation.
• Sets the publication date to November 3, 2026, and uses the official AKS service icon in the hero image.

Merge timing: Docusaurus publishes future-dated posts immediately, so please merge on November 3, 2026.

Dependency: The post links to https://learn.microsoft.com/azure/aks/node-hardening, which is added by MicrosoftDocs/azure-aks-docs-pr#3940. Please confirm that article is live before merging.

Customer impact

Node Hardening helps customers whose nodes run workloads that exceed their memory requests. Headline results from five trials per scenario on a 16 GiB Standard_D4ads_v5 node (AKS 1.35.8):

• When pods held node memory near capacity, standard nodes went NotReady in 5 of 5 trials (median 298 s NotReady), with no kubelet evictions, and an app on the node failed a median 33% of requests. Hardened nodes stayed Ready in 5 of 5 trials, evicted or contained the pods, and the app failed no requests.
• During a runaway memory leak, the lowest available node memory was 776 MiB on hardened nodes versus 11 MiB on standard nodes (median). The kubelet evicted the leaking pods instead of relying on system-wide kernel OOM kills, and the kubelet and containerd never stalled on memory.
• Hardened nodes reported 4% to 15% less allocatable memory, 100 millicores less allocatable CPU, and 1 GiB less ephemeral storage, depending on node size and maxPods.
• On a node with fully saturated CPU, pod starts took 33.4 s on hardened nodes versus 2.9 s on standard nodes, because the kube-reserved cgroup gets a smaller CPU weight. Memory saturation is more common than CPU saturation in AKS, so the post presents this as something to note, not a recommendation to change kube-reserved CPU.
• Hardened nodes never went NotReady in any scenario. Standard nodes went NotReady only when memory was held near capacity. In the runaway-leak and CPU-saturation scenarios, the well-behaved app kept running on both node types.

Validation

• Built the site locally with docusaurus build. The post rendered with its author, tags, tables, and all five images, and the build reported no new warnings.
• Ran codespell with the repository's CI settings on the new post.

[Copilot]

🟡 Changes recommended

The article contains inaccurate resource-limit guidance, an unreliable feature-registration sequence, and noncompliant front matter lengths.

5 open findings

• CPU limits do not preserve capacity for node services · New
• Wait for feature registration before enabling Node Hardening · New
• Blog title exceeds the 60-character limit · New
• SEO description exceeds the 150–160-character range · New
• Clarify reservations versus enforced cgroup limits · New

What changed in this PR

Adds an AKS Node Hardening benchmark article covering resource protection, capacity costs, and CPU tuning.

Changes:

• Adds benchmark methodology, results, limitations, and setup guidance.
• Adds four supporting benchmark images.
• Registers Spencer Libbing as an author.

File	Description
website/​blog/​authors.yml	Adds author metadata.
website/​blog/​2026-10-23-aks-node-hardening/​index.md	Adds the benchmark article.
hero-image.png	Adds the article hero image.
allocatable-memory.png	Visualizes allocatable-memory changes.
memory-pressure-timeline.png	Visualizes memory-pressure results.
cpu-saturation.png	Visualizes CPU-saturation results.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

The front matter exceeds repository limits, and the preview setup proceeds before asynchronous feature registration completes.

0 open findings

5 resolved since last review

• Wait for feature registration before enabling Node Hardening
• CPU limits do not preserve capacity for node services
• Clarify reservations versus enforced cgroup limits
• SEO description exceeds the 150–160-character range
• Blog title exceeds the 60-character limit

Previously missed (1)

In code that hasn't changed since last review

Wait for feature registration before refreshing provider

website/​blog/​2026-10-23-aks-node-hardening/​index.md:192

Feature registration is asynchronous, so immediately re-registering the provider can happen while CustomNodeConfigPreview is still Registering; the following az aks update can then reject the preview option. Poll until the feature reports Registered before refreshing the provider, as other AKS preview instructions do (website/blog/2026-07-13-runai-streamer-vllm/index.md:119-131).

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

The benchmark summary contains inconsistent statistics, and publication remains gated on the release date and unavailable dependent documentation.

0 open findings

Previously missed (2)

In code that hasn't changed since last review

Summary reports maximum instead of median

website/​blog/​2026-10-23-aks-node-hardening/​index.md:31

The summary says all values are medians, but this row reports “up to 7.1%” while the detailed median table reports 3.2% (line 115). Either use the 3.2% median here or identify 7.1% as a maximum and qualify the statement on line 36 so readers don't interpret two different values as the same statistic.

Reference points to unpublished dependency documentation

website/​blog/​2026-10-23-aks-node-hardening/​index.md:47

This dependency isn't currently publicly discoverable at the target URL, and the PR description says it is supplied by MicrosoftDocs/azure-aks-docs-pr#3940. Confirm that article is live before merging; otherwise this first reference sends readers to unavailable documentation for the feature defaults.

This issue also appears in the following locations of the same file:

• line 214
• line 225

🧠 Review effort: Balanced

[Spencer Libbing (stl327)]

Really nice work on this one. The opening hook lands, the NotReady results make the case clearly, and the limitations section keeps it honest.

A few things outside the inline comments:

• Merge conflict (blocking): #5934 merged on October 20 and already added your spencer-libbing entry to website/blog/authors.yml, so this PR now conflicts with master. Rebasing and dropping the authors.yml change should clear it.
• PR description: The merge-timing note still says October 23. Update it once you pick the new date.
• Terminology: "rolling out deployments" in the CPU section refers to the Kubernetes resource, so capitalize it as "Deployments".

The inline comments cover the specific spots.

[Copilot]

🔵 Needs a closer look

Publication requires owner confirmation of the November 3 merge timing and the Microsoft Learn dependency, which could not be verified.

1 open issue

• Correct PR summary to reflect standard-node NotReady results · New

_{Review effort: Balanced}

[Sercan Guler (seguler)]

Can we flip this in customers' voice? It sounds like we're frustrated with customers raising support tickets.

[Sercan Guler (seguler)]

One of the most frustrating issues for engineers is to deal with noisy neighbor and saturated resources...

[Sercan Guler (seguler)]

Is this true? Previously I heard from customers that memory hungry apps actually spike in memory usage and that kubelet eviction handler which works in cycles cannot catch up with it?

[Sercan Guler (seguler)]

Don't we already have hard eviction and kube-reserved for standard nodes? I think part of this is not true. Kubelet likely just didn't have the cgroups slice to protect itself and actually act on hard eviction of 100MB

[Sercan Guler (seguler)]

This table seems Standard nodes are better? Are the columns mixed up?

[Sercan Guler (seguler)]

ah now I understand. I wonder if it would be better to lead with explanation. You could say because we're prioritizing kubelet under cpu contention, we expect the nodes to stay functioning, but the new container startsup and application performance will slightly regress. Is that right?
(also I wondered if customers will actually prefer the opposite - well functioning app and not functioning node)

[Sercan Guler (seguler)]

This is just for us. But I am worried that we moved too much into reliability and threw out the cost optimization aspect here. How do we know that we are not back to what we had before 1.29? Are we being conservative in our reservations? How do we prove that this is the right setting?

This may even be a good thing to prove in this blog. Can we show the impact of just dropping 300MB from kube-reserved and its impact on node readiness?

[Spencer Libbing (stl327)]

Good question. I added two things to the post to address this:

1. How we compare to earlier defaults ("How this compares to earlier AKS defaults"). With 30 max pods, Node Hardening withholds 1.8–3.3 GiB (23% to 5% of node memory) vs. 2.6–6.2 GiB (32% to 10%) before 1.29 and 750 MiB (9% to 1%) on 1.29 and later. So we're at roughly half of the pre-1.29 cost, not back to it, and unlike pre-1.29 the reservation is enforced with cgroups.

2. Your experiment ("Lowering kube-reserved memory"). I reran the memory-held-near-capacity scenario on hardened 16 GiB nodes (AKS 1.35.9), 5 trials each:

Kube-reserved memory	Withheld from pods	NotReady trials	Lowest available memory (median)	Lowest kubelet + containerd page cache (median)
1,377 MiB (default)	2,052 MiB	0 of 5	667 MiB	261 MiB
1,077 MiB (−300 MiB)	1,752 MiB	0 of 5	368 MiB	171 MiB
750 MiB (API minimum)	1,425 MiB	0 of 5	124 MiB	59 MiB

No NotReady, no kubelet restarts, and 0% failed app requests in any variant. Takeaways:

• Enforcement is what prevents NotReady, not reservation size. Once pods are capped and the kubelet/containerd have their own slice, even 750 MiB held up in this scenario. (Standard nodes went NotReady 5/5 with 650 MiB unenforced for the kubelet.)
• Size sets the margin. Kubelet + containerd anonymous memory peaked at ~250 MiB in every variant; the rest of the slice is page cache. At 750 MiB, that cache was squeezed to ~60 MiB and the node had ~125 MiB available, which leaves little room for higher pod density, image pulls, or churn (our node ran well under its 30-pod max).
• So yes, the default is conservative, intentionally, since it scales with maxPods. Cutting 300 MiB still left healthy headroom here, which suggests there's room to tune the per-pod coefficient down if we want to recover capacity. A follow-up at higher pod density (for example, 110 or 250 maxPods with churn) would tell us how far.

[Copilot]

🔵 Needs a closer look

The linked documentation PR returns 404 and enablement guidance conflicts, requiring maintainer confirmation of preview prerequisites, CLI commands, and lifecycle behavior before the planned November 3 merge.

No open issues 🎉

1 resolved since last review

• Correct PR summary to reflect standard-node NotReady results

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Publication requires maintainer coordination for the November 3 merge because Docusaurus publishes future-dated posts immediately and the required Node Hardening documentation remains unverified.

No open issues 🎉

_{Review effort: Balanced}