How GLM Built Its Own Inference Infrastructure: A Deep Dive for Backend Engineers

Most engineers reading about “AI inference infrastructure” skip straight past it, assuming it’s ML research territory. It isn’t. GLM’s writeup on building its own inference stack is fundamentally a distributed systems and backend engineering story — request batching, connection pooling, load balancing, caching layers, and horizontal scaling under brutal latency constraints. If you build APIs in Go or Node.js, or you’ve fought with Docker and Kubernetes to keep p99 latency sane, this is your domain. Let’s break down what GLM actually did and how the same patterns apply to systems you build every day.

Why Inference Infrastructure Is a Backend Problem, Not Just an ML Problem

Serving a large language model at scale looks less like “running a model” and more like operating a high-throughput, stateful API gateway with wildly uneven request costs. A prompt with 50 tokens and one with 8,000 tokens hit your compute budget completely differently — there’s no fixed-cost request like a typical REST endpoint.

GLM’s core challenge mirrors what you’d face building any large-scale service:

• Variable request cost — similar to file uploads vs. simple GET requests, but the variance is 100x-1000x
• Stateful sessions — KV cache persistence across a conversation, similar to session affinity in load balancers
• Batching under latency pressure — grouping requests without blowing SLAs, like Kafka consumer batching
• Heterogeneous hardware pools — GPUs of different generations, similar to mixed EC2 instance fleets

If you’ve tuned a Node.js connection pool or written a custom Go load balancer, you already have the mental model. The complexity is in the constraints, not the concepts.

The Core Architecture: Prefill/Decode Separation

The single biggest architectural decision GLM made — and one that’s become industry standard — is splitting inference into two distinct phases handled by separate compute pools:

Prefill Phase

Processing the input prompt. This is compute-bound and highly parallelizable — think of it like a batch job that chews through all input tokens at once to build the initial KV cache.

Decode Phase

Generating output tokens one at a time, autoregressively. This is memory-bandwidth-bound, not compute-bound. Each step depends on the previous token, so it can’t be parallelized the same way.

Request Lifecycle:
  Client → API Gateway → Router
                            │
                ┌───────────┴───────────┐
                ▼                       ▼
         Prefill Pool             Decode Pool
      (compute-optimized)    (memory-bandwidth-optimized)
                │                       │
                └──────► KV Cache ◄─────┘
                      (shared/transferred)

Why does this matter to you as a backend engineer? Because it’s the same pattern as separating your write path from your read path in a database architecture — different bottlenecks require different scaling strategies. GLM scales prefill and decode pools independently based on their respective load characteristics, exactly how you’d separate a CPU-bound image resizing service from an I/O-bound API gateway.

Request Batching: The Real Performance Lever

Here’s where it gets interesting for anyone who has written a batching layer in Node.js or Go. GLM implements continuous batching, which dynamically adds and removes requests from an in-flight batch instead of waiting for a fixed batch window.

A naive batching implementation (the kind you’d reach for first) looks like this:

// Naive fixed-window batching - what NOT to do at scale
class NaiveBatcher {
  constructor(batchSize = 32, maxWaitMs = 50) {
    this.queue = [];
    this.batchSize = batchSize;
    this.maxWaitMs = maxWaitMs;
  }

  async enqueue(request) {
    return new Promise((resolve) => {
      this.queue.push({ request, resolve });
      if (this.queue.length >= this.batchSize) {
        this.flush();
      } else if (this.queue.length === 1) {
        setTimeout(() => this.flush(), this.maxWaitMs);
      }
    });
  }

  flush() {
    const batch = this.queue.splice(0, this.batchSize);
    processBatch(batch); // one-shot, blocks until all done
  }
}

The problem: a single long-running request in the batch holds up every other request until it finishes. GLM’s continuous batching approach instead treats each generation step as an insertion/eviction point — a request that finishes early frees its slot immediately for a new one, rather than waiting for the whole batch to complete.

// Simplified continuous batching scheduler in Go
type InferenceSlot struct {
    RequestID string
    KVCache   *KVCacheHandle
    Done      bool
}

type Scheduler struct {
    mu       sync.Mutex
    slots    []*InferenceSlot
    maxSlots int
    pending  chan *Request
}

func (s *Scheduler) Step() {
    s.mu.Lock()
    defer s.mu.Unlock()

    // Evict completed slots, backfill from pending queue
    for i, slot := range s.slots {
        if slot.Done {
            select {
            case next := <-s.pending:
                s.slots[i] = newSlot(next)
            default:
                s.slots[i] = nil
            }
        }
    }
    s.runDecodeStepOnActiveSlots()
}

This is directly analogous to how a well-tuned Go worker pool should reclaim goroutines the instant a task finishes rather than draining fixed-size batches — the difference between throughput-optimized and latency-optimized systems.

Caching: KV Cache Is Just a Specialized LRU

GLM’s infrastructure leans heavily on KV cache reuse — avoiding recomputation of the prefill phase when a conversation continues with the same prefix. If you’ve built a caching layer with Redis, this concept transfers almost directly.

Traditional Web Cache	LLM KV Cache
Keyed by URL/query hash	Keyed by token prefix hash
Stores rendered response	Stores attention key/value tensors
Evicted via LRU/TTL	Evicted via LRU + memory pressure
Lives in Redis/Memcached	Lives in GPU HBM, offloaded to CPU RAM/disk
Cache hit = skip DB query	Cache hit = skip prefill compute
Invalidated on data change	Invalidated when conversation diverges

GLM implements prefix caching — if two requests share the same system prompt or conversation history, the KV cache for that shared prefix is reused instead of recomputed. This is exactly like caching a partially-rendered template fragment.

// Conceptual prefix cache lookup — same pattern as HTTP cache middleware
function getCachedPrefix(conversationHistory, cacheStore) {
  const prefixHash = hashTokens(conversationHistory);
  const cached = cacheStore.get(prefixHash);

  if (cached) {
    return { kvCache: cached.kvCache, startToken: cached.length };
  }
  return { kvCache: null, startToken: 0 };
}

The eviction policy GLM describes — combining recency with memory pressure across GPU tiers — is functionally an LRU cache with tiered storage, the same pattern CDNs use for hot/warm/cold object storage.

Load Balancing Across a Heterogeneous GPU Fleet

Standard round-robin load balancing fails badly for inference because requests aren’t uniform cost. GLM’s routing layer factors in:

• Current KV cache occupancy per node
• Queue depth per node
• Request’s estimated context length
• Whether the node already holds a cached prefix for this session

This is essentially consistent hashing with load awareness — similar to what you’d build with a custom nginx upstream selector or a service mesh’s weighted routing, except the weights are computed dynamically from real-time GPU memory state rather than static config.

# Analogous concept in traditional infra: least-connections + sticky sessions
upstream inference_pool {
    least_conn;
    server node1.internal:8080 max_fails=3;
    server node2.internal:8080 max_fails=3;
    sticky cookie srv_id expires=1h;
}

GLM’s router does this at a finer grain — per-request, factoring in cache locality — which is closer to how Kubernetes’ topology-aware routing or a custom gRPC client-side load balancer with health-aware weights would behave.

Networking: Why gRPC and Custom Protocols Win

For inter-node communication (transferring KV cache between prefill and decode pools), GLM avoids REST/JSON entirely. The overhead of JSON serialization for multi-gigabyte tensor transfers would be disastrous. Instead, this space consistently uses:

• RDMA / InfiniBand for GPU-to-GPU transfer where available
• gRPC with protobuf for control-plane messages (scheduling, health checks)
• Custom binary protocols for KV cache serialization

If you’re building Node.js or Go microservices and still default to JSON REST for internal service-to-service communication, this is the reminder that protocol choice matters as data volume grows. The same principle that makes protobuf preferable to JSON for a 50KB payload becomes an absolute requirement at gigabyte scale.

// gRPC service definition style used for control-plane in inference systems
service SchedulerService {
  rpc RegisterNode(NodeInfo) returns (RegisterAck);
  rpc RequestSlot(SlotRequest) returns (SlotAssignment);
  rpc ReportHealth(HealthPing) returns (HealthAck);
}

message SlotAssignment {
  string node_id = 1;
  int32 slot_index = 2;
  bytes kv_cache_handle = 3;
}

Autoscaling: Beyond CPU/Memory Metrics

Standard Kubernetes HPA scales on CPU/memory utilization. That metric is nearly useless for inference workloads, where GPU memory occupancy and queue depth matter far more. GLM’s infrastructure uses custom metrics for scaling decisions — a pattern directly portable to any Docker/Kubernetes deployment running specialized workloads.

# Kubernetes HPA with custom metrics (Prometheus adapter)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-decode-pool
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: decode-pool
  minReplicas: 4
  maxReplicas: 64
  metrics:
    - type: Pods
      pods:
        metric:
          name: gpu_memory_occupancy_percent
        target:
          type: AverageValue
          averageValue: "80"
    - type: Pods
      pods:
        metric:
          name: request_queue_depth
        target:
          type: AverageValue
          averageValue: "10"

This is the exact pattern you’d use scaling a Node.js service on event loop lag instead of CPU percentage, or scaling a Go service on channel backlog depth instead of generic memory usage. Custom metrics beat generic resource metrics whenever your bottleneck isn’t the generic resource.

Comparison: Inference Infra Patterns vs. Standard Backend Patterns

Inference Infra Concept	Standard Backend Equivalent
Prefill/decode separation	Read/write path separation
Continuous batching	Dynamic worker pool with slot reclaim
KV cache / prefix caching	Redis/CDN caching with LRU eviction
Cache-aware load balancing	Consistent hashing + sticky sessions
GPU memory occupancy scaling	Custom-metric HPA (queue depth, lag)
RDMA/gRPC binary transfer	Protobuf over gRPC for internal RPC
Node health/queue depth routing	Weighted least-connections LB
Speculative decoding	Optimistic concurrency / prefetching

Applying These Patterns Without Building an LLM Stack

You don’t need GPUs to benefit from this. Any system with high request-cost variance benefits from the same architecture:

// Example: applying prefill/decode-style separation to a document processing API
// "Prefill" = parsing/OCR (compute-bound, parallelizable)
// "Decode" = incremental summarization (sequential, memory-bound)

async function processDocument(doc) {
  const parsed = await parsePool.submit(doc);       // scales on CPU
  const summary = await summarizePool.submit(parsed); // scales on model calls/sec
  return summary;
}

The mental shift is: stop treating your service as one monolithic scaling unit. Profile which phase of a request is bottlenecked by what resource, split it into its own pool, and scale each pool on the metric that actually reflects its bottleneck.

Key Takeaways

• Prefill/decode separation is a general pattern — split compute-bound and memory-bound phases of any pipeline into independently scaled pools.
• Continuous batching beats fixed-window batching whenever request duration varies significantly; reclaim slots the instant they free up.
• KV cache is architecturally identical to LRU + tiered storage caching you already know from Redis/CDN design — prefix hashing enables reuse across requests.
• Load balancing under variable request cost needs real-time state awareness (queue depth, cache locality), not simple round-robin.
• Internal service communication should move to gRPC/protobuf or binary protocols once payload size or frequency makes JSON overhead measurable.
• Autoscale on the metric