Compare Runtime Categories

Key takeaways

  • Begin every comparison with a scope statement and layer map.
  • Compare candidates only within a reasonably similar responsibility boundary or compare complete stacks explicitly.
  • Normalize model, tokenizer, precision, hardware, input/output, concurrency, cache, and metric definitions.
  • Features are meaningful only with exact version, configuration, limitations, and official evidence.
  • Performance comparisons require controlled experiments and quality gates.
  • When comparable evidence is unavailable, publish a qualitative trade-off matrix—not a ranking.

Runtime boundary

A useful architecture identifies what this layer receives, owns, emits, measures, and refuses to own. That boundary prevents overlapping products from being treated as interchangeable.

Receives

Comparison question, candidate versions, layer/category, official documentation, controlled workload, quality/SLO gates, and operational requirements.

Owns

Comparison fairness, category matching, disclosure, evidence quality, and bounded conclusions.

Emits

Scope statement, normalized evidence matrix, controlled results, trade-offs, limitations, and decision relevance.

Does not own

Popularity rankings, fabricated scores, or combining unrelated public benchmarks.

Failure modes

Category mismatch, stale version, unequal model/precision, vendor-claim aggregation, hidden configuration, and leaderboard language.

Evidence and metrics

Evidence completeness, comparable fields, controlled benchmark coverage, quality parity, unresolved unknowns, and review date.

Architecture diagrams

Each diagram has an equivalent text description and remains readable in light, dark, print, keyboard, and no-JavaScript contexts.

Runtime category boundary map

Runtime category boundary mapA boundary map places compiler and graph runtimes, inference engines, model servers, gateways and routers, distributed runtimes, workflow engines, agent frameworks, agentic application runtimes, and product applications in overlapping responsibility regions. The horizontal axis moves from model artifact execution to business work. The vertical axis moves from stateless request handling to durable state and governed side effects. Overlaps are labeled secondary categories rather than forced single-category assignments.
:root{color-scheme:light dark}.bg{fill:#f7f8fa;stroke:#273142}.node{fill:#fff;stroke:#273142;stroke-width:2}.accent{fill:#e8eef7;stroke:#273142;stroke-width:2}.text{font:600 17px system-ui,sans-serif;fill:#121820}.small{font:500 14px system-ui,sans-serif;fill:#273142}.line{stroke:#273142;stroke-width:2;fill:none;marker-end:url(#arrow)}.dash{stroke-dasharray:7 6}@media(prefers-color-scheme:dark){.bg{fill:#151922;stroke:#d6dbe5}.node{fill:#202632;stroke:#d6dbe5}.accent{fill:#29364a;stroke:#d6dbe5}.text{fill:#f5f7fb}.small{fill:#d6dbe5}.line{stroke:#d6dbe5}}@media print{.bg,.node,.accent{fill:#fff}.text,.small{fill:#000}.bg,.node,.accent,.line{stroke:#000}}
Model-artifact execution → governed business workStateless handling → durable state and side effectsCompilerInference engineModel serverGatewayDistributed runtimeWorkflow engineAgent frameworkAgentic runtimeProduct application
Categories describe responsibilities, not marketing labels; adjacent products can overlap without becoming interchangeable.
Text description

A boundary map places compiler and graph runtimes, inference engines, model servers, gateways and routers, distributed runtimes, workflow engines, agent frameworks, agentic application runtimes, and product applications in overlapping responsibility regions. The horizontal axis moves from model artifact execution to business work. The vertical axis moves from stateless request handling to durable state and governed side effects. Overlaps are labeled secondary categories rather than forced single-category assignments.

A product spanning multiple runtime layers

A product spanning multiple runtime layersAn unnamed illustrative platform spans an inference engine, model-serving API, distributed scheduler, gateway, tool broker, policy service, and product workflow. Solid boxes identify primary responsibilities and outlined boxes identify integrated dependencies. Contract boundaries separate model execution from request governance. The example deliberately avoids a vendor name and does not imply that every product spans these layers.
:root{color-scheme:light dark}.bg{fill:#f7f8fa;stroke:#273142}.node{fill:#fff;stroke:#273142;stroke-width:2}.accent{fill:#e8eef7;stroke:#273142;stroke-width:2}.text{font:600 17px system-ui,sans-serif;fill:#121820}.small{font:500 14px system-ui,sans-serif;fill:#273142}.line{stroke:#273142;stroke-width:2;fill:none;marker-end:url(#arrow)}.dash{stroke-dasharray:7 6}@media(prefers-color-scheme:dark){.bg{fill:#151922;stroke:#d6dbe5}.node{fill:#202632;stroke:#d6dbe5}.accent{fill:#29364a;stroke:#d6dbe5}.text{fill:#f5f7fb}.small{fill:#d6dbe5}.line{stroke:#d6dbe5}}@media print{.bg,.node,.accent{fill:#fff}.text,.small{fill:#000}.bg,.node,.accent,.line{stroke:#000}}
Product workflowPolicy and tool brokerGateway and routingServing and schedulingInference engineExternal compiler and hardware dependencies
An illustrative platform can implement several layers while each responsibility remains separately testable and replaceable.
Text description

An unnamed illustrative platform spans an inference engine, model-serving API, distributed scheduler, gateway, tool broker, policy service, and product workflow. Solid boxes identify primary responsibilities and outlined boxes identify integrated dependencies. Contract boundaries separate model execution from request governance. The example deliberately avoids a vendor name and does not imply that every product spans these layers.

Comparison scope

State the decision, exact candidates/versions, runtime layers, deployment, model, hardware, workload, and excluded responsibilities.

Implementation

Name whether the comparison is engine-only, model-server, serving platform, edge runtime, browser runtime, or complete stack.

Operational implications

vLLM versus SGLang can be an engine/serving comparison; Triton versus vLLM requires a scoped explanation because responsibilities overlap differently.

Measure

Scope completeness, versions, layer match, and excluded components.

Category and boundary matching

Use the taxonomy to distinguish compiler/graph runtime, inference engine, model server, platform, local runtime, edge/browser runtime, and agentic infrastructure.

Implementation

Map each candidate to primary and secondary responsibilities and list required external components.

Operational implications

Do not penalize a focused engine for lacking platform features unless the decision requires a complete platform.

Measure

Boundary coverage, external components, integration count, and mismatch flags.

Feature evidence

For each feature, cite current official docs and record version, status, configuration, limits, and support level.

Implementation

Distinguish stable, preview, experimental, deprecated, and third-party integration.

Operational implications

A checked feature box without constraints is not evidence.

Measure

Source type/date, version coverage, limitations, and unsupported/unknown fields.

Performance control

Use the same artifact, tokenizer, precision, hardware/topology, software environment, workload distribution, warmup/cache, client method, and quality gate.

Implementation

Publish raw results and methodology. Use Goodput and tail latency.

Operational implications

Never combine unrelated vendor numbers into one controlled chart.

Measure

TTFT/TPOT/E2E/Goodput/errors/memory/quality and variance.

Operational comparison

Compare installation, artifact workflow, readiness, scaling, rollout, observability, failure recovery, upgrades, security, licensing, governance, and team capability.

Implementation

Exercise load/unload, canary, overload, node loss, retry, cancellation, and rollback.

Operational implications

An engine that wins a kernel test may create more operational cost.

Measure

Time to ready, recovery, upgrade effort, trace completeness, incident toil, and total cost.

Qualitative trade-off matrix

When controlled performance data is unavailable, describe design philosophy, strengths, constraints, portability, maturity, and best-fit workloads.

Implementation

Mark unknowns explicitly and avoid turning prose into a disguised numeric leaderboard.

Operational implications

The matrix should help readers design their own proof.

Measure

Unknown count, evidence level, review date, and decision questions.

Review and correction

Comparisons age quickly as projects release features and deprecate paths.

Implementation

Display last reviewed UTC, record source versions, schedule review, and provide a correction route.

Operational implications

Correct stale claims transparently rather than silently rewriting history.

Measure

Review age, broken links, correction time, and version drift.

Topic

Implementation

Operational implications

Measure

Reference tables

Valid comparison examples
Comparison Valid scope Required caveat
vLLM vs SGLang vs TensorRT-LLM LLM inference/serving engines Exact versions, model, hardware, features
ONNX Runtime vs OpenVINO Portable graph inference on specified hardware Execution-provider/delegation overlap
Triton vs KServe Serving components/platform responsibilities They can be combined, not pure substitutes
ExecuTorch vs LiteRT On-device deployment stack Model export and delegate ecosystems differ
WebGPU vs WebNN Browser execution API paths Not products; implementation support varies
LangGraph vs a model server Invalid direct comparison Different layers and responsibilities
Comparison evidence levels
Level Evidence
A — controlled Reproducible same-environment experiment with raw data
B — official verified Current primary documentation or repository evidence
C — independent scoped Reputable analysis with disclosed method
D — anecdotal Community report; discovery only
Unknown No sufficient evidence; explicitly marked

Decision checklist

  1. What exact decision and layer does the comparison address?
  2. Are candidates actually substitutes at that layer?
  3. Which external components complete each stack?
  4. Are model, precision, hardware, workload, and quality controlled?
  5. Which feature claims are official and versioned?
  6. How do operations, security, portability, and licensing differ?
  7. What remains unknown or untested?
  8. When will the page be reviewed again?

Common mistakes

  • Comparing a model server to a compiler as direct substitutes.
  • Using current “latest” labels without version numbers.
  • Combining vendor benchmarks with different models/hardware.
  • Checking features without recording limits or status.
  • Ignoring quality and error rate.
  • Ranking by popularity or stars.
  • Publishing a winner with no decision context.
  • Failing to update or correct stale claims.

Sources and further reading


  1. ONNX Runtime architecture
    (opens in a new tab)

    ONNX Runtime · Official documentation · reviewed 2026-06-21 UTC

  2. Triton architecture
    (opens in a new tab)

    NVIDIA · Official documentation · reviewed 2026-06-21 UTC

  3. KServe ServingRuntime
    (opens in a new tab)

    KServe · Official documentation · reviewed 2026-06-21 UTC

  4. MLPerf Inference
    (opens in a new tab)

    MLCommons · Benchmark specification · reviewed 2026-06-21 UTC

Last reviewed: 2026-06-23 UTC


Keep the source visible

Help others find this page

If this page was useful, link to it from a website, blog, public newsletter archive, resource page, documentation page, or social post. A public link helps the work remain discoverable after the feed moves on.

Get sharing text
Learn why links matter