LMRuntime.com / Public page

Evidence & Benchmarks

Validation posture, benchmark boundaries, and release gates stated without overclaiming readiness.

Public package metadata establishes package availability, target framework, dependencies, and licensing. Generated API pages establish documented contracts. Deterministic fixture evidence establishes only the recorded fixture outcomes. Broader compatibility and performance claims require separately retained evidence.

Works Today / Not Claimed

Works Today

Public NuGet package family
.NET 9 package targets
Local verified GGUF facade on managed CPU
Explicit backend registry and fail-closed probes
Generated API reference
Deterministic fixture evidence

Not Claimed

GPU inference from package installation alone
Broad model compatibility
Benchmark leadership
Model downloads
Provider API access
Production certification
Safety certification

Evidence ladder

Artifact evidence: package identity, target framework, dependencies, assemblies, XML documentation, README, license, and public NuGet availability.
Contract evidence: documented types and members, validation behavior, and ownership boundaries.
Fixture evidence: bounded GGUF, tensor, tokenizer, and generation inputs with recorded expected outputs.
Backend-selection evidence: registration order, declared capabilities, requested runtime identifier, probe diagnostics, selected device, fallback state, and failure reason.
Managed execution evidence: restore, build, tests, cancellation, disposal, and negative cases in a named environment.
Parity evidence: reference versus selected execution paths with exact or declared numerical tolerances.
Real-model compatibility evidence: named artifact identity, architecture, tokenizer, quantization, prompt, output, environment, and limitations.
Performance evidence: reproducible latency, throughput, memory, workload, and measurement protocol.

What deterministic means here

Deterministic greedy selection chooses the highest-ranked token under a fixed logit vector and tie rule. End-to-end reproducibility also depends on exact model bytes, tokenizer behavior, prompt bytes, session state, execution path, floating-point behavior, limits, package graph, and runtime environment.

Backend proof ruleA package reference proves package availability. A capability declaration proves intended surface. A successful probe proves local discovery. Only retained execution evidence proves that model work ran on the named backend and device.

Reference and selected-path parity

The scalar/reference path is the correctness anchor for operations it covers. CPU dispatch, vectorization, intrinsics, half-precision, quantized readers, mapped storage, and future execution paths require operation-specific parity evidence before stronger claims are made.

Tokenizer parityCompare exact token IDs, special-token policy, decoded bytes, invalid-text behavior, templates, and truncation.

Tensor parityCompare descriptor interpretation, byte geometry, dequantized values, row bounds, and ownership.

Kernel parityCompare reference and selected tiers over edge shapes, invalid inputs, and declared tolerances.

Session parityCompare positions, logits, selected tokens, KV-state fingerprints, reset/restore behavior, and stops.

Minimum benchmark contract

Category	Required disclosure
Artifact	Model name, immutable digest, byte count, architecture, quantization/storage types, tokenizer, and license source.
Software	Resolved package graph, application commit, build configuration, runtime settings, and OS/runtime details.
Hardware	CPU model, core/thread policy, memory, power mode, thermal state, and instruction-tier selection.
Workload	Prompt bytes and tokens, context state, requested output length, sampling settings, warm-up, and repetitions.
Metrics	Time to first token, prefill rate, decode rate, elapsed time, peak memory, failures/cancellations, and aggregation method.
Limitations	Unsupported behavior, fallbacks, excluded runs, and cold/warm state.

Roadmap promotion gates

Experiment before capability claimThe experimental roadmap requires a fixed baseline, complete evidence envelope, failure injection, recovery compatibility checks, controller-overhead reporting, negative results, and effective operator termination before any host-control pattern is described as proven.

Claim rules

Supported language“This named artifact produced this recorded result under this declared environment.”

Unsupported inference“Therefore all GGUF models, all LLaMA derivatives, all hardware, or all applications are supported.”

Supported comparison“The selected implementation matched the reference within the declared tolerance for these cases.”

Unsupported comparison“Fastest,” “production certified,” “safe,” or “broadly compatible” without an explicit scope and retained evidence.