OWL Measurement Methodology

How OWL measures the energy cost of compute tasks — and what it doesn’t measure.

Scope

OWL measures what happens inside one machine when it performs a real task. This is intentionally narrow. The energy cost of streaming is distributed across data centres, networks, and consumer devices — each with different measurement challenges and attribution problems. We start with the layer we can measure directly, at the wall, with no modelling assumptions.

This scoping decision means OWL results are not lifecycle assessments and should not be cited as total-cost-of-delivery figures. They answer a specific question: how much additional energy does this server draw to perform this task, above its idle baseline?

One deliberate extension exists: since 2026-07 the findings catalog also carries client-device decode panels measured on a separate rig under the same protocol; each states its own device scope (see Test Types).

Measurement Principle

OWL uses wall-power delta measurement: the difference between what the server draws at idle and what it draws under load, captured by an external smart plug.

OWL is the bench half of GoS’s dual-track methodology: its sister programme REM (Remote Energy Measurement) observes real devices in the field at fleet scale — where effects exist — while OWL quantifies why, and by how much, under controlled conditions. Both tracks share one instrument principle, documented by GoS’s LEM (Local Energy Measurement) programme: real hardware running real workloads, measured externally at the wall by a smart plug read over the local network — milliwatt-resolution readings at second-scale cadence, no software estimation.

Measurement Protocol

Every test in OWL — video, LLM, image generation, RAG — follows the same core protocol:

1. Focus mode. Suppress background system tasks (apt, cron, man-db, fwupd, etc.) that would introduce energy noise. Managed via systemctl stop with dedicated sudoers rules.
2. Model unload (LLM/RAG only). Send keep_alive=0 to Ollama and wait 3 seconds for GPU memory release. Ensures a cold start when cold-inference mode is selected.
3. Baseline capture. Poll the Tapo P110 at 1-second intervals on each of two staggered meters for a configurable period (currently 5 polls — configurable in Settings). The mean of these readings becomes W_base — the server’s idle power draw.
4. Lock. Acquire /tmp/gos-measure.lock to prevent concurrent measurements from overlapping. A FIFO queue manages waiting jobs.
5. Execute task. Run the actual workload (ffmpeg, Ollama inference, SD-Turbo diffusion) while continuing to poll the Tapo P110 at 1-second intervals on each of two staggered meters. Thermal sensors (CPU Tctl, GPU junction, GPU PPT) are read in parallel.
6. Compute energy. Calculate delta power, total energy, and per-unit metrics (see formulas below).
7. Persist. Write the full result to a JSON file — parameters, energy report, raw poll data, thermal readings, confidence flag. Every result is reproducible and exportable.
8. Focus exit. Restart suppressed system timers in parallel (via ThreadPoolExecutor) to minimise downtime.

Between sequential runs (e.g., CPU vs GPU comparison), a configurable cooldown (currently 10 seconds — configurable in Settings) allows the system to return to thermal equilibrium.

Energy Calculation

All formulas use wall-power from the P110 (system-level), not component-level readings. The GPU’s self-reported power (its vendor sensor — amdgpu PPT or nvidia-smi power draw) is captured for reference but is not used in the primary energy calculation — it covers only the GPU die/board, not the full system delta (CPU, RAM, drives, fans, PSU losses).

Isolating the encoder — transcode vs encode

Every video figure above is the energy of a full transcode — ffmpeg decodes the source, converts colour space, scales, then encodes — not the encoder in isolation. For most comparisons that is the honest number (you cannot encode without first decoding), and when the input is held constant the decode cost is a near-constant offset that cancels out of the comparison. Where the encoder’s own share is wanted — currently in the REM file-prep flow (/prepare-rem) — OWL runs a second, decode-only pass under the identical protocol (the same source decoded and discarded to a null sink, no encode) and subtracts it:

All three figures — transcode, decode, and the derived encode — are reported side by side, so the attribution is shown rather than asserted.

Confidence Framework

Every OWL result carries a traffic-light confidence flag. Under the CR-028 Phase 2 model (designed with Tania Pouli), the flag answers one defensible question per run: can this run be distinguished from idle? It is a per-run confidence interval, not a fixed-watt rule of thumb.

We keep the raw per-poll power samples from both the baseline window and the task window, form a standard error on the measured power increase ΔW, then convert ΔW into a one-sided confidence that the task really draws above idle:

The confidence framework follows GoS’s broader principle: if it can’t be measured, it shouldn’t be asserted. A 🔴 result is not a failure — it’s an honest signal that the measurement instrument isn’t sensitive enough for that task. Publishing it transparently is more useful than hiding it.

Calibration integrity

The variance calibration runner (/variance/run) executes 30 pairs of H.264 CPU + H.265 GPU encodes with 50 seconds between them, and computes three coefficients of variation: idle (raw P110 baseline readings, captures system noise), CPU (run-to-run reproducibility of the CPU encode ΔW), GPU (same for GPU). Their mean becomes variance_pct.

The runner is hardened against silent encode failures: every ffmpeg invocation’s exit code is checked, only successful encodes contribute ΔW, and per-side failure counters are tracked. If ≥50% of either side fails, the runner refuses to update settings — the result JSON is still returned (with cpu_failed, gpu_failed, failure_stderr, abort_reason fields) for forensics, but variance_pct stays unchanged on disk. This protects against the failure mode where partial-encode ΔW values contaminate the calibration without the operator noticing.

Diagnostics & Pre-calibration

Two layers of measurement-discipline tooling sit alongside the calibration:

Thermal-recovery probe

Before trusting a calibration result, the system needs to know that variance_cooldown_s is long enough — the idle samples taken between encodes must come from a thermally recovered system, not from the tail of the previous workload. The bin/probe-thermal-recovery diagnostic characterises this empirically. For a sequence of distances d after each of a CPU and a GPU encode (defaults: 0, 2, 5, 8, 12, 18, 25, 35, 50, 70, 95, 120 seconds), the probe samples idle power for 8 polls and writes the mean / std / CV to a CSV under results/diagnostics/.

On the GoS1 hardware the recovery is fast (see chart above): post-CPU and post-GPU baselines converge to the settled idle floor by d = 5–8 s with within-window CV around 1–2.5%. So the configured cooldown of 50 seconds is comfortably more than necessary — useful as a margin, not as a correction.

The same curve is also on the Settings page (lab access), where it refreshes live from the probe endpoint. Each probe run overwrites nothing — it leaves a fresh timestamped CSV pair under results/diagnostics/ so historical curves can be diffed if hardware or thermal conditions change.

Why the probe matters

The probe was the seam that exposed the scale_vaapi leak (the GPU encode failed within 90 seconds of starting the diagnostic) and the silent-failure path in the calibration loop. Generalisable lesson: measurement code should fail loudly, not interpolate around brokenness. It now runs queue-aware from the Settings page (/precalibration/run, lab access) as well as the CLI.

Hardware Disclosure

All results are tied to specific hardware. Different CPUs, GPUs, RAM configurations, and PSU efficiencies will produce different numbers. OWL results should always be cited with their hardware context.

Test Types

Video transcoding

Transcode a source file (default: Netflix Meridian 4K, CC BY 4.0) to a target codec and 1080p. Measures the energy cost of the full encode pipeline — decode, colour-space conversion, scale, encode. Supports CPU vs GPU comparison: both paths are run sequentially with a cooldown between them, and results are presented side by side.

Six presets across three codecs: H.264 (libx264 / h264_nvenc, 4000 kbps), H.265 (libx265 / hevc_nvenc, 2000 kbps), AV1 (libsvtav1 / av1_nvenc, 1500 kbps). A seventh Compare all codecs preset runs all six in sequence and produces a cross-codec energy matrix. (Encoder names track the installed GPU — the live list is in the Hardware Disclosure table above.)

All presets use ABR (Average Bit Rate) rate control at a shared per-codec bitrate target, so CPU and GPU receive the identical encoding task — output file sizes match across devices as confirmation. All GPU presets use the full hardware pipeline: hardware decode (-hwaccel cuda) + scale_cuda + hardware encode, with frames GPU-resident throughout. This represents real live-encoding workflows (Harmonic, Ateme); an earlier partial pipeline (CPU decode + GPU encode) has been replaced because it was unrepresentative and bottlenecked on CPU decode overhead.

The ffmpeg command used for each run is logged in the result JSON, editable from the page (signed-in GoS members and lab access), and reproduced in the result card for full transparency.

Perceptual quality (VMAF). Comparison runs (CPU vs GPU, or all codecs) also report VMAF — Netflix’s perceptual quality metric (0–100, higher is better) — so the energy figures sit next to a quality figure rather than an unstated assumption that the encodes are equivalent. It is computed at the delivered 1080p, comparing each encoded output against the source downscaled to 1080p (the distorted side is cropped to strip hardware-encoder padding, never upscaled). VMAF runs after the measurement window closes, so its compute cost is excluded from the reported energy. It is a quality cross-check, not a primary GoS measurement.

VMAF model version — v1 since 2026-07-17. OWL scores with Netflix’s VMAF v1 (model vmaf_v1.0.16, released 2026-06-20 with libvmaf 3.2.0). v1 drops the VIF feature, adds a banding detector (CAMBI) and VMAF’s first chroma (colour) feature, and corrects v0’s over-prediction on high-motion content — failure modes that sit exactly where OWL’s tests live (starvation-bitrate encodes band; sports content is high-motion). Every stored score carries its model identity (vmaf_model); results without that field predate 2026-07-17 and were all scored by vmaf_v0.6.1 — they remain valid v0 scores. The two scales are not comparable (the same degraded-vs-clean 1080p pair scored 77.95 on v0 and 83.59 on v1 on this bench), so result cards label every score v0 or v1 and comparisons are only ever made within one model. Scoring runs through a dedicated newer ffmpeg build (needed for libvmaf ≥3.2.0); the pinned encode binary is untouched, so the model upgrade cannot confound energy measurements. Measured scoring cost on this bench is within a few percent of v0 either way — removing VIF roughly pays for CAMBI + chroma.

No-reference quality (CompressedVQA-HDR). Enhancement / super-resolution runs on uploaded or general content have no ground-truth reference (the AI adds detail the source never had), so full-reference metrics do not apply there. Instead, OWL reports a no-reference score from CompressedVQA-HDR (Sun et al., arXiv:2507.11900, Apache 2.0 — winner of the ICME 2025 HDR/SDR VQA grand challenge), a learned model that scores each file independently and handles both HDR10 and SDR content. Like VMAF it runs after the measurement window closes, so its compute cost is excluded from the reported energy. Being a learned opinion of perceptual quality rather than a measurement, it is presented as a relative indicator within a run — never as an absolute quality claim — and is subject to further refinement pending validation.

Full-reference fidelity on ladder fixtures — two axes, read together. For the ten degraded-ladder fixtures OWL generated from its own pristine 4K masters (Big Buck Bunny and Meridian), a ground truth does exist, and enhancement runs on them additionally report full-reference VMAF (v1, 4K model) against that master — computed after the measurement window closes, like all quality scoring. Fidelity and perceptual quality are different axes, and OWL reports both deliberately: the no-reference score rates how good the output looks on its own; the full-reference score measures similarity to the reference, not perceived quality in the broader sense. Enhanced output is rarely one thing or the other: in practice it mixes genuinely recovered structure, reconstruction from learned priors, and newly synthesised detail, and the synthesised share can look convincing without matching the master’s exact pixels. Enhancement therefore typically raises perceived quality while moving the signal further from the master, and an output scoring below the naive-upscale anchor on fidelity while beating it on the no-reference axis is an expected signature of generative enhancement — not a defect of the enhancer or the metric, and not evidence that the output looks worse; the divergence between the two axes is itself the measurement. Two anchors give the fidelity score context, both paying the same pipeline encode as the measured output: a plain lanczos upscale of the degraded source (naive-encode anchor — what dumb scaling preserves) and the pristine master itself through the same pipeline (pristine-encode anchor — its gap to 100 is encode cost alone, landing around 89–91 on these fixtures). The degraded source scored as a player would display it (no pipeline encode) is shown as a separate path, never compared directly. Full-reference scoring requires an SDR 4K output — the anchors are 4K-denominated, and VMAF across HDR/SDR transfer functions is not meaningful — so other output targets, uploads, and non-ladder sources stay no-reference only.

Contributed technology. Some measured workloads use technology contributed by GoS member organisations — the AI video-enhancement harness measures Pixop Live, contributed by Pixop. Contributed technologies run on GoS hardware under this methodology; results are energy data, not endorsements. GoS members can contribute streaming technologies for measurement on the same terms.

4K HDR enhancement — reduced-capacity note. The HDR → 4K super-resolution combo sits close to the GPU’s memory ceiling on this hardware (peak ~94 % of 16 GB). To run it reliably, OWL applies two Pixop-supplied pipeline settings that lower the in-flight memory pressure (a smaller input buffer and fewer concurrent super-resolution threads), which bring the peak down to ~85 %. A controlled A/B on identical content found this changed the measured energy and encode throughput by less than the run-to-run noise — so the figure is reported normally, with the settings recorded in the result JSON. This note applies only to the 4K HDR combo; every other enhancement preset runs at the pipeline’s default settings.

AI workloads — beta, exploratory

Video transcoding is OWL’s core benchmark. Three AI workloads run alongside it on the same protocol and confidence framework, but they are explicitly beta — useful for relative comparisons, with headline numbers still being hardened (see Open Questions). In brief:

• LLM inference — mWh/token across a model ladder (TinyLlama 1.1B, Qwen3 1.7B/4B/8B, Mistral-NeMo 12B, Phi-4 14B, up to GPT-OSS 20B), cold or warm, CPU or GPU, with an optional batch mode. Prompts are saved in the result JSON; output streams word-by-word as live-run proof.
• Image generation — Wh/image across the distilled diffusion panel (~0.6B–3.5B), CPU or GPU, with a Compare-Models mode that fixes prompt, seed and resolution so the model is the only variable.
• RAG — the energy delta of retrieval: baseline (no retrieval) vs RAG with 3 context chunks vs 8, retrieved from a document corpus via ChromaDB + sentence-transformer embeddings, compared side by side.

Framing (GoS Language Lab position paper, Jan 2026): AI in streaming is neither inherently sustainable nor unsustainable — type, size and deployment decide net impact. The type matters enormously: streaming leans on small specialised CNNs (per-title encoding, scene classification, super-resolution) that are orders of magnitude cheaper than the general-purpose LLMs and diffusion models these tabs measure as an upper bound. OWL measures the energy AI adds (inference only); it does not measure the infrastructure energy AI avoids through better compression, caching or routing — both halves are needed for net impact, and OWL has the first. Each AI result is also shown as a multiple of a real video encode (the pinned canonical H.265 GPU encode of Meridian-120s) so the number stays anchored to a streaming workload rather than floating free. Full framing: Language Lab AI position paper →.

Client-device decode panels — external rig, 2026-07

Encode energy is paid once per title; decode energy is paid per viewer — so in 2026-07 the protocol went to the client side. A portable rig (a Google TV set-top box; Raspberry Pi 5 and Pi 400) measured decode power per codec, per decode path (hardware vs software) and per regime (paced playback vs flat-out), with the same baseline/ΔW/confidence method and local-mW metering. Headlines: a hardware decoder cuts decode power ~3.6–4× on the same board; codec choice is nearly free on decode silicon but moves software decode power by up to ~60%. Full results, replication counts and caveats are in the findings catalog; these panels state their own device scope and are not GoS1-server results.

Energy budget & encode parity

Operators rarely ask “how small can the file be.” They fix a quality target — most often VMAF 92 — and then try to hit that quality for the least energy. The transcode budget calculator answers the inverse: given an energy budget, how many minutes of video can you push at your target VMAF, on which hardware, with which codec?

That calculator runs on a measured calibration table, built under the same protocol as every other OWL test. For a fixed source we sweep an ABR bitrate ladder across each codec (H.264 / H.265 / AV1) on both the CPU encoder (libx264 / libx265 / libsvtav1) and the GPU encoder (NVENC), measuring wall energy and — as a terminal pass — VMAF. For a chosen target VMAF we read the bitrate that hits it off the measured curve, and the watt-hours per minute that bitrate costs.

Model note: the current calibration table and the VMAF 92 operator target were measured in vmaf_v0.6.1 terms (the model in force at calibration, 2026-06). Live scoring moved to VMAF v1 in 2026-07; because the scales are not comparable, the budget page stays in v0 terms — and says so — until the next re-calibration re-scores the table under v1.

Encode parity — is the GPU really “worse”?

Operators often say hardware encoders score lower than software, “especially for AV1.” To test that fairly we measure the GPU twice at each bitrate — once with OWL’s current NVENC arguments (baseline) and once with a quality-knob bundle (tuned: -preset p7 -multipass 2 -spatial-aq -temporal-aq -rc-lookahead, B-frame references) — so the VMAF difference, and its energy cost, are measured rather than asserted. What the first full run (90 encodes, all 🟢) showed:

• Energy: NVENC uses 2.5–4.4× less energy per minute of video than the CPU encoder — and the win is speed, not lower draw: instantaneous wattage is similar (~70 W either way); the GPU simply finishes far sooner.
• Parity: the “GPU is worse, especially AV1” effect is real but only on low-complexity content at low bitrate (AV1 on Big Buck Bunny trailed by up to ~9 VMAF at 1 Mbps). On high-complexity content (Meridian) the gap nearly vanishes — and NVENC AV1 actually beats libsvtav1 at its default preset at mid-to-high bitrate.
• Tuning, measured and rejected: the “tuned” bundle cost 1.6–2.8× the energy and lowered VMAF for H.264 and AV1 (adaptive quantisation trades a fidelity metric like VMAF for perceptual quality). So the live encode path keeps the baseline NVENC config; we do not pay energy to make the metric worse.

What an “ABR ladder” means here

A real delivery encodes the same source at several resolutions so a player can adapt to bandwidth. The calculator’s “full ABR ladder” unit is a 5-rung ladder: 1080p (the quality-anchor rung, whose bitrate is set by the VMAF target) plus fixed lower rungs at 720p / 540p / 480p / 360p (lower rungs scale per codec). Ladder energy is the sum of all five rungs; the “1080p only” unit is the top rung alone.

The calibration is keyed by a hardware fingerprint (GPU, CPU, ffmpeg version) and is re-runnable from the Lab when the encode hardware changes — e.g. when dedicated ASIC/FPGA transcode cards arrive — so a hardware swap produces a new dataset rather than silently reusing stale numbers. Full method note: docs/encode_parity_calibration_2026-06.md.

Known Limitations

From energy to CO₂e — for reference only

OWL is a power meter, not a carbon calculator. The number OWL produces and stands behind is energy — watts at the wall and watt-hours per task, measured directly by the P110. Everything else on this page is about getting that energy number right. We lead with power because it is what we can measure at the wall with no modelling assumptions; carbon is one modelling layer removed.

Every result also carries a gCO₂e figure, but only as a downstream convenience: we multiply the measured energy by a grid carbon-intensity factor (Wh × gCO₂e/kWh) so the energy can be read against everyday activities. That makes it a reference estimate, never a GoS measurement. Carbon attribution — allocation, boundaries, double-counting, marginal vs average intensity — is a hard problem that GoS deliberately leaves to the bodies whose job it is. This follows the GoS principle directly: “if it can’t be measured, it shouldn’t be asserted” — and what OWL measures directly is energy. Read the energy figure as the result; the CO₂e is a footnote.

🟢 Direct = the energy figure (P110 polling at the wall, validated method, GoS primary measurement — this is what we cite). 🟡 Indicative = the gCO₂e figure (Wh × third-party grid intensity — context, not citable as GoS data). Vocabulary follows the Greening of Streaming Language Lab AI position paper (Jan 2026), which proposes this 🟢/🟡/🔴 traffic-light for the entire ICT-energy-measurement landscape and rates IEA top-down energy figures as 🟡 Amber. OWL applies the same framework to its own outputs — every result-card carbon block carries the 🟡 chip; the energy headline retains the green palette.

For what it’s worth, the intensity used is lifecycle-basis (IPCC AR6 factors): the live French grid mix via Eco2mix when reachable, ElectricityMaps as a backup, and Ember annual country means as the fallback (also used for the stable comparison cities). The value and which source produced it are recorded in every result JSON and CSV export (CSV header carries a leading comment marking the carbon columns indicative). A result’s carbon dropdown also shows the same energy on a few past French grids for context. Module status — live cache, source, age, fallback — is at /carbon.

Open Questions

These are questions OWL has surfaced but not yet answered. They are published here in the interest of transparency.