InfraPulse · thermal map

One hall, 24 hours, three questions: where is it hot, what did it cost, what do you do about it?

This is what the InfraPulse report looks like for a 32-rack GPU hall. Scrub the day, switch between temperature, throttling, and delivered throughput, and click a finding to see which racks it touches and what it cost in GPU-hours.

Synthetic data — a modeled hall, not a customer fleet. Real reports use your telemetry and your cost basis.

Hall A · 32 racks · 256 nodes · 2,048 GPUs

Each cell is one rack (8 nodes × 8 GPUs). Hover for detail, click to inspect. Four rows, hot aisles between B/C.

14:00

Rack inspector

Click a rack to see its nodes and 24-hour trace.

Findings

Raised by the agents. Click to highlight. Nothing here acted on its own.

Illustrative on-demand rates; edit to your rental price or amortized capex. Loss in GPU-hours is unchanged — only the dollar line moves.

Where the agents help

Four agents sit on top of the telemetry. They watch, attribute, and recommend; they never touch a scheduler or a clock. Every finding on this page carries who raised it, when, the evidence, and a confidence — and waits for a person.

Watcher

Continuous · per GPU

Learns each GPU's normal from its own history under real load and flags when a device leaves it — before the dashboard threshold, and before the job fails. Drift, not just alarms.

On this hall: raised B3-6 at 05:40 when its ECC slope separated from its seven siblings, eleven hours before anyone would have noticed a slow collective.

Attribution

On every shortfall

Takes a hot or under-delivering cell and lines up the signals — throttle bits, power, temperature, tensor activity, scheduler context — into a cause with a timeline. Separates "hot because working" from "hot because cooling" from "cool because starved."

On this hall: read A1–A4 as an input stall rather than a healthy cool row, because busy was 100% while tensor activity and temperature both said otherwise.

Advisor

When a finding is confirmed

Turns a cause into a specific, reversible next step — what to move, drain, rebalance, or check — priced in GPU-hours and explained. It recommends; it does not execute. Scheduler hints and automation are opt-in, later, per operator.

On this hall: suggested moving C6's two long-running jobs to row D during the afternoon peak and draining B3-6 at its next checkpoint, with the GPU-hours each saves.

Reporter

Daily · and on demand

Writes the account: delivered vs. paid-for GPU-hours, each loss named and priced, what changed since yesterday. Answers questions in plain language against the same data — "why did D2 lose 400 GPU-hours this week?" — with the evidence attached.

On this hall: the four findings and the KPI row above are the Reporter's output for one day; the Path B written analysis is the same thing, produced once from your export.
A person approves every action. Agents propose; operators dispose. There is no write path to Kubernetes, Slurm, DCGM, or a BMC unless an operator turns one on later.
Evidence over confidence. Each finding shows the signals it rests on and a confidence level. A Medium finding says so and asks for confirmation instead of pretending.
Same data boundary as the collector. Agents run on numeric metric samples only — no job code, no payloads, no credentials. What leaves your network is unchanged by adding them.

What the map is actually made of

Every cell rolls up fields your GPUs already emit. The demo models them; the product reads them.

1

Temperature

Per-GPU DCGM_FI_DEV_GPU_TEMP and memory temperature, or amd-smi junction temp, averaged per rack with the hottest GPU tracked separately — because the rack average hides the one device that trips the limit.

2

Throttle reasons

The CLOCK_THROTTLE_REASONS bitmask, decoded: HW thermal slowdown, SW thermal, power cap, sync boost. Throttled % is the share of GPU-minutes with any throughput-costing bit set — not just "temperature was high."

3

Delivered throughput

SM active and tensor active against SKU rating and each GPU's own baseline. This is the number that turns a hot rack into a finding: a rack can be cool and still deliver 55% if it is starved for data.

See this for your hall instead of a modeled one

Send 24 hours of DCGM or amd-smi output and get the real version back as a written analysis. No agent, no access, no NDA.

Send 24h of metrics How InfraPulse works