- Service code
- BH-MNT-03
- Family
- Maintenance
- Cadence
- Continuous attestation, weekly health review, monthly on-site servicing window
- Response
- Faulty node isolated within 2 hours of detection; replacement node staged within 1 business day where spares permit
Overview
An accelerator fleet fails in ways that general compute does not. A single degraded link in an NVLink domain, one marginal memory row, or a baseboard management controller with a stuck sensor can turn an eight-way node into a straggler that slows an entire job. This product runs the node lifecycle: burn-in, health attestation, predictive replacement of wear items, fault isolation, and retirement — with every action tied back to delivered throughput rather than ticket count.
What is covered
- Burn-in and acceptance testing for new nodes, including full-fleet synchronised stress before production handover
- Health attestation per node: memory error rates, ECC counters, link error counters, thermal and power telemetry
- Straggler detection across training jobs, with correlation to specific node or link faults
- Predictive replacement of wear items: fans, PSUs, cables, transceivers, cold plates and quick-disconnects
- Firmware and driver currency management across BIOS, BMC, accelerator VBIOS, NIC and switch firmware
- Fault isolation and RMA management, including evidence packaging for vendor claims
- Spares pool sizing based on observed failure rates rather than vendor defaults
Deliverables
- Fleet health register: per-node status, error counters and trend since commissioning
- Good-GPU-hours report — delivered capacity against theoretical maximum, with losses attributed
- Failure analysis summaries with root cause classification and recurrence tracking
- Spares recommendation with observed MTBF evidence
- Firmware baseline matrix and drift report
Who this is for
Teams whose training throughput is gated by a small number of degraded nodes that general monitoring does not surface.
Specification
- Accelerator families
- Contemporary data centre GPUs and custom hyperscaler ASICs
- Node scales
- Single node through multi-thousand-node clusters
- Attestation coverage
- Memory, interconnect, storage path, thermal, power, BMC sensor validity
- Interconnect
- NVLink domains, InfiniBand NDR, RoCEv2, Ethernet 400/800G
- Primary metric
- Good GPU-hours delivered per scheduled GPU-hour
Related services
| Service | Scope |
|---|---|
| Network Fabric Integrity | Optics, cabling and fabric configuration maintained so collective operations stop being your bottleneck. |
| Workload Characterisation Lab | Establish what your workload actually is, so every downstream sizing decision stops being a guess. |