- Service code
- BH-MNT-04
- Family
- Maintenance
- Cadence
- Continuous link telemetry, monthly optical inspection, quarterly topology and configuration audit
- Response
- Fabric degradation acknowledged in 30 minutes; degraded link isolated within 2 hours
Overview
Distributed training turns the network into a compute component. Collective communication patterns mean one degraded optical link can serialise an entire job, and the usual symptom is a mysterious drop in model-FLOPs utilisation with no interface errors worth investigating. This product maintains the fabric as a performance-critical system: optical health, cabling discipline, topology verification and congestion behaviour.
What is covered
- Optical transceiver health: Rx/Tx power levels, bias current, error counters, and replacement before marginal links fail
- Fibre and copper plant inspection: cleanliness, bend radius, connector grading, patch field discipline and labelling
- Topology verification against design intent — rail-optimised wiring, leaf-spine balance, NVLink domain integrity
- Congestion and buffer telemetry: pause frames, ECN marking, PFC deadlock detection, incast hotspots
- Fabric configuration review: routing, QoS, adaptive routing, SHARP or equivalent in-network reduction settings
- Switch health: fan and PSU status, ASIC temperature, firmware currency, control-plane stability
- Latency baseline and regression tracking across the fabric
Deliverables
- Fabric health register with per-link optical margin and error trend
- Topology as-built verification against as-designed, with deviations itemised
- Congestion report identifying hotspots and recommended mitigation
- End-to-end latency baseline with regression alerts
- Cabling remediation plan where patch-field hygiene is degrading maintainability
Who this is for
Multi-node training estates where model-FLOPs utilisation has plateaued below expectation without an obvious cause.
Specification
- Link speeds
- 100G, 200G, 400G, 800G Ethernet and InfiniBand NDR
- Optical reach
- Copper DAC, AOC, multimode SR, single-mode DR/FR/LR
- Fabric types
- InfiniBand, RoCEv2, lossless Ethernet, NVLink domains
- Telemetry
- Streaming counters, optical DDM, congestion and buffer occupancy
- Verification
- As-built vs as-designed diff, rail-optimisation check
Related services
| Service | Scope |
|---|---|
| Accelerator Fleet Lifecycle Maintenance | Node-level servicing for accelerator fleets, measured against the metric that matters: good GPU-hours delivered. |
| Predictive Thermal & Cooling Maintenance | Keep high-density AI racks inside their thermal envelope before throttling costs you throughput. |