The operations stories enterprise IT leaders actually read.
Volume 342, Issue 5 · 8 articles
A playbook explains how production AI degrades without visible errors and what evaluation, observability and governance practices catch it.
Production AI can appear healthy, with good availability, latency and error rates, while its outputs become less accurate because data, context or user behavior has changed, and the article cites research finding temporal degradation in 91% of machine learning models studied.
Common causes of silent failure include data and concept drift, stale or incorrect retrieval context, tests that share the implementation's blind spots, and degraded dependencies.
The article argues that evaluation, behavioral observability, guardrails and governance form the missing layer between a working AI feature and a production-ready system.
A survey of 500 SOC workers finds AI boosts job satisfaction and skills, though many fear fewer entry-level roles and over-reliance.
A Swimlane survey of 500 US and UK security workers found 62% say AI helps them build skills, with most reporting higher job satisfaction and AI cutting time spent investigating known threat patterns, triaging alerts and drafting remediation recommendations.
However, 47% expect SOC analyst jobs to become harder to get, and about one in five worry their teams will over-rely on AI.
Roughly half would trust their own instincts over conflicting AI output, underscoring that analysts are shifting toward validating evidence and making higher-impact decisions.
A five-part framework for checking AI readiness across performance, power and cooling, security, cost strategy and scalability.
The article gives IT leaders five checks to run before committing AI budgets, starting with whether servers, storage and networks work together at AI densities, since a GPU cluster can be throttled by storage throughput or thermal limits, and whether power and cooling can scale together, because a 5 kW rack design will not serve a 20 kW AI node.
It also urges treating layered physical, network and access security, plus AI-specific risks such as data leakage, as foundational, and aligning finance and engineering early on whether the organization needs predictable or flexible spending, using FinOps guardrails.
Finally, scalability should be built into the design from the start.
A $1.07M New Jersey penalty over unpermitted gas generators shows the environmental compliance risks of on-site data center power.
New Jersey regulators fined an AI data center operator $1.07 million for running 62 natural gas generators without air permits, showing that operators turning to behind-the-meter generation to get around grid constraints take on new obligations around air emissions, noise and permitting.
Forrester's Abhijit Sunil notes that easing pressure on the grid does not remove the environmental cost and recommends project assessments that cover grid upgrades, water, land use, emissions and noise alongside economic benefits.
IDC survey data shows operators rank waste heat reuse and water reduction near the bottom of their sustainability priorities.
Facing growing community pushback, data center developers are leaning on transparency, grid funding and local benefits.
Data Center Knowledge's Drew Robb reports that local opposition to AI data centers over water, power, noise and rising electricity rates has grown into a cacophony, with some governments imposing bans or moratoria, and sessions at Data Center World Power in Dallas offered tactics for winning support.
Developers are funding battery storage, transmission upgrades, transformers and substations so costs don't fall on ratepayers, and some utilities are formalizing such arrangements.
Hecate Energy's Diane Sullivan urges early transparency and retiring the NDA playbook, and developers are prioritizing sites with supportive residents.
Redundant cooling equipment can still fail together when pumps and CRAHs share a single controller, network path or sensor.
Cooling plants that look N+1 on a mechanical schedule can still behave as a single failure domain when redundant pumps, CRAHs, chillers or CDUs depend on one supervisory controller, network switch, control power source or sensor, so the author separates mechanical redundancy from operational independence and asks how much of the plant one fault can disable.
The recommendation is to design degraded-mode behavior on purpose, with local PLCs and unit controllers keeping basic circulation running, holding setpoints or fixed speeds, when BMS or supervisory communication is lost.
The issue is growing as liquid cooling adds more control and communication layers, and fallback behavior should be commissioned and tested rather than discovered during an incident.
Supercapacitors can absorb millisecond AI load spikes, but experts see them complementing rather than replacing UPS systems and batteries.
AI training creates sharp load swings that conventional power architectures struggle to absorb, with transients reaching more than 150% of a power shelf's rated capacity, and supercapacitors offer the high power density and millisecond response suited to voltage stabilization, peak shaving and short-term backup.
IDTechEx describes three types (EDLCs, lithium-ion hybrids and pseudocapacitors) and projects the data center market reaching $950 million by 2037.
Backup runtime is short, around 2 minutes for EDLCs and up to 10 for some hybrids, so batteries and UPS systems still carry sustained loads and rack designers are keeping slots flexible rather than committing to one approach.
Multi-region architectures can still fail through shared control-plane dependencies, so resilience must be tested, not assumed.
In an InfoQ article, Alexey Golev argues high availability can mask a lack of resilience when failures fall outside design assumptions, describing a TLS 1.3 upgrade that silently broke Route 53 health checks requiring TLS 1.2, causing a CDN to drop an otherwise healthy region.
Multi-region designs can still share dependencies such as DNS health checks, IAM and routing.
Golev warns against 'performative resilience' where recovery is assumed, urging explicit ownership, recurring failover testing and maintained recovery procedures to expose hidden dependencies.
More from Operations · All IT section newsletters · This week