Slot utilization up to 90% across a million-core compute farm

AI-driven predictive analytics for a semiconductor leader, forecasting job runtimes and resource needs across 8M jobs a day to improve scheduling efficiency and cut waste.

Outcome

Slot utilization raised to as much as 90% from a range of under 50% to 75%, downtime down 20% through proactive issue detection, and higher throughput with faster time to market.

See the results

Talk to an amp'd specialist

At a glance

Client & industry

Global semiconductor technology leader, testing its products under various scenarios on a massive compute farm

Engagement

AI-powered predictive analytics, from domain understanding and data discovery through exploratory analysis, feature engineering, and ML modeling

Scale

1 million cores, 80 petabytes of storage, and 8 million jobs a day

Maps to below services

Cloud & Platform Engineering, AI & ML Engineering

Where it started

A million-core farm running below capacity, because job runtimes and resource needs were hard to predict.

Business problem

At this scale, resource optimization was hard. Job durations varied widely, which made resource allocation inefficient and left slots underused, and business units ran at utilization levels from under 50% up to 75%.

Technical barriers

CPU, memory, and core requirements were estimated manually for each job, which often meant over-provisioning or under-provisioning resources.

Agivant's role

Data discovery, predictive modeling, anomaly detection, and integration into the client's scheduling and resource allocation systems

What Agivant built

Agivant began with in-depth consultations with the client's experts to understand their environment. Analysis of job data, execution patterns, and infrastructure usage identified the key optimization opportunities and finalized the use cases. Exploratory analysis examined data patterns and anomalies in job behavior and resource use, feature engineering selected and refined the inputs, and machine learning models were built to predict job runtimes, memory needs, and CPU and core requirements. Value in days and weeks.

Job runtime prediction

Models estimate job runtimes from historical data and job characteristics, so the client allocates resources more efficiently and reduces idle time.

Resource requirement forecasting

Models predict memory, CPU, and core requirements for each job, for optimal allocation with minimal waste.

Anomaly detection

AI algorithms detect unusual patterns in job behavior, identifying potential failures early and preventing costly downtime.

Data collection and processing

Pipelines collect, clean, and transform job data from logs, metrics, and job parameters.

Feature engineering

Relevant features extracted from the data improve the accuracy and reliability of the models.

Scheduler integration

Agivant worked closely with the client's engineering teams to integrate the models into existing scheduling and resource allocation systems.

Real-time monitoring

A monitoring system tracks model performance and raises alerts on any anomalies.

Continuous improvement

A feedback loop refines the models based on real-world performance data.

How it is built

Models	Data sources	Integration	Monitoring	Controls
Regression and time series analysis	Job logs, metrics, and job parameters	Existing scheduling and resource allocation systems	Real-time model performance tracking and alerting	Security measures applied to the model to protect sensitive data

Before

Manually estimated resources, unpredictable runtimes, underutilized slots, and resources over- or under-provisioned.

After

Predicted runtimes and resource needs feeding the scheduler, proactive anomaly detection, and models refined on live data.

Proof in production

A job enters the queue and the models estimate its runtime and its memory, CPU, and core needs, so the scheduler allocates what the job requires. If the job's behavior turns unusual, anomaly detection flags it before it becomes a failure. The monitoring system tracks every model's performance, and real-world results feed back to refine the next round of predictions. Across business units, slot utilization now reaches as much as 90%.

Measured results

Increased efficiency with an improved user experience, reduced operational overhead, and higher throughput with faster time to market.

Up to 90%

Slot utilization, from under 50% to 75%

20% down

Downtime, through proactive issue detection

8M

Jobs a day

$5M

Potential annual infrastructure savings from the custom prediction model

Protected

Sensitive data, with security measures applied to the model

The repeatable pattern

Reusable asset

The custom prediction models stay integrated with the client's scheduler and keep improving through the feedback loop on live performance data. Every build amplifies the next.

Challenges overcome

Building a deep understanding of a unique environment at a scale of 1 million cores and 80 petabytes, and predicting runtimes and resource needs for jobs whose durations varied widely.

Enterprise movement

Foundational

Operational

Path to autonomous