Why Your AI Fine-Tuning Job Runs Longer Than Expected
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Why Your AI Fine-Tuning Job Runs Longer Than Expected on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

Hugging Face reports that a CatBoost model predicted end-to-end runtimes for 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. Its best tested FLOP-based estimate had a 12.41-hour error, but the data came from one production environment and no live scheduling savings were measured.

Hugging Face says a CatBoost model predicted the end-to-end runtime of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The result was substantially more accurate than the best tested FLOP-based estimate in the same dataset, a finding that could help GPU operators plan workloads, though it has not yet been validated across other clusters or in a live scheduling trial.

The jobs ran on shared NVIDIA H200 GPUs in one corporate production environment and used about 7,382 GPU-hours in total. Their durations ranged from 1.6 to 172.4 hours. The dataset included model families up to 15 billion parameters, LoRA and full fine-tuning, varied dataset sizes, and allocations of one, two or four GPUs. The machines were part of a shared pool of 8 to 32 H200s across one to four nodes, with other training and inference workloads running alongside them.

Hugging Face defined duration as wall-clock time from GPU allocation until job completion; time spent waiting in a queue was excluded. The CatBoost model used 24 features selected after testing candidate approaches. Inputs available before dispatch included configuration and model metadata such as evaluation cadence, checkpoint settings, per-GPU token volumes and fine-tuning regime. A FLOP-based runtime estimate was also included as one feature.

In the reported cross-validation, CatBoost had an R² of 0.98, a root mean squared error of 3.52 hours and a median absolute error of 0.51 hours. The FLOP baseline’s best tested setting, assuming 30% Model FLOPs Utilization (MFU), had a 12.41-hour mean absolute error. Errors were higher at 45% MFU, at 13.01 hours, and 60% MFU, at 14.62 hours. The preview says a broader sweep from 25% to 65% MFU did not resolve the baseline’s shortcomings.

At a glance
reportWhen: Study preview published; the full paper…
The developmentHugging Face has previewed a study in which a CatBoost model outperformed tested FLOP-based estimates when predicting the duration of 272 production fine-tuning jobs.
AI Training · Briefing

Why Fine-Tuning Takes Longer Than FLOPs Say

Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.

1.71 hours
CatBoost mean absolute error
12.41 hours
Best tested FLOP baseline error
272
Production jobs evaluated
7,382
Approximate GPU-hours

What the estimates capture

The target is time from GPU allocation to job completion; queue time is excluded.

Compute

The training loop

FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.

Configuration

The whole job

CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.

Shared systems

Other runtime costs

Setup, data handling, network and storage can add time alongside training.

Prediction error in the tested data

Lower error means predicted duration was closer to observed duration on average.

CatBoost1.71 hours
FLOP baseline, 30% MFU12.41 hours
FLOP baseline, 45% MFU13.01 hours
FLOP baseline, 60% MFU14.62 hours

What this could mean for GPU scheduling

More useful duration estimates could help teams plan scarce GPU capacity.

When jobs finish early
  • GPUs may sit idle until another job starts.
  • Operators can misjudge when capacity will be free.
When jobs run over
  • Other work can be delayed.
  • Delivery dates may shift.

How far the result reaches

The preview reports cross-validation on one corporate production environment.

1
Tested on one pool

Runs used shared NVIDIA H200 GPUs and varied configurations.

2
Generalization remains open

Accuracy on other clusters, software stacks and workload mixes is not established.

3
Operational impact remains unmeasured

The preview reports prediction error, not savings in GPU time or scheduling delays.

What to watchThe full paper is forthcoming; validation on independent runs and a live scheduler trial would show whether the result carries over.
The takeaway
Configuration-aware prediction beat the tested FLOP estimates on these jobs, but teams need broader validation before relying on the same accuracy for their own clusters.

Better Estimates Could Ease GPU Scheduling

Runtime estimates inform when queued work is dispatched and how GPU capacity is allocated. If a job finishes earlier than forecast, GPUs can sit idle until another task starts; if it runs longer, it can delay other jobs and shift delivery plans. A more accurate estimate on a cluster’s own workloads could help operators anticipate those mismatches.

The study compares prediction errors, not operational outcomes. It does not report a controlled deployment showing that the model reduced GPU idle time, compute costs or project delays. The size of the reported accuracy gap is notable within this dataset, but it cannot by itself establish the practical savings a scheduler would achieve.

Products Worth Considering

Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Bluetooth 5.3, USB-C Charging, IPX4, Black

Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Bluetooth 5.3, USB-C Charging, IPX4, Black

As an affiliate, we earn on qualifying purchases.

Why FLOP Estimates Miss Job Time

A common estimate approximates dense Transformer training compute using parameter count and training-token count, then divides by the number of GPUs, their peak throughput and an assumed Model FLOPs Utilization. MFU represents the share of theoretical peak throughput reached in practice; the source places typical values in a broad 30% to 60% range. Such calculations focus on training compute and depend on an assumption about hardware efficiency.

The study’s target covers more of the job lifecycle. Model setup and loading, tokenization, evaluation, checkpoint uploads and final storage can add time, and their effects may vary with configuration and shared network or storage capacity. Hugging Face says the researchers included scheduled evaluation tokens in the baseline comparison and tested multiple MFU assumptions, rather than relying on a single setting.

“Everything is known before dispatch.”

— Hugging Face, in its study preview

How Well Results Travel

The findings come from 272 runs at one production site, and five-fold cross-validation on those runs is not the same as testing on independently collected clusters. The preview does not establish accuracy on other GPU types, software stacks, workload mixes or sites. It also does not provide results from a later, unseen time period or a live scheduling trial.

The full paper has not yet appeared, and the preview gives no publication date. Further details about the dataset, evaluation design and feature selection may be available in that paper. It is also unclear how often a deployed predictor would need retraining as configurations and shared-cluster conditions change. The reported metrics describe prediction performance in this dataset; broader reliability and operational impact remain unconfirmed.

Full Paper and External Tests

Hugging Face says it plans to post the full paper on arXiv, but has not announced when. That release may provide additional information needed to evaluate the study’s methods and results.

To determine whether the approach can support wider scheduling decisions, the model would need testing on runs from other periods or environments, followed by evaluation inside an operational scheduler. Such tests could show whether the reported prediction accuracy persists beyond the original production pool and whether improved forecasts translate into fewer idle GPU intervals or delays. Until then, the findings are evidence from one dataset rather than a demonstrated cluster-wide benefit.

Key Questions

What did the Hugging Face study find?

Hugging Face reports that a CatBoost model predicted runtimes for 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The dataset came from one corporate production environment.

How did the model compare with the FLOP-based estimate?

The best tested FLOP-based setting, at 30% MFU, had a 12.41-hour mean absolute error. The CatBoost model’s reported error was 1.71 hours on the same study dataset.

Does the runtime include time waiting in a queue?

No. The study defines runtime as wall-clock time from when a job receives its GPUs until it finishes. Queue time is excluded.

Has the model been shown to save GPU time or money?

The preview reports prediction errors, not a controlled deployment measuring GPU time, cost or delays saved. Those operational benefits have not been established by the reported results.

When will the full paper be available?

Hugging Face says the full version will be posted on arXiv, but no publication date has been provided.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Why Egyptian And Nigerian English Are Now On The Open ASR Leaderboard

Hugging Face added two private Liva AI speech-test splits totaling 2.89 hours, expanding Open ASR Leaderboard coverage to Egyptian and Nigerian English.

Is AI Changing The Way We Work? Here’s What You Need To Know

OpenAI’s recent study reveals AI is crossing occupational boundaries in workplace tasks, potentially altering job roles but not yet confirming job losses or changes.

LinkedIn recruitment spam becomes Olde English prose after user hides AI prompt injection in bio — bots also also manipulated to address user as ‘My Lord’

A LinkedIn user injected an Old English prompt into their profile, causing recruiters’ messages to appear in 900 AD language, highlighting AI manipulation risks.

AI Breakthrough: ByteDance Seedance 2.5 Launches 30-Second Single-Take Story Videos

ByteDance Seedance 2.5 releases a 30-second single-take AI video capable of storytelling, marking a significant step in AI video generation technology.