“AI infrastructure” is not one thing. Most Australian enterprises running AI in production are running four distinct workload shapes at once, and the infrastructure profile each one demands has almost nothing in common with the others. Sizing a facility against the wrong one is the most expensive mistake available in this category, in both directions: buying training-grade capacity for an inference estate, or trying to serve production traffic from a cluster designed to run one long job.
Here is what each workload actually does to a rack.
Pre-training: one long synchronous job
Pre-training builds a foundation model from scratch. It is the workload that produces the headline numbers, and almost no enterprise does it.
The defining characteristic is not scale, it is synchrony and duration. Meta’s Llama 3 405B run used 16,384 NVIDIA H100 80GB GPUs over 54 days. Every GPU in a job of that shape steps forward together, exchanges gradients, and waits for the slowest participant. That produces three infrastructure consequences.
Power moves in lockstep. Meta reported that tens of thousands of GPUs raising or lowering consumption simultaneously, for example at checkpoint boundaries or job start and stop, produced instant fluctuations across the data centre on the order of tens of megawatts. Diversity factors that work on a mixed enterprise floor do not apply.
Interruption is expensive. A run that loses power or overheats resumes from its last checkpoint, so the cost of a facility fault is measured in GPU-days.
The interconnect is the bottleneck, not the GPU. Gradient synchronisation across nodes needs high bandwidth and low latency at the same time, and a fabric that stalls turns a 16,000-GPU cluster into an expensive queue.
Facility profile: the highest density racks you can get, liquid cooling, concurrent maintainability so a 54-day job survives a UPS service window, and an interconnect fabric specified before the GPUs are.
Fine-tuning: shorter, smaller, and far more common
Fine-tuning adapts an existing open-weight model to your domain, your tone or your task. This is where most enterprise AI work actually happens, and the resource profile is different by orders of magnitude.
Full fine-tuning updates every parameter and carries close to training-scale memory requirements. Parameter-efficient methods do not. The LoRA technique freezes the pre-trained weights and injects small trainable rank-decomposition matrices into each transformer layer. Compared with fine-tuning GPT-3 175B using Adam, the authors reported reducing trainable parameters by a factor of 10,000 and GPU memory requirements by a factor of three, with a checkpoint at rank 4 dropping from 350GB to 35MB.
That changes the procurement question. A fine-tuning workload is bursty: a handful of GPUs for hours or days, repeated whenever the domain data changes, rather than a standing cluster. It suits reserved GPU capacity you can schedule against rather than a dedicated pod, and the operational requirement shifts towards fast provisioning and a place to store many small adapters against a small number of base models.
Facility profile: moderate density, scheduling and queueing that actually works, and storage designed for versioned artefacts rather than raw throughput.
Inference: two workloads wearing one name
Inference is what users touch, and it is the workload most often sized incorrectly, because it is really two phases with opposite characteristics.
The prefill phase processes the entire input prompt at once and builds the key-value cache. It is compute-bound and tends to saturate the GPU. Its user-visible metric is time to first token.
The decode phase then generates one token at a time, reading the model weights and the growing KV cache out of high-bandwidth memory at every step. It is memory-bandwidth-bound rather than compute-bound. Its metric is inter-token latency, sometimes expressed as time per output token.
The practical consequence is that GPU utilisation is a poor capacity signal for an inference fleet. A decode-heavy serving pattern can show modest compute utilisation while being completely bandwidth-saturated, and adding compute will not help it. KV cache growth is the other constraint people meet late: the cache scales with context length and concurrency, and it can exhaust GPU memory long after the model weights themselves have comfortably fit.
Inference is also the workload where physical distance stops being an abstraction. Every request and response makes a round trip, so a model served from offshore infrastructure adds oceanic cable latency to every interaction, on top of whatever the model itself costs.
Facility profile: lower per-rack density than training, higher availability expectations, capacity planned against memory bandwidth and KV cache headroom rather than FLOPS, and a network path measured to Australian population centres.
Retrieval: the workload planned last
Retrieval-augmented generation is now the default enterprise pattern, and its infrastructure profile is the one most often left out of the sizing exercise, because most of it is not GPU work at all.
A retrieval pipeline has four stages with four different resource shapes. Ingestion and chunking is CPU and storage throughput. Embedding generation is GPU work, but short and batchable rather than sustained. Index storage and search is memory-resident and latency-sensitive: a vector index of any size wants RAM and fast local NVMe, not accelerators. Reranking is a small model doing frequent short inferences.
Add to that the part nobody budgets for: re-embedding. Change your embedding model, your chunking strategy or your document corpus, and the whole index is rebuilt. That is a periodic GPU burst against a system that otherwise looks like a database.
Retrieval also concentrates the compliance question. The retrieval index contains your source documents in vectorised form, and the retrieved passages are placed directly into prompts. Wherever that index lives and wherever those prompts are processed is where your source material is being handled.
Facility profile: general compute and memory rather than GPU density, fast local storage, and the same jurisdictional treatment as the source data itself.
Matching the profile before you buy the rack
Most enterprise AI estates end up running three of these four concurrently: occasional fine-tuning, continuous inference, and a retrieval layer underneath both. Pre-training is rare, and treating it as the design case produces a facility specification that is expensive in exactly the places your workload does not use.
A useful sequence is to size the retrieval layer against your corpus, size inference against peak concurrency and context length rather than average request volume, and treat fine-tuning as scheduled burst capacity rather than a standing allocation. Then check the facility against whichever of those has the highest density and the tightest continuity requirement, rather than against the largest number in the vendor deck.
The workloads are different. The infrastructure should be too.
Related reading: Inside an AI-ready data centre: what makes it different and Migrating AI workloads to cloud in Australia.
Frequently asked questions
What is the difference between training and inference infrastructure? Training is a long synchronous job that stresses rack density, cooling and the inter-GPU interconnect, and tolerates being interrupted only at checkpoint boundaries. Inference is continuous, latency-sensitive traffic that stresses memory bandwidth and GPU memory capacity more than raw compute, and needs a short network path to users. The same GPUs can do both; the surrounding infrastructure is specified differently.
Do I need GPUs for retrieval-augmented generation? Only in bursts. Embedding generation and reranking use GPUs, but the bulk of a retrieval pipeline is CPU, RAM and fast local storage, because vector search is a memory-bound database problem. The GPU requirement spikes when you rebuild the index after changing the embedding model, chunking strategy or corpus.
Why is GPU utilisation a poor capacity metric for inference? Because the decode phase of generation is bound by memory bandwidth rather than compute. A serving fleet can show unremarkable compute utilisation while being fully bandwidth-saturated, in which case adding GPUs of the same type buys throughput you cannot use. Track time to first token, inter-token latency and KV cache headroom instead.
Is fine-tuning cheaper than training a model from scratch? By a very large margin, and parameter-efficient methods widen the gap further. LoRA reports around 10,000 times fewer trainable parameters and roughly three times lower GPU memory than full fine-tuning of GPT-3 175B with Adam. For most enterprise use cases the practical choice is between fine-tuning an open-weight model and retrieval against an unchanged one, not between fine-tuning and pre-training.