NVIDIA Releases AIPerf for High-Concurrency LLM Testing

NVIDIA has launched AIPerf, a ground-up rewrite and successor to its GenAI-Perf tool. Unlike its predecessor, which ran on top of Perf Analyzer and suffered from Python's global interpreter lock under high concurrency, AIPerf uses a multiprocess architecture. It distributes workloads across worker processes while separate record-processor services handle results, coordinating everything over ZMQ. This design ensures the benchmarking client itself does not become a bottleneck during high-volume testing.

The tool supports more than 15 endpoint types, including chat, completions, NIM rankings, and image generation. To simulate realistic production workloads, engineers can use public datasets like ShareGPT or replay trace formats from Mooncake, Baseten, and WEKA AgentX. AIPerf allows users to shape traffic using constant, Poisson, or gamma arrival distributions with tunable burstiness, gradual ramping, and synthetic distributions like the vLLM/SGLang range-ratio for variable input and output sequence lengths.

AIPerf measures critical performance indicators, including time to first token (TTFT), inter-token latency (ITL), end-to-end request latency, and output token throughput. It reports these metrics with detailed percentile breakdowns from p25 to p99, alongside averages and standard deviations. When integrated with DCGM or pynvml, the tool also captures GPU telemetry, such as power draw, utilization, and memory consumption, allowing developers to correlate latency spikes with hardware events.

In a test environment, practitioners can deploy a small model like Qwen3-0.6B on vLLM to establish a measurement loop. For static benchmarks, flags can pin workloads to exactly 128 input and 128 output tokens. For dynamic scenarios, engineers can configure a Poisson arrival pattern with an average request rate of 10 per second, introducing variance like a 128-token standard deviation around a 512-token mean input. This flexibility helps teams move beyond informal scripting to gather reproducible, production-grade performance data.