AWS has published a walkthrough for autoscaling Kokoro-82M, an 82-million-parameter text-to-speech model, in a custom container on an Amazon SageMaker AI real-time endpoint. The post says its approach derives scaling settings from two measurements: Q_max, the highest concurrency a single instance can sustain within a stated service-level objective, and T_total, the end-to-end time required for a scale-out operation.

The walkthrough uses Amazon SageMaker AI’s ConcurrentRequestsPerModel metric, which reports a maximum value over a 10-second window. The accompanying tooling compares that metric with concurrency measured by the benchmarking client and converts the results into Amazon CloudWatch units. AWS says the ratio was 1.0 at every concurrency level in the Kokoro-82M run described.

For the reported test, the service-level objective was a 95th-percentile time to first audio byte of 3,000 milliseconds. The endpoint met that objective at concurrency levels from 1 through 50, with a p95 value of 2,805.7 milliseconds at 50. At concurrency 60, p95 time to first audio byte reached 3,374.8 milliseconds and breached the objective, making 50 the reported Q_max.

The post also describes AWS CDK stacks and command-line tooling that measure the endpoint before autoscaling resources are synthesized. The deployment gate requires both scaling settings and a feasible measurement plan artifact. AWS notes that the sample container uses an earlier PyTorch version with a known advisory and recommends reviewing and upgrading that dependency before non-proof-of-value use.