Source item

Today we are launching InferenceMAX! We have support from Nvidia, AMD, OpenAI, Microsoft, Pytorch, SGLang, vLLM, Oracle, CoreWeave, TogetherAI, Nebius, Crusoe, HPE, SuperMicro, Dell It runs every day on the latest software (vLLM, SGLang, etc) across hundreds of GPUs, $10Ms of infrastructure is purring every day to create real world LLM Inference benchmarks InferenceMAX answers the major questions of our times with AI Infrastructure. How many Tokens are generated per MW of capacity on different infrastructure? How much does a million tokes cost? What is the real latency vs throughput tradeoff? We have coverage of over 80% of deployed FLOPS globally by covering H100, H200, B200, GB200, MI300X, MI325X, and MI355X. Soon we will be over 99% with Google TPUs and Amazon Trainium being added. All results and such can be accessed at https://t.co/GtYxufjO4P And the code and everything is open sourced here https://t.co/9IyL5oYu8p Methodology and explanation of results are here https://t.co/9IW5Mx31p3 To be clear this was an immense lift from the technical staff at SemiAnalysis. They are the shining stars here. They did something that I think will reshape how people make infrastructure purchasing decisions. Also huge thank you to Nvidia and AMD as well, and they engineers countless hours of effort.