Source item

AMD reused our code + modified for the benchmarks that show they're faster than Nvidia for training Few major caveats They didn't use pytorch native attention backend They used out of tree attention for MI300, but didn't give NV the same benefit or even the main tree version 1/2 They coulda used main tree or FA3 and achieved way better perf They used an out of tree GQA attention, but didn't enable the pytorch native gqa for Nvdia. Nvidia likely retests with bunch of Megatron and Vboost profiles to show even higher perf 2/2 https://t.co/riIuq6X8Zn