We’ve scaled Kubernetes clusters to 7,500 nodes, generating a scalable infrastructure for massive fashions like GPT-3, CLIP, and DALL·E, but in addition for fast small-scale iterative analysis akin to Scaling Regulations for Neural Language Fashions.
