ByteDance's founder refuses distillation shortcut in AI race

The founder of ByteDance, Zhang Yiming, has said distillation will not be used to train the company’s AI models. This differentiates the company from other Chinese AI labs that have been accused of using the distillation technique.

This comes as US officials have accused Chinese labs of using the technique to bridge the gap with American AI models. Washington officials have gone as far as threatening sanctions.

Zhang draws a line when everyone else is cutting corners

The Information report states Zhang Yiming is vehemently against distillation in training the company’s AI models. Zhang has decided to be more hands-on in ByteDance’s AI ambitions and has met with senior researchers to get first-hand knowledge, while keeping tabs on the company’s recruitment efforts.

This is important because it lands at a critical time in the AI race. Just this same month, US officials have upped the ante in their fight with Chinese labs over the practice of distillation. ByteDance also owns Seedance, a profitable and performance-leading video model, and a rare feat amongst Chinese AI products.

Why Washington frowns at distillation

Distillation is simply the training of weaker AI models on the responses of stronger models. This ensures that the smaller AI systems learn to imitate the larger system. Distillation is done openly and happens to be a routine part of the AI industry. It is considered legal, though it is usually done on a small scale.

Michael Kratsios, who serves as director of the White House Office of Science and Technology Policy has alleged that Moonshot AI distilled Anthropic’s Fable model when building Kimi K3, its AI system.

He stated this in an X post that Moonshot ran the operation via a purpose-built internal platform, which rotated access routes in a bid to evade detection and used NVIDIA GB300 servers in Thailand, which are barred from being exported to China by US export rules. 

Anthropic reportedly traced 3.4 million Claude conversations to Moonshot via fake accounts. Moonshot is yet to respond to these allegations.

How ByteDance built Seedance, its leading AI model

ByteDance had its turning point in a totally different way. The company took a gamble on Seedance 2.0 despite internal wrangling over the model’s high cost and aggressive scale. The video model was built to scale past the 200 billion parameters mark, but was considered too ambitious by some within the company.

The move panned out as the models’ gross margin hovered around 70-90%, after the company spent big on GPUs, data and world-class recruitment.

Washington says the AI gap is closing

The battle over distillation is the tip of the iceberg. Scott Bessent, the Treasury Secretary has floated placing sanctions if and when Chinese companies adopt distillation on an industrial scale. A bipartisan House bill would make it easy to identify and sanction distillation attempts.

The numbers have prompted urgent action. Despite outspending China on AI by a 23 to one ratio, the US has lost its lead over China in the AI race. As per Stanford’s Institute for Human-Centered AI, the top models from both countries are pretty much the same. 

If you're reading this, you’re already ahead. Stay there with our newsletter.