Weird argument they’re making, since distillation doesn’t give you the AI. It just gives you the style of the AI.
If you distilled Google’s newest Gemini into a GPT-2 model, all it would do is just sound like it. Your GPT-2 model isn’t suddenly going to be as capable as the model you distilled.
It can speed up training and reduce the amount of training data needed. Well, at least if you have access to the output vector, which I guess you don’t over the API. I think most “distilled” models you see for download don’t do full backprop training, because it takes enormous amounts of compute, and instead use low-rank methods. The low rank methods really only change the style. Alibaba has the compute to do proper training.
Weird argument they’re making, since distillation doesn’t give you the AI. It just gives you the style of the AI.
If you distilled Google’s newest Gemini into a GPT-2 model, all it would do is just sound like it. Your GPT-2 model isn’t suddenly going to be as capable as the model you distilled.
It can speed up training and reduce the amount of training data needed. Well, at least if you have access to the output vector, which I guess you don’t over the API. I think most “distilled” models you see for download don’t do full backprop training, because it takes enormous amounts of compute, and instead use low-rank methods. The low rank methods really only change the style. Alibaba has the compute to do proper training.