It can speed up training and reduce the amount of training data needed. Well, at least if you have access to the output vector, which I guess you don’t over the API. I think most “distilled” models you see for download don’t do full backprop training, because it takes enormous amounts of compute, and instead use low-rank methods. The low rank methods really only change the style. Alibaba has the compute to do proper training.