This is the reason why conventional AI can’t make strides in mathematics. The LLM is a statistical model that generates human text. The training data is the output text divorced from the process that generates it in humans. If you feed an LLM lots of mathematical text, it will confidently produce mathematical texts of its own, but the LLM can’t add two numbers together, and it doesn’t know that it can’t add two numbers together. It can be trained to use a calculator, but humans don’t interrupt their writing to say “And now I’m using my calculator to determine the value of 1 + 1” so the LLM is just going to draw upon its training data to predict that "1 + 1 = " is followed by “2” or maybe “3 (for large values of 1).” Maybe someday it will learn that “9 + 10 = 21.”
This is the reason why conventional AI can’t make strides in mathematics. The LLM is a statistical model that generates human text. The training data is the output text divorced from the process that generates it in humans. If you feed an LLM lots of mathematical text, it will confidently produce mathematical texts of its own, but the LLM can’t add two numbers together, and it doesn’t know that it can’t add two numbers together. It can be trained to use a calculator, but humans don’t interrupt their writing to say “And now I’m using my calculator to determine the value of 1 + 1” so the LLM is just going to draw upon its training data to predict that "1 + 1 = " is followed by “2” or maybe “3 (for large values of 1).” Maybe someday it will learn that “9 + 10 = 21.”