• hubobes@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    7
    ·
    17 hours ago

    That doesn’t make much sense. When Anthropic moved to Sonnet 5 they introduced a new tokenizer which increased token use up to 35%. If these would be unrelated kinds of tokens why would the usage go up when the process of tokenization changes?