A note that this setup runs a 671B model in Q4 quantization at 3-4 TPS, running a Q8 would need something beefier. To run a 671B model in the original Q8 at 6-8 TPS you’d need a dual socket EPYC server motherboard with 768GB of RAM.

  • 小莱卡@lemmygrad.ml
    link
    fedilink
    English
    arrow-up
    2
    arrow-down
    1
    ·
    edit-2
    2 months ago

    Fwiw i can run the deepseek7b model with RX6600 (~$200, shud be cheaper nowadays) relatively fast, tho i haven’t used it for compiicated tasks.