See, you have more experience in the matter than I do, hence the caveat that I’m not an expert. Thanks for sharing your experience.
Then again, I’d consider 128GB of memory to be fairly serious hardware, but if that’s common among hobbyists then I stand corrected. I was operating on the assumption that 64GB of RAM is already a lot
All in all, 106 billion parameters on 128GB of memory with quantization doesn’t surprise me all that much. But again, I’m just going off of the vague notions I’ve gathered from reading about it.
The focus of my original comment was more on the fact that self-hosting is an option, I wasn’t trying to be too precise with the specs. My bad if it came off that way
See, you have more experience in the matter than I do, hence the caveat that I’m not an expert. Thanks for sharing your experience.
Then again, I’d consider 128GB of memory to be fairly serious hardware, but if that’s common among hobbyists then I stand corrected. I was operating on the assumption that 64GB of RAM is already a lot
All in all, 106 billion parameters on 128GB of memory with quantization doesn’t surprise me all that much. But again, I’m just going off of the vague notions I’ve gathered from reading about it.
The focus of my original comment was more on the fact that self-hosting is an option, I wasn’t trying to be too precise with the specs. My bad if it came off that way