For playing around, no. My GPU is 9 years old and has 8GB VRAM. I generating my 384x512 profile picture with KoboldCPP (Vulkan) and Z-Image-Turbo model locally.

I just regenerated my profile picture with KoboldCPP a bit differently. I used some clearer LoRA configurations than I did initially. For Z-Image-Turbo it’s CFG 4 and for Flux2 Klein 4b CFG 1, both used Euler. For Anime V1 LoRA I started with anime illustrious, and for Koni Animestyle LoRA with anime_style, . :

384×512 & Z-Image-Turbo & Anime V1 & 8 steps (2min 42s):

384×512 & Z-Image-Turbo × Anime V1 & 25 steps (8min 16s):

384x512 × Flux2 Klein 4b × Koni Animestyle × 4 steps (22s):

384x512 × Flux2 Klein 4b × Koni Animestyle × 8 steps (35s):

384x512 × Flux2 Klein 4b × Koni Animestyle × 25 steps (1min 28s):

Upscaling it with RealESRGAN x4plus anime 6B upscaler only takes 10 seconds longer with this dimension.

KoboldCPP outputs how long generation takes like:

[11:09:15] Generating Image (8 steps)
  |==================================================| 480/480 - 0.00MB/s
  |==================================================| 480/480 - 0.00MB/s
  |==================================================| 480/480 - 810.94MB/s
  |==================================================| 480/480 - 0.00MB/s
  |==================================================| 8/8 - 19.64s/it
[11:11:57] Generating Media Complete

Ah yeah, I think you need a 6GB gpu for images and probably like a 12Gb GPU for videos.

I think you can run with models without, but it just slows way down

Yeah. Basically there like giving an LLM a set of tools it can use on its own. It’s a lot like having an intern at a company. You don’t give them access to anything you don’t want destroyed. But you can give them simple/safe buttons to push.

Ha ha no, I never went as far as needing embeddings for a language model. MNIST is actually, you know, the very simple classification model. It’s a bit the ‘Hello World’ of machine learning. It’s a dataset of handwritten digits that you have to classify in 0, 1, 2, 3, 4, 5, 6, 7, 8, 9.

It is a good test because you can train it in minutes if not seconds even on crappy hardware and unoptimized code.

So when I’m talking about negative numbers, I am talking about negative numbers. I am talking about weights that are needed to be negative to have negative influence on the output. Like “this pixel in the center is white, so the likelihood of the number being zero decreases.”

I still don’t understand gradient descent fully, but can you explain why you think it should be replaced and with what?

Honestly, I’m just talking about it because we are being silly, but I am not sure that’s an idea I actually want to defend.

I just have this feeling that gradient descent is a good mathematical construction for what we try to achieve, but that mathematical purity maybe, just maybe, gets in the way of efficient computing. Of course, there are thousands of very competent, highly paid people who already explored that venue, so I’m pretty sure that if something better was possible and within the reach of one person, it would already have been discovered.

(Counterpoint: we routinely rediscover things that were invented in the 90s that are now good ideas now that we have very good computing)

The thing is gradient descent is used to tell you in which direction you’re supposed to move a weight to lower the loss of your results. In other words, to minimize the error of your network.

Gradient or partial derivatives are like an ideal mathematical tool to do that. We are able to derive it for a lot of functions, linear or not, and it is a well-studied mathematical object, so it really makes sense to use that.

The direction of the gradient will tell you the direction in which the parameters need to move. More precisely, the partial derivative of a given parameter will tell you if you need to increase it or lower it in order for the loss to improve.

Thing is we use the sign that’s clear but the intensity I am not sure it is that relevant because we keep fighting against things like gradient vanishing problem where very deep networks tend to have very low gradients and we compensate a lot of its problem through optimizers, choices and tricks.

I wonder if there would not be a pure computer science way of just keeping track of the direction in which you want the parameter to change.

I don’t know, maybe triple all the calculations by one tick in both directions? or just use gradients on one bit when it makes sense? Or find a function that’s very fast to compute but that just approximate gradients and that is just better than randomness at finding the sign.

Like I said, that’s just an itch to scratch. That’s not a strong conviction that there is something. But if you were to give me two weeks salary to just work on that, I would be very happy to.

deleted by creator