Whenever I download bigger files, my RAM fills up and the tab/download crashes. Is it possible to “fsync” downloads with Firefox (really Librewolf, but still) so that it goes directly into storage instead of RAM? Or to adjust the amount of memory used before it is dumped into storage?

  • tal@lemmy.todayEnglish
    3·
    9 days ago

    Is it possible to “fsync” downloads with Firefox (really Librewolf, but still) so that it goes directly into storage instead of RAM?

    Fsync will write data that your userspace program has already handed off to the OS. This probably won’t be your problem; your OS’s kernel should already write out that data if it’s under memory pressure. Assuming that this is Linux, you can twiddle Linux’s kernel settings to make it write out more frequently. The sync command will also write out existing data that hasn’t been written to disk by the kernel, and you could do something like $ while true; do sleep 1; sync; done.

    But you probably have an issue inside the application, not in handling data that needs to be fsynced.

    I’d also add that I’d be a little surprised if this happens with Firefox’s native downloading. Maybe someone could have some sort of custom download Javascript something-or-other that stages stuff in memory on a website, though I don’t think that even a poorly-written Javascript should be able to produce a tab crashing.

    • printf("%s", name);@piefed.blahaj.zoneOPEnglish
      1·
      9 days ago

      Those are some amazing ideas and insights! I’ll try that intermittent sync just for the fun of it, but I’ll also check if I can loosen up Librewolf’s anti fingerprinting and whatnot. Probably messing with lots of common functions, JS included.

  • Auster@thebrainbin.org
    2·
    9 days ago

    Haven’t tried this on Firefox, but would making a swapfile (so you can use the storage it takes as RAM) help? Or maybe get the download link from the download history and run it on wget from the terminal?

    • printf("%s", name);@piefed.blahaj.zoneOPEnglish
      1·
      9 days ago

      Thanks! It actually ate my swap too. 🤣 I could of course increase the size of the swapfile, but that doesn’t solve the underlying problem of the download going into RAM/swap. But maybe that’s just how it is?

      Holy crap that was a great idea with wget! I’ll try it later. Do you know if I’m able to authenticate logins with wget? As in, I’m actually downloading from a cloud service and I’m worried that it’ll give me a 401 Unauthorized.

      • SteveTech@aussie.zone
        2·
        9 days ago

        Do you know if I’m able to authenticate logins with wget? As in, I’m actually downloading from a cloud service and I’m worried that it’ll give me a 401 Unauthorized.

        If you open the network tab on the devtools, you can usually copy a request as a curl command. I believe downloads also show up there and it will let you curl the file with all the cookies and headers that the file was originally requested with.

      • Auster@thebrainbin.org
        2·
        9 days ago

        About authenticating, not too familiar with it to give a good answer. All I know is that some download links come with validation tokens attached and that those links with tokens can be copied into external downloaders while the token is active. But things like logging in within wget or solving captchas is being me.

      • tal@lemmy.todayEnglish
        2·
        9 days ago

        Do you know if I’m able to authenticate logins with wget? As in, I’m actually downloading from a cloud service and I’m worried that it’ll give me a 401 Unauthorized.

        Once upon a time, Netscape Navigator stored cookies in a “cookies.txt” file. While Firefox doesn’t do so these days, Firefox has a “cookies.txt” addon that will let you export the cookies from your browser session in said format. You can use these if you require some sort of authentication that needs to be done interactively from the browser: $ wget --load-cookies cookies.txt <URL to download>.

        wget also natively supports authentication with a password, but it’s possible that whatever service you’re using requires interactive authentication in a browser, and for those, this is a workaround.

  • moonpiedumplings@programming.devEnglish
    2·
    9 days ago

    One potential workaround is probably to use a download manager, that supports more advanced and nuanced control over how the files are downloaded.

    Here is what I have used in the past:

    https://github.com/subhra74/xdm

    https://addons.mozilla.org/en-US/firefox/addon/xdm-browser-monitor/

    The browser extension hijacks firefox’s download function, so that when you download a file, it instead connects to XDM, and offers to download that there.

    It also offers save and resume functionality, so if it dies in the middle you can resume.