After a year online the free speech-focused instance ‘Burggit’ is shutting down. Among other motivations, the admins point to grievances with the Lemmy software as one of the main reasons for shutting down the instance. In a first post asking about migrating to Sharkey, one of the admins states:
This Lemmy instance is much harder to maintain due to the fact that I can’t tell what images get uploaded here, which means anyone can use this as a free image host for illegal shit, and the fact that there’s no user list that I can easily see. Moderation tools are nonexistent on here. It also eats up storage like crazy due to the fact that it rapidly caches images from scraped URLs and the few remaining instances that we still federate with. The software is downright frustrating to work with, and It feels less rewarding overall putting effort into this instance because it feels like we’re so isolated.
A few weeks later, in the post announcing that Burggit was shutting down, another admin says the same:
The amount of hoops that burger has to go to in order to bring you this site is ridiculous. To give you an idea of how bad this software is, there’s no easy way to check all the images uploaded to the site (such as through private messages). When the obvious concern of potential illegal imagery is brought up to lemmy devs, they shrug and say to plug in an expensive AI image checker to scan for illegal imagery. That response genuinely has me thinking that this is by design, and they want it to be like this. We can’t even easily look at the list of registered users without looking through the DB, absolute insanity.
The other thing is there’s no real way to manage storage properly in Lemmy, the storage caches every image ever uploaded to any instance forever.
Also the software is constantly breaking.
They also say that Kbin has many of the same problems, so I’m just curious to know if the admins of bigger Lemmy & Kbin instances feel the same way about these software.
(instance admin here, but for a small one) woof well, for me, I agree, but I wouldn’t use that wording.
Lemmy for sure isn’t a plug and play site. Setting it up took leaps and bounds, learning way more about nginx than I ever really cared to, and figuring out documentation that was very clearly out of date. Very little logging or error messaging exists to help with that problem.
Very little errors exist at all, it’s very much a “happy path” project. That’s why we get constant spinners everywhere, because when an HTTP error occurs there’s no actual error message. (Come on guys, just add it to your standard HTTP messages, if statusCode < 200 || >= 300 then show a toast message).
But yeah, the moderation tools have to be the worst. Lemmy has an amazing development group that’s separate from the main developers who have patched together a good set of tools, from automods to CSAM and illegal scanning, huge props to them - but these issues are routinely ignored by the main devs. I was shocked, honestly shocked that when we were under CSAM attacks that there was not an immediate roundtable of the head devs to try to solve the problem officially. Here was a problem that 99% of countries would immediately and gladly throw us, the instance admins, in jail over and they just handwaved it away. In fact, I don’t know that there was ever an official post about it, or even that there are things coming to help with it.
I love Lemmy and being here, and the devs have done a great job at building this platform for us, but we’re at a critical point right now. It’s no longer software that is just fun side projects and building stuff that looks cool, it has some real issues now that it has a real userbase. I’m definitely one to say “But it’s FOSS, and other people can pick up and submit a PR” - but it also says something when the head devs just completely ignore a massively huge issue with it.
Bugs and caches and that sort of thing I can overlook. Those I can wait on and see them get smoothed out over time. Actual issues that could land me in jail or get the feds to beat down my door? Those I kind of expect a fast response.
So, I’ll say I’m extremely conflicted. I want to host lemmy long term, and I’m happy to bring the fediverse to a few more people, but the csam attacks really altered my view of the devs.
Edit - because my favorite manager said “Bring me solutions, not problems” a few things that would really help immediately -
- Integrate db0’s CSAM checker natively, more or less a plug and play option, or a checkbox. His checker sits at an endpoint. The admin page of lemmy could easily have you plop in the endpoint and it would start checking
- Have an image management portal, with capabilities to:
- Auto remove images after X time (to help with ballooning storage costs)
- Perma-delete images and users (maybe blurred too if the CSAM checker flagged it, so I don’t need eye bleach) (Edit again, 0.19.4 might have fixed this, I need to upgrade so I’ll see)
- Federating image purges, so one purge on one server will force purge it on everyone else’s
- ~~Disabling of caching other server’s images ~~ (Edit again, I see 0.19.4 just dropped which has this, so this is good). This way I’m only responsible for my own users.
- View images that are not related to a post (DM’d messages that I’m hosting, or people just uploading images to my site)
- Bring in a logging system into the UI itself, so I can keep tabs on the error logs. I can pipe them somewhere, but this would be a major plus as an admin
Dont forget the complete ignorance of gpdr. That shit will get you fucked over, and its not as simple to follow as it seems.
In fact, Im not 100% sure that federation works with gpdr, since you cant garantee all data will be deleted.
I’m not sure it’s that difficult to follow. If you offer a service in the EU, you are responsible for your server deleting personal data (or, even better, not even hosting it in the fist place!); you are not responsible for other people not deleting their copy of personal data.
But I’m not that well-informed in the actual legalese so my best understanding is the big issue is the EU’s definition of “provide service to the EU” more than anything else. They seem to think that just because your users might upload a local copy of a picture of someone from the EU, even if you yourself are not allowing connections from the EU, then you are serving to the EU. And with how nazi the EU has been going lately with stuff like ChatControl, the last thing I’d want as an instance owner is to be upheld to arbitrary boomers’ (lack of) understanding of technology.
but we’re at a critical point right now. It’s no longer software that is just fun side projects and building stuff that looks cool, it has some real issues now that it has a real userbase. I’m definitely one to say “But it’s FOSS, and other people can pick up and submit a PR” - but it also says something when the head devs just completely ignore a massively huge issue with it.
This is a general issue I think, not just for lemmy but the whole fediverse (whatever one’s opinions might be on particular priorities).
It’s all non-profit and being run and built at a much smaller scale than many users would appreciate (I think). Sure there are plenty of people here, but not that many. Combined with no obvious revenue streams, such as ads or subscription fees, there really is only so much that can be done. Some time last year even the Mastodon team (by far the most successful fediverse platform) admitted that they didn’t have the capacity to work on new things for a while … they were just busy keeping things running. And they are (apparently) notorious at being slow to ship new features. Meanwhile platforms like firefish just straight up died last year.
So yea, it might be a critical point, for sure. But putting more on the core dev teams may not be the answer for the simple reason that it’s just not viable in the long run.
If we enjoy the bigger community focus and open and non-profit organisations that makeup the fediverse, the “answer” at this critical point might be to find a way to give back somehow … to organise, build communities, run fund-raising campaigns, think of ideas for more sustainable funding, find devs who can help etc etc. It’s perhaps onerous and annoying, even to read perhaps … but this is likely the tradeoff we have to make for a place like this.
I fully understand that, and I explained it in my reply to nutomic. It’s not that I wanted to just pile on, but rather at an emergency like that, an all-stop would have been justified in my opinion, to stop all work and go into emergency “What can we all do to stop this and prevent it”. All other issues were secondary in that moment compared to stopping CSAM, and it didn’t feel that way. They added it to the pile of issues, and I’m glad to see changes came out, but in that moment I didn’t see much if any support from them
I hear you for sure on this.
I personally appreciate the push the core devs make for a “less demanding” relationship between users and open source devs. I think they have a point and it’s good for long term sustainability and I’ll probably find myself defending it.
But I like you’re framing, and in retrospect it seems (as is usually the case) that some people organising was what was missing just to get everyone helping each out as much and efficiently as possible. While any member of the community can (and should) do that, at some point it makes sense for the core devs to take on a task like that, I agree.
But yeah, the moderation tools have to be the worst. Lemmy has an amazing development group that’s separate from the main developers who have patched together a good set of tools, from automods to CSAM and illegal scanning, huge props to them - but these issues are routinely ignored by the main devs. I was shocked, honestly shocked that when we were under CSAM attacks that there was not an immediate roundtable of the head devs to try to solve the problem officially. Here was a problem that 99% of countries would immediately and gladly throw us, the instance admins, in jail over and they just handwaved it away. In fact, I don’t know that there was ever an official post about it, or even that there are things coming to help with it.
My impression at the time was that admins were handling the CSAM wave just fine with existing mod tools and through Matrix chats. A roundtable wouldnt have solved anything except make people feel good. Besides we still were extremely busy at the time to scale up Lemmy and resolve problems revealed by the huge amount of new users. Keep in mind that Lemmy is still at version 0.x which means that its not feature complete. So if something is missing that you find important, consider waiting a year or two and checking back then. Or get it implemented yourself, thats what open source is all about.
That said most of the features you mentioned have already been implemented, including a list of all locally uploaded images.
I was not handling it fine, it was generally chaotic, and in the Matrix chats I remember it being chaotic, for both attacks. Luckily by the second one we had db0’s tooling to help a bit more, but there still many of us who were exposed to the images. We lost a lot of instances during those two attacks from admins who justifiably didn’t want to take on the risk.
I completely understand how crazy it was, but the lack of response from you guys was disheartening, it really did make me wonder if I should continue hosting or if I should bail out. Ultimately, I decided to stay obviously, but had to do some hard extra steps, like reducing privacy and registering with the feds for CSAM.
So like I said, I’m torn. I respect you guys for everything you do, but that was a moment where all other development should have stopped to immediately address a real problem, and while you think a roundtable would have just been feel good, I think we could have kept a lot of instances online if it had been done. Assurances that yes, new changes are coming, and official suggestions like “Here are the endpoints to delete the images”, or nominating db0 or someone as the person in charge of the outbreak. It was honestly a scary time, and for us owners who accept a lot of risk, for many of them it was too much.
Anyway, I have a habitual case of foot in mouth disease, so it was immediately after posting that comment that I heard about 0.19.4, and immediately felt stupid. I tried it last night but I kept getting timeout errors and something about “Could not get user’s /inbox” or something, I’ll try 0.19.5 today. Thank you for bringing additional mod tools, they’ve been hugely needed. I know they’re not glamorous to make, but they keep the communities healthy and strong.
Edit: 0.19.5 also failed. I wrote up a github bug on it, until then I unfortunately have to stay on 0.19.3 https://github.com/LemmyNet/lemmy/issues/4850
The entire time after the Reddit migration was extremely chaotic. I dont remember when exactly the CSAM attacks happened, but around that time we were already very exhausted from all the urgent work we had to do on scaling, patching security vulnerabilities and fixing countless bugs. I also dont remember receiving any requests from admins to help out with this. So if you notice something similar in the future, feel free to message me directly. Anyway we are only two people working full-time on Lemmy, and have lots of different tasks to take care of. So it gets very difficult to give everything the attention it deserves, and to prioritize things correctly.
I honestly think you peeps need to somehow invest in your communication strategy somehow. Such communication breakdowns is/was causing schisms in the lemmy community which is an extreme shame as that’s in turn driving away the same potential contributors that would help the software improve faster to cover these same points. I would argue that saying things like “we’re still in beta, come back in 2 years if you can’t handle the heat” is not doing you any favours. I know you are technically correct, but there’s no reason to phrase it like that, yanno? Not everyone interprets such statements the same way and for non-ASD/ADHD people, this can parse very hostile and confrontational, even if you honestly didn’t mean it to be read like that.
Apologies in advance for the unsolicited advice, but have you considered reaching our for some community outreach person to join your team? Such positions won’t necessarily fill themselves and you need to ask for it. But at this point I think it might significantly help the lemmy project avoid such drama.
How else would you say this? And who do you suggest reaching out to? Keep in mind that it would have to be a volunteer position as we dont have the funds to pay for it.
Yes I understand that. I meant making an open call for volunteers on that aspect. Note that I’m in the same boat with my own FOSS project where it’s not always easy to find the volunteers, but at least I don’t have the same pressures as you face on this area. Not saying this is going to be an easy fulfillment, but it can be perhaps something to pursue.
How else would you say this?
Note that I’m AuDHD, so I’m not the best person to actually do these things. It’s because I know my limitations that I have compensated by learning to notice these pitfalls in communication. I would also need someone to help me in the same situation as you.
Nevertheless, If I were to speculate, I wouldn’t even go in that direction. From what I’ve seen, most people who know what they’re doing in this aspect will just say something like “we acknowledge the issue and we’ll do what we can to handle it asap” or whatever, just so people don’t feel left in the cold, you know? Again, don’t take what examples I write as the exact practice as it’s not my area of expertise. I’m just (badly) parroting what I saw work better.
Its easy to say this now, more than half a year later. But youre ignoring that we were completely overworked and exhausted back then. That said Im taking your feedback into account and will hopefuly to handle it better in the future.
You could have volunteers working on the communication aspect. That doesn’t have to be a paid position.
I’ll let db0 suggest a better phrasing for the “we’re still in beta” part
Sure but its not so easy to find volunteers. Would you or db0 be willing to do this?
Nice suggestion
I was around when a user when the CSAM attacks happened. That was crazy stuff, sorry you admins had to go through it.
The introduction of a plugin system seems interesting here, though it’s still alpha and basically looking for feedback from would-be plugin developers
AFAICT, there’s no established way for a plugin to surface affect the UI, which would be a somewhat unwieldy problem anyway due to the apps and frontends ecosystem. Probably the best path for any plugin that provides a UI would be to have a system for aggregating links to plugin UIs.
With something like that in place, plugins and other services that just use the DB/API, could really go a long way to filling these holes, if they haven’t already.
So it seems that the work needed here is perhaps “distribution” work … where there’s a more “plug and play” Lemmy distribution with plugins etc bundled?
This is what excites me the most. There are huge potentials for plugins, and I think it’ll ease some of the strain from the core engineers. Most of the “ideas” I see posted really could be plugins. Things like badges on posts, verification of links, etc etc could all be plugins that individuals could make. The problem with developing against the core repo is that you have to learn and understand the core repo, so you don’t fuck up something else in some other place accidentally. Plugins are a neat way where we can say “I’m a function that does one thing, just do the thing here, and then do what you need to with that data”
Absolutely!
The bit I’m conceptually stuck on (not know much at all about how a good plugin architecture would work) is how a plugin can surface or affect the UI, especially in an ecosystem with multiple UIs/Apps/Frontends, and, a federated ecosystem at that.
Given the apps, I figure it’s not possible without a convention of plugins providing APIs which apps can then implement against when available, which adds a good amount of complexity but should be viable for popular/useful plugins. Though, tangentially, this does affirm for me that the whole native mobile app expectation is a bit of a trap for a social system like the fedi (as webUIs are naturally more universal and maleable).
So, for immediate results, I can see only two options:
- a plugin operates on the backend directly manipulating or creating content not unlike a bot
- a plugin provides its own webUI which is made available through a simple and dedicated location in the UI
Is there something I’m missing about how a plugin system could work?
When I started working on PieFed I was all enthusiastic about the idea of moderation tools. But when it came time to actually code that functionality it was like pulling teeth. Just. Sooo. Boring. It took weeks longer than it should have, for that reason. This was really surprising to me because I’m deeply passionate about moderation and ‘gardening’ a community.
That’s the thing about open source, people just do the fun stuff. There’s always some fun stuff to do which distracts from the boring-but-necessary.
Not just that its boring, mod tools also require a huge amount of work because you need to make changes across all parts of the code (database, api, federation and frontend).
There’s people who like building this sort of code. I don’t mind it for example and I’ve already written multiple of them for the lemmy ecosystem (fedi-safety, fediseer, threativore etc), I’m just too busy with my own projects to contribute even more. If you are not having fun doing them, try to find and retain people who do.
there was a discussion about this same post before, I’ll just copy paste my comment…
That post complains about not being able to view/manage images hosted by your instance, but v0.19.4 already fixed that last week? So that kinda disproves them saying the Lemmy developers didn’t want it to be possible. Also the post complains about the amount of storage used by caching images but that was also fixed/improved in v0.19.4
I’d really love it if people stop saying “it’s by design” when they can’t point to any motivation for that design. When the quoted admin says “thinking this is by design” this is equivalent to saying “Lemmy developers prefer that there be no image moderation tools.”
Like, what. Why would they want that. They clearly don’t want that. They’re working on changing that.
It’s very hard for people to accept that there are other things that may need to be worked on before their requested fix/feature. Every big project has a huge backlog of issues/feature requests, you can’t do them all in 1 day or even 1 year. Especially with low funding lol.
It also is true that ideologically-motivated-coding is an actual thing.
Imagine someone hating that their propaganda gets deleted by moderators, so they make it difficult for moderators to function that way…
while they, themselves, just so a SELECT on their DB to see the images, to delete all the ones they don’t want…
Remember, it isn’t only corporations who are committed to enforcing the Enshittocene, ideologues do, too.
Those comments are proven false by the dot-4 release of Lemmy, but I’m not accusing the Lemmy devs of being the way those post-quoted comments said.
I AM stating, bluntly, that deliberate torque on the use of ANY aspect of an app, is a thing, now, and need be considered as ONE of the possibilities.
_ /\ _
Though Lemmy has funding for full-time developers.
And it’s not like other features get implemented in the meantime. Progress is really slow here, even compared to hobby projects.
Edit: Lol, thanks for downvoting.
Did you read the changes in 0.19.4? Those are only the highlights, there is also a full changelog linked. 0.19.0 before that had even more features. And I doubt you can show any hobby projects that have faster progress with only two devs.
Since I’m dabbling in AI at the moment: What about llama.cpp? Dude handles like 50 pull requests a week, coordinates everything and codes himself. And it’s really complicated stuff and not the only project. And I mean there is lots of Linux software I use, (web-development) frameworks, smarthome stuff and electronics projects that I participate in and I’m always fascinated by their pace and how they manage to do that in addition to a day-job?! And they regularly push new features… I’ve had contact with some, filed bugreports and sometimes the next day they solved my issues and pushed a new version.
With Lemmy, my UI bugreports from a year ago are still open and not fixed. And it feels like contributions and bugreports are more a burden to the devs here and not that welcome like I’m used to from other projects. And yeah, I’m glad the last release was a bit bigger. But I mean it took 5 months… And moderation tools are traditionally an issue here. I’m glad something gets implemented. But we’re still far from where we need to be. Same with the image handling and proxying.
I’m not sure what to make of this. Sure, software development ain’t easy. But every new release I check the changelog and usually it’s just some minor bugfixes. And twice a year a bigger release like this month with new features, yet the last bigger user-facing feature I can remember was instance blocking in december. And this is more or less adding the ability to hide posts and change how voting is displayed, if you’re just a user.
Edit: I appreciate the work, though. And I like the idea of the platform. It’s just that I’d like it to grow and flourish. But to me it seems we’re often taking baby steps. And in the meantime stuff breaks and admins complain they barely cope with everything with the tools they have.
I looked at some of the pull requests and most of them seem very small, only changing a couple of lines. Still impressive but not really comparable to implementing a new feature in Lemmy. For that we need to make changes to various different parts of the code (database, federation, api, js library, frontend), then test it and pass code review. All that takes a lot of work because we need to ensure that existing functionality doesnt break. In this way a web server like Lemmy has much higher standards because there should be no bugs at all. If your AI project has some bugs, users can easily roll back their local install to an earlier version.
Youre right about lemmy-ui, unfortunately it doesnt have enough contributors. I dont know why that is, you’d think a project written in a popular language like Typescript would easily find contributors.
Youre right about lemmy-ui, unfortunately it doesnt have enough contributors. I dont know why that is, you’d think a project written in a popular language like Typescript would easily find contributors.
Random thoughts:
- Is it obvious enough that one can contribute to the UI separately from the backend and that it’s a Typescript SPA style UI?
- If not, maybe a bit of a “dev recruitment campaign” could help … let people people know and what sorts of issues could really do with new contributors lending a hand? Maybe even a bit of a “Inferno isn’t that different from all of the other SPA frameworks/libraries spiel?”
- Is the use of Inferno as oppose to one of the big 3 React/Vue/Svelte a repellent? (perhaps a downside to the “diversity” of frontend frameworks?)
- Are would-be UI contributors more inclined to make their own front-end or app than contribute to the default webUI?
More generally:
- Would a server side rendered webUI be welcome?
- Then the contributions would mainly be on templates and their “simpler” logic, which might be more attractive or easier to get started on?
- Plus, it might be more efficient? The current UI feels to me like it would suit server side rendering well.
- Is this where the new leptos UI is heading … more server side rendering (I don’t know much about leptos)
- Do you have a sense of usage numbers for the different apps and frontends? Obviously you only run lemmy.ml, but do you have a sense of how much the front-end gets hit versus the API directly?
- I ask, because If the default WebUI is really the main interface, then it makes sense to try to organise some more contributors (It’s certainly my main, nearly exclusive interface, as much as I’ve like some of the alt front ends or apps)
- Is it obvious enough that one can contribute to the UI separately from the backend and that it’s a Typescript SPA style UI?
Yeah, you’re not doing it right. On Github you have to click on “Insights”. And alike Lemmy which is split into two parts, llama.cpp also has a backend called ggml that does the (tensor) maths. Combined, the git stats are as following for the last four weeks:
- Lemmy (+UI) 207 files changed, +7,841 additions and -6,472 deletions
- llama.cpp (+ggml) 707 files changed, +157,754 additions and -95,611 deletions.
So they definitely touch a lot more code regularly. Whichever PRs you clicked on, they added 50 times as much new lines of code in the same timeframe. And coding things like that is maths heavy and you also need to read the scientific papers and implement the maths. And they did quite some maths themselves and contributed their quanitzation techniques and benchmarked and studied them in addition to the coding. I’m really impressed by the guy. And he seems nice and attracted quite some contributors with his excellent and fast software. Reviews and comments their ideas and integrates them fast. And now it’s a flourishing project that leads in its field. And the project isn’t even that old…
I get it. Software development isn’t that easy. Especially the ‘touching different parts of the code’ is something I don’t really like. I mean it is like it is. And having architectural patterns like this is fairly common (logic, database, UI) and you have like 2 models of the data, one for federation and then the internal representation. I’m not that familiar with the Rust frameworks and how cumbersome it is to deal with them. With the correct database abstraction toolkit and other frameworks it gets better and you can often tie the stuff together. Also helps with the bugs. If it’s really bad, maybe the architecture isn’t optimal. Or the chosen frameworks suck. Other than that it’s the job of a programmer to tie those aspects together, deal with the complexity and combine it into a working product.
I’m not even sure if you can assure that Lemmy has no bugs… I mean unit tests, integration tests and reviews won’t cut it with distributed or federated software, right? I mean you’d need to roll out a small cloud of instances and do end to end tests, check if everything federates and if there are performance regressions… I’m not sure where Lemmy is regarding this. I occasionally observe when something big happens like federation breaking.
Sure. And UI programming is also something that is not really fun to me. I’m also not sure why it hasn’t more contributors. Maybe the atmosphere isn’t that welcoming to new people. Or the userbase in total is just too small. I mean fediverse observer reports like 50k Lemmy users, and that’s not that much people if we’re talking about the subset of people who learned programming and have the spare time to contribute. Maybe it’s too interlinked with the rest of the code or not documented enough. I’d say it’s probably not that attractive to get involved because it’s mainly small bugfixes that can be implemented without also getting involved with the rest of the project. And apart from drive-by pull requests, people usually have some bigger vision when they join a project.
Though Lemmy has funding for full-time developers.
barely, edited it to say low funding
I hope the plugin system will attract more contributors, especially since it supports a variety of languages
I’m not sure about the numbers but it should be like 6,600€ a month?! join-lemmy.org shows 3,656€ per month from donations, plus ~750€ a week they said in their last AMA from the NLnet fund.
I’m not sure if I’d consider that low… Sure it’s not much compared to the revenue of a commercial platform. But still, you can build something with like 2x40h weeks. (plus a community)
Maybe they already factored in the 3k from NLnet and it’s just 3.6k in total, I don’t really know. But they’re always talking about two full-time developers plus one more they’d like to pay… So that makes me think it’s probably 6.5k€. Maybe someone can fact-check it.
Our last NLnet funding round is from 2022 which is just getting completed now. At a total of 60.000€ over two devs and ~24 months thats around 1250€ a month. So about 3050€ per month which is quite low for a software developer. Additionally the NLnet payments are very irregular as they are not monthly but when specific new features are implemented. The number of 750€ a week is for estimating the payment for NLnet milestones, but a large part of our work cannot be funded by them. NLnet only funds development of new features, but we also need to spend a lot of time fixing bugs, reviewing pull requests, preparing releases etc.
Thank you for the details.
Thanks. So the number on join-lemmy.org already includes the NLnet fund? I suppose that means you get ~600€ a month from the other (independent) supporters?
I’m confused. Liberapay 1.679$ + Patreon 1.165€ + OpenCollective 935$ + Crypo
adds up to the ~3.600€ but in which category are the NLnet bank transfers?
Funding isn’t that high, they launched a funding drive in October:
Before the Reddit migration, our income was almost exclusively made up of generous donations from the NLnet foundation. This funding was based on getting paid for implementing new features, specified in advance.
We’ve known that this funding could not last indefinitely, and that after several years of funding, NLnet’s resources are better spent getting other projects up and running. Additionally, much of our time is spent on other equally important work: reviewing changes from community contributors, fixing bugs, doing support, and various organizational tasks.
That is why we are launching our first annual funding drive. The goal is to increase monthly, recurring donations from currently €4.000 to at least €12.000. With this amount @dessalines and @nutomic can each receive a yearly salary of €50.000 which is in line with median developer salaries. It will also allow one additional developer to work fulltime on Lemmy and speed up development.
https://join-lemmy.org/news/2023-10-31_-_Join-Lemmy_Redesign_and_Funding_Drive
I couldn’t find the details of the current status of the NLnet funding at the moment, maybe if someone has that number?
The donation pages shows 3600€ for the both of them: https://join-lemmy.org/donate
AFAIK the NLnet funding is still running and there were still some milestones to claim as I last asked them in some AMA. Should be paying them an additional 3.000€ a month?!
They really should be more transparent an link that stuff and their progress.
Should be paying them an additional 3.000€ a month?!
How much do you think a full time Rust developer makes?
Kinda depends on productivity. I’d say 45k to 60k€ is alright for an average coding job in some company. I don’t know the details here. For self-employed people that varies a lot and developing Lemmy propapbly doesn’t compare to a salaried job at all.
Lemmy is heavier on the cpu than for example mastodon or matrix are but a lot less on the harddrive. Its insane. I use matrix the most, then lemmy, then mastodon. Still, mastodon sucks hdd like crazy, matrix as well. Only lemmy is easy on the hdd.
Of course I‘m running private instances. You can never know why public instances go bonkers. Maybe someone is abusing it for other things.
Anyway, lemmy does have some moderation tools but I agree. They need work. You have to know a lot about linux, databases and have to figure out a lot of stuff if you have problems with lemmy. Thats not okay.
But as always with foss. If you run a public instance, you should accept donations and if they dont cover the costs so you can pay someone to make a fork for you that has what you need, you might be doing something wrong.
It would be interesting to investigate why Lemmy has high CPU usage. In principle it should be quite efficient as its written in Rust. Its also not doing anything particularly performance intensive, unless you are subscribed to lots of communities or have lots of users.
From other comments I read, the database seems to be the issue, not lemmy. Is uses postgres and that uses the resources. Lemmy itself is no problem.
I suppose there is still room for database optimization then, but its hard to find people who know how to do this.
Hi there! Thanks for taking the time to answer my comment. You folks are doing incredible work and I wanted to say thanks! :)
Sadly, I‘m not a database expert, otherwise I would help. So donations are the only form I can help with at this point.
Ain’t that the truth! I struggle for the same reason on the AI Horde ;_;
Here it isn’t so heavy on CPU, but the database eats RAM like crazy. My guess would be that their server is constantly swapping out stuff from RAM due to insufficient surplus RAM and this as a result creates the high CPU load.
@[email protected] if this is true then you should try adding some swap space.
The language it’s written in has very little, almost nothing, to do with how efficient larger applications are.
This is almost entirely up to the design and day-to-day decisions of the developers. These almost always outweigh the efficiencies of the underlying languages themselves (within reason).
A single location of poor data access patterns could negate the aggregate performance gains of your entire application, as an example. A framework that prevents you from making simple mistakes and drives you towards more efficient patterns goes much further than the language is written in.
Between Rust, C#, Java, and Go you’re essentially even on performance for large applications (with C# pushing ahead of the pack). What you are not even on is engineering efficiency, it’s going to take considerably longer to build the same set of features in rust than any of the others listed. And the performance is likely the same, potentially even worse depending on the maturity of the ecosystem.
Rust is a great systems design language and a great language to choose when developing high efficiency libraries & frameworks for I/O and data processing. It’s not really a great choice for application development due to how slow it is to actually get things done in.
I fully expect to see alternate backends written in more operationally efficient languages over the next decade that will catch up to the official Lemmy codebase, and potentially even replace it. It actually sounds like a super fun project, funding is always a problem though.
deleted by creator
Thanks for commenting on this. It‘s cool to have more peeps share their experiences.
On the lemmy part I‘m partly with you. Lemmy itself is not the issue. It is what it does with postgres. I have multiple postgres instances, one for each service. The lemmy postgres is insane while the mastodon postgres is easy on the cpu.
well, kbin is somewhat defunct. mbin is under active development, and many of these things are on their radar. moderation tools are coming along, user list is not really an issue. they even got mfa workin!
these applications and the environments they are building are in their infancy. instance admins cannot expect a mature product, or that they wont need to get their fingers dirty to implement features they want. lemmy is at what, 0.19.4?
the image caching/filtering is good example of an ongoing concern 'verse-wide. as the lemmy devs pointed out, if this is a solid concern as an admin you would implement file scanning. if you absolutely want to use lemmy and need a user list, build it.
this shit is not turn-key
e. ive been running mbin at https://moist.catsweat.com almost a year
I’d just like to point out that most of the complaints referenced in this post are at least partially being addressed with the latest release of Lemmy, 0.19.4
From the release announcement
Image Proxying
There is a new config option called image_mode which provides a way to proxy external image links through the local instance. This prevents deanonymization attacks where an attacker uploads an image to his own server, embeds it in a Lemmy post and watches the IPs which load the image.
Instead if image_mode is set to ProxyAllImages, image urls are rewritten to be proxied through /api/v3/image_proxy. This can also improve performance and avoid overloading other websites. The setting works by rewriting links in new posts, comments and other places when they are inserted in the database. This means the setting has no effect on posts created before the setting was activated. And after disabling the setting, existing images will continue to be proxied. It should also be considered experimental.
Moderation enhancements
With the URL blocklist admins can prevent users from linking to specific sites.
Admins and mods can now view the report history and moderation history for a given post or comment.
The functionality to resolve reports automatically when a post is removed was previously broken and is now fixed. Additionally, reports for already removed items are now ignored.
The site.content_warning setting lets admins show a message to users before rendering any content. If it is active, nsfw posts can be viewed without login, after consenting.
Mods and admins can now comment in locked posts.
Mods and admins can also use external tools such as LemmyAutomod for more advanced cases.
Media
There is a new functionality for users to list all images they have previously uploaded, and delete them if desired. It also allows admins to view and delete images hosted on the local instance.
When uploading a new avatar or banner, the old one is automatically deleted.
Instance admins should also checkout lemmy-thumbnail-cleaner which can delete thumbnails for old posts, and free significant amounts of storage.
As you can see, there were many improvements to the moderation tools and image hosting UX/storage requirements. There’s obviously still room for improvement, but everything else out there has glaring issues as well. Mbin/PieFed are even less polished than Lemmy at this stage, although they are progressing quickly.
Also worth noting that most of the replies to the first post express a strong user preference for Lemmy over Sharkey.
Good riddance to burggit. Manny legitimate complaints but absolutely good riddance.
What’d they do?
This is why I made Lemmy Post Purger (LPP).
I haven’t had big problems honestly. Still have less than 100GB of storage, which is like 0 dollars on object storage.
I found it very easy to start with the ansible setup and later changed to self managed docker. I even moved servers once and it worked fine.
Overall I haven’t had any issues, but I’m also a software engineer so maybe I know more than others. There are some missing features for sure, but I don’t think it’s so bad so far at least.
This Lemmy instance is much harder to maintain due to the fact that I can’t tell what images get uploaded here, which means anyone can use this as a free image host for illegal shit, and the fact that there’s no user list that I can easily see. Moderation tools are nonexistent on here.
0.19.4 provides a way to see uploaded images (although not the best) but this version was only recently released so I can see where the frustration is coming from especially since the CSAM attacks happened nearly a year ago. At the time, I had to make a copy of pictrs, view everything on a file manager, and manually remove those images. People can still upload images without anyone seeing it however.
It also eats up storage like crazy due to the fact that it rapidly caches images from scraped URLs and the few remaining instances that we still federate with.
This was fixed in 0.19.3 (released 7 months ago) where you can disable image “caching”. This has solved storage costs for us together with pictrs’ image processing.
plug in an expensive AI image checker to scan for illegal imagery
It’s unfortunate that we need this. Not everybody has the resources to run fedisafety nor does everyone live in USA where they can use Cloudflare’s CSAM scanner. I think a good way to deal with the issue is to have images that are not public, not be stored (or have no private images at all). This way images can be easily reported.
Overall, I understand the frustration and to some degree I also feel the same but I also limit my expectations considering the nature of the project.
What are the big mbin instances? I can’t seem to find any
mbin.social ;-)
https://mbin.fediverse.observer/list https://fedidb.org/software/mbin
we’re “spread” out on purpose to avoid becoming an unwieldy behemoth like kbin.social…
fedia.io is the largest one
fedia.io is the biggest one and the one I’m using, there’s also kbin.run and kbin.earth
also kbin.run and kbin.earth
They’re asking about mbin, not kbin, no?
Kbin.run runs mbin, the name is historical
Same with kbin.earth. Unfortunately, one problem with the fediverse is that everything (users, magazines, posts, comments, etc.) is tied to the instance’s domain, so it can’t really be changed at all.
Ah, thanks. Even if it is a bit confusing, it’s cool that they were able to switch out the underlying software without breaking everything. I guess it helps that mbin is a fork of kbin.
@[email protected] Both of these instances use mbin, there’s pretty much no instances left on kbin except kbin.social, almost all instances migrated to mbin
Only the storage usage has been a problem for me, but that is much easier to handle now. I do wish there was some better documentation for exactly that.
Should there be something like a lemmy admin’s community for talking about the ins and outs of running a lemmy instance? Something like that over time can be some sort of band-aid for missing documentation (it’s searchable etc).
I’m aware that there’s the matrix room and a lemmy support community … but maybe just opening things up to any chat about hosting lemmy in one place could actually help (in part because you can easily search specifically within one community).
There is documentation, if anything is missing we definitely appreciate contributions.
https://join-lemmy.org/docs/administration/administration.html
I used to do a pretty frequent task to find any image in my media directory, which has atime longer ago than a week ago, which hasn’t already been crunched in some previous round, and crunch the hell out of it with either
mogrify -qualityorpngquant. All of these softwares like to keep full-quality copies of a basically infinite number of images which there’s about a 99.9% chance will never be needed again, and you can crunch them down to a tiny fraction of their original size, and they still honestly look more or less fine.I’m not sure about the software always breaking, I haven’t had this issue. I will say though this most recent update (19.4) has me frustrated, mainly because the instructions are clear as mud (especially pictrs 0.5). Once I get it figured out I’ll have to post a real upgrade doc instead of what is currently available. I have never seen the lemmy matrix as busy as it is now with upgrade questions and puzzled admins.
How does piefed compare python/flask is far easyer to develop I really think moving away from developer with such extreme politics is a good idea.
deleted by creator
