Updates in Linux are far more tolerable. There’s really no reason to delay Debian stable, imo, unless you absolutely can’t risk some downtime.
Server rats excepted, it’s just a process that goes in the background and at most, you have to reboot the kernel.
There’s no staring at the Blue Screen of Boredom while windows update holds your machine hostage.
I work at a medium size company with hundreds of Linux servers and none of them get updated. Because it’s more important that they keep running as they are than to have the latest updates. I bet this is very common for most companies.
There is nothing more important than security patches on a system.
I used to work at an FMI, which’s motto was “keep things stable”. Even the ciso department bought that crap. Until we hired a white hat hacker. The only thing given was the name of the company. He managed to get into the building, access an employee’s workstation and install a root kit on one of the most important financial message tracking systems (you know, the one that instructs other systems to transfer money), using a security bug, which would have been patched if they kept a regular (security) update cycle. After shit hit the fan, many people were fired and an update cycle was introduced.
No system is important enough to not patch. And if you believe it is, you’re wrong.
Yeah, but that just takes way too much work. You think I really care about the company’s/bank’s money if I’m not getting paid enough for that job? Security patches can also introduce new problems, like x changes, so y doesn’t work, so the main app doesn’t work… and what, then I have to manually edit code, introduce the thing that x relied on so that y can work again?
I’m sorry, but this is not your average IT department’s job… or if it is, I expect a damn good compensation for it.
I’ve updated and rolled back snapshots because of shit like this… nah, not gonna try and figure out what the problem was… at least not for the salary I’m currently getting paid. If it burns, it burns, so be it.
I’d be surprised if you actually saw anything change from security updates tbh, I don’t think I’ve ever seen anything break from a quick patch.
Dist upgrades are when things might break, but they’re only once every few years. Leave them too long though and you may end up with compatibility issues if you need to make changes.
Fair enough if you’re not getting paid enough, the company should hire more people to stay on top of that though.
Usually you upgrade everything though, not just sec patches. And it’s a risk that something stops working, and nobody wants to spend time on that…
No, you update strictly only security patches.
Exactly… because it’s tidious and time consuming… and I won’t get extra pay for it. Meanwhile I’m also expected to do everything else I do… sorry, just not gonna bother at all.
They have and that is why I don’t do them any more. Happened a few months ago in fact. Updated one of the Debian servers for one of webapps we have running (a black box piece of shit VM that’s stuck in 2010 I think)… suddenly, the app in the VM doesn’t work. The VM does start, but the app doesn’t work, just throws a 404. Why? Beats me, don’t have time to troubleshoot. Roll back a snapshot, everything works again. Conclusion, don’t update that.
See, around here, you don’t keep your job by messing around with things that already work. They work, period, why did even feel the need to mess with that 🤨. If that’s management’s view on security, fine, so be it 🤷.
I still file reports on things not being up to date, just so that if shit hits the fan, I’m not the one taking the fall for it.
God I hope I don’t use any products from your company.
They make multimedia, so you’ve probably “used” it at some point… or not… depends how much TV you watch.
“Way too much work” — if you ever said that where I work I’d fire you or not hire you in a heartbeat. An administrator’s role is not only to the stability of the system but the security too. You’re a hackers wet dream.
The phrase “Fuck you, pay me” comes to mind.
Cheapskates don’t get top of the line security hardening. Pay more now or suffer a breach and pay contractors $1000/hr to fix your broken shit because you paid minimum wage for an administrator position and wanted them to do 5 jobs at once.
This guy gets it… and probably doesn’t live in the US, cuz he knows the term “work 5 jobs at once”.
Surely you meant the opposite? Working multiple jobs is a very USian thing. Now I’m curious, where are you from?
No no no no, I meant what I said. Working 5 jobs AT ONCE (at one postion, one place)… AT THE SAME TIME. I do hardware, software, scripting, installing, configuration, maintenance (hardware and software… the whole shbang, DB included), Linux, Windows, BSD, servers, workstations (over 800 of them, and it’s me and 2 other guys under me!)… I even do freaking rig dusting! And ALL that, for 700 freaking euros a month! Not to mention pirating, that is also included cuz… they’re too cheap to pay for licenses for… well, anything really.
So, excuse me if I don’t care if the servers are up to date, mmmk.
From Macedonia BTW.
One, you have no idea how much or little I’m getting paid. Two, you have no idea where I live and the struggles I have to face every day. Three, even if I do work “as expected”, I won’t get paid more (agan, you don’t know where I live).
It’s real easy to bitch about work ethics on a full stomach while getting back from work in a nice car with heated leather seats.
What I say still holds. If you ever uttered that sentiment in front of me or did not follow through on patching when asked you’d be out on your ass. Has nothing to do with your situation or what you’re making. Your righteous indignation on patching has no place in a business plain and simple.
Not trying to start an argument here but you sound very far removed from individual contributors, so maybe from your point of view it would simply look like adding it to a pile. More important than adding it to a pile is to make sure there’s systems in place to make sure OSs are patched. You wouldn’t be complaining to the IT/sysadmin guy about your servers’ vulnerability or patching schedules, you’d be talking to your cybersec department who’d have oversight. And if there’s a breach and your only defense is “I added it to the IT guys pile”, 100% you are getting fired as well.
I get your point, but capitalism is about doing the least amount of work to maximize your pay. And if the owners (who have the most skin ulin the game) don’t care about infrastructure then why should anyone else?
No, it doesn’t… because as I said, you have your needs (food, shelter, good car, nice place to live) met… you wouldn’t be talking like this if your place was a dump or you ate the cheapes shit on the market (cuz that’s what you can afford).
Every individual concern you’ve brought up in this thread is valid and not uncommon. The important part that @[email protected] is focusing on is that your mindset on how you approach these challenges is incredibly unprofessional.
Having more to do than you realistically have time to do is the reality of working in IT. Everything the business does ultimately comes back to IT at some point in the process, so we naturally have to work with every single branch of the business.
Being underpaid is a reality of work for most people in the modern world. The professional thing to do is to decouple your feelings of how you’re being paid from individual tasks or duties that are expected of you. “I’m not paid enough to deal with this” should be limited to tasks outside of your scope of work. If you’re ultimately not paid enough to do your job, complaining about individual tasks that are part of your job being bnove your paygrade is just saying you aren’t willing to do your job. I can’t tell you the best solution to your pay situation. Maybe changing jobs or even changing industries will help, but also changing your mindset can do wonders for your mental health. For example shifting to instead saying to yourself “I’m woefully underpaid but at least I’m working in IT and not at X” can greatly help ease the pain until you reach whatever milestone which does help improve your situation
What you need to do regarding your workload is have a conversation with your manager/superior about prioritization. You say “hey, I have this this this and this that need to get done right now and I can’t realistically do them all, what do you want me to prioritize and deprioritize” and if something important hasn’t been given priority in a long time (such as patches) you need to then push back and say “we haven’t been able to apply security patches in quite a while, I think we should reprioritize this so we can put some time into patching this week” this is how you manage a gigantic workload is by shifting priorities. The longer important maintenance tasks are ignored, the larger the impact and the harder it will be to complete the tasks
The important part that @[email protected] is focusing on is that your mindset on how you approach these challenges is incredibly unprofessional.
I know it is.
Now, ask me why that is.
The professional thing to do is to decouple your feelings of how you’re being paid from individual tasks or duties that are expected of you. “I’m not paid enough to deal with this” should be limited to tasks outside of your scope of work.
Lol 😂, no one has actually told me to keep the servers up to date, the only thing I was ever told was “keep shit running”. I’ve done updates on my own incentive, since I’m the senior IT engineer in the company. When things turn to shit after an update, hey, I’m rolling back a snapshot, I did not sign up for this 🙌.
And that is basically it 🤷… I get the same salary regardless if I do them or not. After a few failed ones, I just gave up. F it, not worth the time or the effort.
If you’re ultimately not paid enough to do your job, complaining about individual tasks that are part of your job being bnove your paygrade is just saying you aren’t willing to do your job.
Nope, I’m saying “Fuck you, pay me!”.
You think they care about updates and security? I’ve mentioned it a few times at meetings… “yeah, we’ll talk about that later”. OK 🤷. They obviously have no idea how fucked up things can get if you’re not up to date regarding security… but hey, they have been warned more than a few times 🤷.
What you need to do regarding your workload is have a conversation with your manager/superior about prioritization. You say “hey, I have this this this and this that need to get done right now and I can’t realistically do them all, what do you want me to prioritize and deprioritize” and if something important hasn’t been given priority in a long time (such as patches) you need to then push back and say “we haven’t been able to apply security patches in quite a while, I think we should reprioritize this so we can put some time into patching this week” this is how you manage a gigantic workload is by shifting priorities. The longer important maintenance tasks are ignored, the larger the impact and the harder it will be to complete the tasks
🤦… dude, you really have no idea where I live 😂🤣😂… otherwise you wouldn’t be saying this.
Things have been said more than once… I have asked, have pleeded for more personel… deaf ears. I have put it in writing, no use. Fine, then I just keep things running and that’s basically it 🤷.
Oh, and regarding workload, I already prioritize. The priority is to keep shit running, not to be up to date (obviously)… so, I just keep things running.
Well, getting promoted is difficult when you watch Netflix or browse lemmy on company time
No no no, I am not getting promoted anyway… no promotions here. No one ever gets promoted, you don’t get bonuses, you don’t get overhours pay, you don’t get raise… so why bother 🤷.
The I can only recommend you to start automating everything you do, to make tour job easier and make more time to slack 😝
Start small, and build on that.
Try Ansible, it is easy and allows you to do some really cool stuff. It helped me migrate 500+ systems from KVM to vmware, where no commercial tool was able to help me…
Creating automation takes time. I have done it with certain things, but not everything. Some things are just way too complicated to easily create automation for them.
Though I do agree that things like Ansibel can help and I have been thinking about this for a while, but then I’d have to drop everything else I’m doing for like at least 2 weeks and do JUST that, nothing else… which is not an option currently.
Plus, there is always the risk of “shit not working” after an update… and frankly, we’ve had that before and again, I am not getting paid enough to deal with the backlash of things like that happeneing.
If it’s important that it keeps running then it should just be redundant and taking one node down for an update shouldn’t be an issue. I know this is wishful thinking for a lot of services but I refuse to be on call for something if the client can’t be bothered to make it redundant.
Jup same here. We have a colleague that constantly reminds everyone that we’re not properly patched (even running eol versions) but there’s always something to be done that’s a higher priority.
Exactly. Shit needs to just work, period. Why? Because otherwise, I’m the one getting 2AM calls… and I would be OK with that if I’m properly compensated for it… which I’m not.
Did you think of testing security updates on a staging environment before going in production with it, if you suspect in can break things?
I think there is no excuse to apply security fixes wich have a CVE number.
If you are on Debian stable unattended updates are not a problem.
See, building and configuring a staging environment also takes time and money… money which they are not willing to spend on something “for testing” and not in actual use. Plus, I’m not gonna get paid for doing that either, so why actually do it… to be honest, I would do it, even for free, but you gotta caugh up the money for the hardware man. I’ve been told “just use what you have in the scrap pile”… for what, a server 🤨? Are you serious? They barely spend any money on that even, why should I bother creating something as e staging environment.
This sounds so horrible, I would consider finding a better employer. I hope, you are not stuck with them.
Actually, I kinda am. Can’t really afford to spend a month or two without pay, so if I do find anything better (which I seriously doubt, every company here is more or less the same regarding IT practices), it would have to be a drop in replacement, which is also hard to do here (they’re gonna try and squeeze as much free labour as possible from you, so you’ll probably be stuck with a 200, 250 euro freelance salary for the next month or two, and as I said, I just can’t afford to do that right now, money is tight, got a family now).
Not at all.
Typically monthly or quarterly patching depending on severity and DMZ exposures. When log4j or shellshock hit it was patch once the patch was released and tested
If it’s a personal server that can manage being down for 15min or so. You could just setup auto updates with email if anything goes wrong and reboot off hours. Containers also make it less risky although it does fail to update every once in a great while.
All of that can also be tested in a preproduction environment as well, downtime is really a poor excuse for not patching
The auto updates tools in Debian does allow you to specify security only or security and packages but not kernel. Mix and match, specify a version to stay on, include back ports, etc.
Do you work for the North Korean government or something OP? Why discourage people from keeping their systems secure?
What they are referring to is people just don’t update their server because during that time they wouldn’t be able to make a profit. This goes more to middle siszed businesses but happens rather often
Blows my mind, lol. Usually means no redundancy that allows one set to be done while the other set handles the traffic.
“Why should we pay for another server one works just fine, a second would just be waisted money.”
Also
“We need 9 9s of reliability or the company will fail.”
Yeah, it is quite common, I can confirm… well, at least around here it is.
Just a hunch, but it feels like you are living in the same country as I.
Macedonia? 🤨
Okay, maybe not… sorry
Meeh, worth a shot 🤷. Shitty contries all over the place 🤷 😂.
I was making a joke
Joke transfer unsuccessful. Server crashed. Time to update the joke server.
Well seems like i missed this one
Security is an art… the art of not giving a fuck about your data
-Op, probably
I find this to be least acurate with debian… on other distros a patch may or may not install a new version of that package. that can bring changes to the behavior. On debian stable the security issues are backported. So you can patch and be sure that there is no changes to the behavior of the system. It is basically the reason all vm’s i manage are debian stable.
It is also true they never crash. But that is expected of linux. It is the extreme reliabillity that is the debian killer feature for me.Me with my ‘homelab’ nas:
system (user-facing) package has an update? It’ll auto-update overnight
dockerized service has feature updates? Let watchtower handle it with the weekly schedule
dockerized service with security patch? yeah, let’s hit that this afternoon
actual system update? EVERYTHING IS GOING OFFLINE -4 SECONDS AGO FOR THIS
The system is going down NOW.
https://youtu.be/Z1TlbLfaJp8?si=nL9C6MqHUbWm0cy-
The system is down
Here is an alternative Piped link(s):
https://piped.video/Z1TlbLfaJp8?si=nL9C6MqHUbWm0cy-
Piped is a privacy-respecting open-source alternative frontend to YouTube.
I’m open-source; check me out at GitHub.
Debian updates are not usually that big of a deal especially if you have HA configured
i’m pretty sure security updates are optional.
Just put a “these colors don’t run” text in the log in
Unatended-upgrades keeps all systems securly patched. But there is a need for a reboot for kernel updates now and then.
I remember when Linux fan boys would give Windows users shit for needing to restart for updates.
Linux does not need to reboot, if you use KLP.
I do not.
Hi Microsoft Bootlicker 👋
That was 1st about the forced restarts (Linux always only told you that it is necessary, but let you decide when to do it) and 2nd about the commomness of it happening on Windows (restarts are necessary more often on Windows).
True except for the one BOFH admin on the team who actually cares about best practices.
And yes, most distros have painless updates, the devs and everyone else don’t care.
Hi. It’s me. The guy bitching about best practices every other meeting. Sorry, but some of my past and present coworkers are clowns.
I’m a compassionate operator from hell. I will compromise with the devs on some practices but I force security updates on them with impunity.
“Until you crash, no on
careswill reboot you.”- until there’s a PCI audit.
I have two words for you, “compensating controls.”
It’s like goddamn magic.
yes, im guilty of this. haven’t got time to update my server to v12
Isn’t live patching a thing?
Meirl
But Debian has security updates and you can set up unattended upgrades.
