So my manager today asked me if I could stay later when there’s broken things in prod, and then today his star dream employee yolo’ed a full stack change into prod without review. It’s fucking massive and implements new API endpoints, touches >20 files. Many of the diffs are too large to render in the browser.
It’s almost comical, but something immediately broke.
Most of my day, I’m digging through code to identify bugs created from this shit, just to get a stealth merge midday.
I kind of don’t know what to do.
Don’t provide bailouts for neither your manager nor your colleague. Highlight as far up the chain as you can the damage their behavior is having on the business.
Do NOT under any circumstances fix it. The other guy needs to and you need to be clear he broke it and how
I came here to say exactly this. The offending dev and your manager aren’t going to understand the severity of issue until he spends days trying to find the bugs. Not your circus, not your monkeys.
I would immediately fire somebody with such a mentality. Letting bugs live in production just to prove your point, that’s borderline criminal. I get you’re not happy with the vibecoder’s job but there are other, smarter ways to deal with this than sabotaging the company.
These bugs shouldn’t have reached production. This is why you do reviews in chunks.
Probably very true (again, it depends on the product/company).
Gunna be honest, if you really feel that way you’d fire the vibe coder. I don’t give a shit if it’s broken ifs due to the negligence of management and other employees.
Poor planning on your part does not constitute an emergency on mine. And ofc this varies based on the service as all things do. But it’s rarely important enough that I’d stay over. Like if lives where at risk or it’s a “bring the company down but” maybe I’d stay. But there would be hell to pay for whoever broke it later. Either that or the company would loose me as an employee
By that logic wouldn’t you be firing the vibe coder for not rolling back the commit?
Fallback and fix is the best solution.
Maybe. Context is not sufficient (mistakes can happen, intention counts) but breaking prod may be bad depending on the product type.
Other ways such as?
I dunno, human things like communication and letting people know what’s wrong here and how not to let that repeat. If it’s pathological then maybe that work environment is not fit for you.
You need SRE concepts. First, if you break it then you fix it; in a system where anybody can make a change, it’s the changer’s responsibility to meet service objectives. Second, if your boss doesn’t find that acceptable then they need to appoint a service owner and ensure that only the owner can make changes; if the owner breaks it then the owner fixes it. Third, no more than half of your time should ever be spent fixing things; if something is constantly broken then call a Code Yellow or Code Red, tell your service users that you cannot meet your service levels, and stop working on new features until the service is stable again.
Under no circumstances, ever, should anybody stay late. There should only be normal business hours, which are best-effort, and an on-call rotation which is planned two months in advance. Also, everybody on call should be paid hourly minimum wage on top of salary for their time.
Roll it back and go home.
I would say no to staying back. The employee that broke it needs to fix his code, at the very least roll the change back.
At the next meeting or post mortem bring up why code can be pushed to prod without review. Your git forge should be able to block merging to prod by a single person.
Yeah. I log off at EOD and don’t check messages. It’s not my families fault he lets his employees vibecode shit into prod. The Vibelord is the main problem, but others do similarly stupid things.
Adding to that, about the “stealth merge”, couldn’t it be just the rollback? Don’t change what works, as the saying goes.
And expanding on the meeting or post mortem idea, take this time not to fix the code, but to make an essay showing what broke, how it broke, and how it could be avoided in the future. Also preferably not letting your tone getting acid or those hearing from the company could instinctively refuse to listen to you even if your points are solid.
If only there was some way of reverting the merge commit and redeploying.
Pushing to prod without review and breaking the running application is a resume-generating event in many companies. In many others it’s not even possible because of programmatically enforced policies.
If your company’s response is not to prevent or dissuade it but to have other people work overtime to fix the mess then that’s a major management fail.
Try to educate your boss about best practices. This incident should give your arguments some more weight.
Deployment to prod should not be something a developer can do by themselves; a proper CI/CD system can be configured so that prod can only be deployed to by people with an appropriate role (product owners or lead devs if your company doesn’t have POs).
If you don’t have such a system, make it an explicit policy: Only Steve the lead dev (or someone specifically appointed by him while he’s absent) can push to prod; if anyone else does it they get invited to an uncomfortable meeting with Steve. If they do it again the meeting will be with HR.
But seriously, you should lobby for a proper CI/CD system (if none is present) and for the system to be configured so that a) you can’t merge to the main branch without a code review and b) deploying to prod only works from main and with explicit approval by a PO/lead dev. That should stop most of the shenanigans.
Roll it back and go home
If your manager won’t hear anything negative about Vibelord, polish up your resume. There’s nothing you can do to change the culture of your team if the boss isn’t on board with it.
Not sure what kind of person your boss is, but if you want to try to win him over, try to figure out what might sway him. If he’s technically illiterate, talk to him about best practices (like, I dunno, code reviews) and cite reliable sources and data if you think that would help. Avoid calling out Lord Vibington, the main thing is to put a picture into your boss’s head of what this could/should look like. He might not realize that these issues are preventable.
The goal here is to get your boss to take more ownership of the team’s culture, and start insisting on preventative measures. Mr Viberator will either have to conform, or there will be increasing friction between him and the boss.
Ask to be paid by the hour
Explain to your manager that if an employee can merge code without reviews, he should expect breakages on regular basis.
Propose that your team can meet to talk about how to make the process better, so no breakages occur.
Mention personal responsibility and knowledge - it’s a good practice that a person who wrote the latest code changes either helps either is responsible for making the service working again, if there are issues - specifically because they might have some idea what’s broken.
If your manager will ignore it, I’d start looking.
You render diffs in the browser?
And no, let the yolo fix the things he broke, or he will never learn to be a responsible coder.
GitHub and similar tools show PR diffs in the browser.
Changes that big need to be broken down into smaller PR-sized steps. To do that you need an actual plan of how to get from here to there. Bro is swaggering like a senior then coding like a junior.
What to do? Well now you have evidence that AI tools are not yet good enough to write code that doesn’t need human review before deploying to prod. Chat with whoever is in charge of Process, and get some team guidelines in place to prevent unreviewed code going to prod, and unreviewable PR’s being submitted for review.
This is not how you do development. Like, at all. If changing jobs is an option for you do it. If not, well, not much you can do. You’re not going to fix the entire company if management and other devs don’t know how it should work and don’t want to change.
Edit: I should explain first that I do not think you can change your employees or manager. A technical solution will never fix a management problem. Perhaps your manager is getting enough heat to be open to better management controls. You should be in “sanity protecting” mode.
Original: While the posts include a lot of fiction, but “overemployed” crowd have some good advice. Start applying elsewhere, downshift your effort at $job_one, and move to collecting a check.
Or ramp up and take your manager’s job or get fired trying. Gather data and allies in the C-suit, and stage a coup. Or unionize (or post pro-union flies in the bathroom).
I’d just talk to the guy, clarify HE needs to stay late not me, then log off. If he’s nice about it I’d consider ataying but from your description, he’s anything but.
What you gotta do in situations when someone is beingan asshole is spark a meeting with the higher ups so they can see the BS you deal with and how it’s none of your business that prod was messed up.
