So my manager today asked me if I could stay later when there’s broken things in prod, and then today his star dream employee yolo’ed a full stack change into prod without review. It’s fucking massive and implements new API endpoints, touches >20 files. Many of the diffs are too large to render in the browser.
It’s almost comical, but something immediately broke.
Most of my day, I’m digging through code to identify bugs created from this shit, just to get a stealth merge midday.
I kind of don’t know what to do.
You need SRE concepts. First, if you break it then you fix it; in a system where anybody can make a change, it’s the changer’s responsibility to meet service objectives. Second, if your boss doesn’t find that acceptable then they need to appoint a service owner and ensure that only the owner can make changes; if the owner breaks it then the owner fixes it. Third, no more than half of your time should ever be spent fixing things; if something is constantly broken then call a Code Yellow or Code Red, tell your service users that you cannot meet your service levels, and stop working on new features until the service is stable again.
Under no circumstances, ever, should anybody stay late. There should only be normal business hours, which are best-effort, and an on-call rotation which is planned two months in advance. Also, everybody on call should be paid hourly minimum wage on top of salary for their time.