Blahaj.zone experienced a security breach and is handling it to properly reduce the risk of harm to their users. the current eta for their reture is in about 7 hours.
Blahaj.zone experienced a security breach and is handling it to properly reduce the risk of harm to their users. the current eta for their reture is in about 7 hours.
Excellent writeup, and I appreciate the transparency. I have some suggestions on how to mitigate something like this from happening in the future.
This is simpler than trying to control postgres permissions granularity. Even if one application that connects to a database gets owned, it doesn’t have access to other postgres databases, preventing data leaks/exfiltration.
Kata containers is a container runtime, that is virtual machine.
There is also Gvisor and Syd Box, which are application kernels. Application kernels are reimplimentations of the parts of the Linux kernel needed to run apps, and in this case both Gvisor (Go) and Syd Box (Rust) are in memory safe langauges.
Kata containers are faster, but you will need nested virtualization in order to use them. Application kernels are slower, but you can install them anywhere, including hosts where virtualization is disabled (like a VPS that doesn’t let you enable nested virtualization.
Both take a tiny bit more resources intensive due to no longer being able to share the host kernel, but for most part, it is worth it. They don’t bring an entire kernel along, just what is needed to run apps.
Both offer similar levels of isolation, and preventing applications running inside them from touching the host kernel directly. They effectively manage to prevent issues like copy fail, dirty frag, and so on, from owning your host.
They are fairly easy to install, docker has some docs here: https://docs.docker.com/engine/daemon/alternative-runtimes/ . But if you are using podman or kubernetes, you can also install them there.
A large part of the draw of stable Linux like Debian or Red Hat, is that they only do security updates. They don’t do feature updates, or even bug fixes (except for critical ones). In doing so, there is essentially a guarantee of reliability, where it is impossible for updates to break anything.
This makes it possible to enable automatic security updates, and you can even configure it to automatically reboot in order to load a new kernel that includes mitigations against issues like dirty frag. Make sure your docker containers are configured to automatically restart and everything will be smooth.
“Just patch” is a good but it is never enough, and I am frustrated hearing it so frequently. The way I view it is, any time I have to patch, what I really need to do is to improve my security architecture so I never have to “patch” this specific issues again. Patches are the exact kind of security toil that I complain about in this comment.
unique passwords is good practice, but separate db server for each of the services is extreme. it brings much more resource consumption. the solution here is being subscribed to security releases and updating soon. those application kernels also sound like a good idea. and as I understand, postgres permissions were not at fault, the permission system had a bug.
except that because of the bug, anyone with query permission could have become postgres superuser.
If a user can’t log in to a DBMS, they don’t have query permission.
Yes. It consumes more resources. But it’s not that much more, and you can make it fit easily. Many users using docker compose unwittingly do this since docker composes often bring their own database containers. When done consciously, you make a trade off for peace of mind.
I addressed takes like these in the last part of my previous comment. The linked comment also elaborates on my opinions about manual updates, manually watching security releases, and other forms of security toil.
weren’t they having access through peertube? you can’t revoke access to something that needs it.
sure, they could run dedicated postgreses for each service, but I think this kind of bug is so rare that it is not worth wasting lots of RAM on that, and even then, it would rather just deter people from hosting services they want.
also at that point, the question arises why not just run everything in qubes OS, each service in its own VM and its own VLAN. and the response is that it is unfeasible, and it was unfeasible even before the RAM shortage, unless you have enterprise hardware with 100+ GB RAM, consuming way too much power compared to a server made from desktop PC hardware, even when there is no meaningful CPU load.
I run 3 separate database servers on the same OS because I made a bad decision at one point, and I regret it. I just don’t have time to merge them.
The peertube database did not have the vulnerable extension enabled. They got access but connecting to another database:
From that database they escaped. So if they only had access to the peertube database, that path would have been cut off for the attackers.
VLAN’s suck. They allow for traffic to travel within each VLAN them, unmonitored and unrestricted. Sometimes red teamers call that “hard outside, soft inside, like an eggshell”.
Better, is a private VLAN. Private VLAN’s enable the firewall to monitor or block all traffic within them. For many usecases, a single private VLAN can replace the complex, many VLAN setups that people spend so much effort setting up. You just block everything within itself from communicating with eachother, and then you can explicitly allow the stuff that actually needs to talk.
Often, there is no reason to allow devices with a VLAN to communicate with eachother. And even if you are allowing intra-pvlan communication within a private VLAN, you can now monitor the traffic, which you can’t do with a regular VLAN.
Yeah. That’s basically what virtual machine or application kernel based runtimes are doing. The trick is that they are designed for this usecase, with the goal of making the isolation as cheap and performant as possible. Qubes runs full Linux VM’s since it is designed for a desktop isolation usecase.
Anyway. It’s a matter of threat model and effort. I like application containers/vm container runtimes due to how simple and easy they are to run, and the clear security benefits they provide. Install in < 30 min, configure the container runtime, restart your containers, done. Switching from a regular Linux desktop to Qubes is not as simple, so it’s harder for me to recommend that, and it becomes a matter of threat model.
It’s also common to separate services out in different Proxmox VM’s, which offers isolation benefits akin to Qubes.
Congrats on the security isolation!
In all seriousness, you should be able to migrate them into the same database by dumping the database with pg_dump or a similar utility, and then loading them up using pg_restore. Other databases have equivalent tools. I used the Dbeaver open source database GUI to do this a few days ago.
how? did they use the same database user account for all databases? unless I misunderstood it, peertube’s database user shouldn’t be able to operate in other databases of the same server.
nonsense. if you don’t use VLANs, you are essentially using a single huge VLAN. of course using VLANs don’t exclude monitoring and firewall restrictions. could as well say, networking sucks, lets disconnect everything. if you don’t set up monitoring, and you don’t set up restrictions, then yeah, there will be no monitoring and no restrictions, like on any network.
didn’t you just say VLANs wholesale suck?
also, peertube will require access to the internet…
only if it was so simple. you can set up routing restrictions between subnets on the IP level, but the switch will gladly forward all traffic anywhere inside of the VLAN, according to the destination MAC address. with that, a compromised system can confuse every other on the VLAN with ARP poisoning and faking DHCP servers.
yeah, that’s the easier part. but one of the databases is mysql, which I want to get rid of completely. and conversion is nontrivial. probably dbveaver could handle it though.
This is what private VLAN’s fix. The switch isolates all traffic, forwarding it to the firewall instead, where the firewall can then monitor and control it. It’s usually done with a combination of port isolation, where layer2 traffic within a subnet is essentially blocked, and proxying the arp requests. Nothing can communicate with each other directly at the Layer 2 level, only with the switch, which then replies to all ARP requests instead. Then the switch forwards the traffic that was supposed to go to that IP address through the firewall and back as Layer 3 traffic.
This is what Private VlAN’s fix. Any ARP requests a device attempts is simply blocked at the switch level via port isolation. Only the switch is able to communicate with attached devices unrestricted. It serves the DHCP, and assigns and IP address… and then if the device attached to the port attempts to lie about it’s IP address or the like, the switch simply ignores it since it knows what ports are supposed to have what IP addresses (port security).
This allows for much more granular isolation, but also a simpler architecture, since you can replace complex multi VLAN setups with fewer private VLAN’s. Maybe even just a single private VLAN. I say VLAN’s suck because by default, yeah they kinda suck. You pointed out the problems. I guess private VLAN’s are a type of VLAN but they are a different thing, just like not having any VLAN’s is really technically just one big VLAN.
Is it a custom service? Or a service you are consuming from someone else? If the services supports mysql/maria/postgres, I would be worried that the adapter under the hood might be using different schema or something like that.
It does look like dbeaver has a data transfer feature: https://dbeaver.com/docs/dbeaver/Data-transfer/#supported-formats
Where you can transfer data across tables, across databases, and even across database types. There is also “Data export” mentioned on that page, where you can export data to json, or other non sql formats. If the schema match, maybe you could use that to transfer.
If the services supports both mysql and postgres, you might be able to start by making a backup of the mysql database, loading that into a test database, and then setting up a clone of the service, but pointed at postgres instead. Then you could investigate the schema, or safely attempt data transfer from the mysql replica into the postgres database. Or mariadb, but mariadb is compatible with mysql (fork of it), so I am assuming you meant postgres which is different.