Podman is awesome—and totally frustrating

So Podman is an open source container engine like Docker—with “full”^1^ Docker compatibility. IMO Podman’s main benefit over Docker is security. But how is it more secure? Keep reading…

Docker traditionally runs a daemon as the root user, and you need to mount that daemon’s socket into various containers for them to work as intended (See: Traefik, Portainer, etc.) But if someone compromises such a container and therefore gains access to the Docker socket, it’s game over for your host. That Docker socket is the keys to the root kingdom, so to speak.

Podman doesn’t have a daemon by default, although you can run a very minimal one for Docker compatibility. And perhaps more importantly, Podman can run entirely as a non-root user.^2^ Non-root means if someone compromises a container and somehow manages to break out of it, they don’t get the keys to the kingdom. They only get access to your non-privileged Unix user. So like the keys to a little room that only contains the thing they already compromised.^2.5^ Pretty neat.

Okay, now for the annoying parts of Podman. In order to achieve this rootless, daemonless nirvana, you have to give up the convenience of Unix users in your containers being the same as the users on the host. (Or at least the same UIDs.) That’s because Podman typically^3^ runs as a non-root user, and most containers expect to either run as root or some other specific user.

The “solution”^4^ is user re-mapping. Meaning that you can configure your non-root user that Podman is running as to map into the container as the root user! Or as UID 1234. Or really any mapping you can imagine. If that makes your head spin, wait until you actually try to configure it. It’s actually not so bad on containers that expect to run as root. You just map your non-root user to the container UID 0 (root)… and Bob’s your uncle. But it can get more complicated and annoying when you have to do more involved UID and GID mappings—and then play the resultant permissions whack-a-mole on the host because your volumes are no longer accessed from a container running as host-root…

Still, it’s a pretty cool feeling the first time you run a “root” container in your completely unprivileged Unix user and everything just works. (After spending hours of swearing and Duck-Ducking to get it to that point.) At least, it was pretty cool for me. If it’s not when you do it, then Podman may not be for you.

The other big annoying thing about Podman is that because there’s no Big Bad Daemon managing everything, there are certain things you give up. Like containers actually starting on boot. You’d think that’d be a fundamental feature of a container engine in 2023, but you’d be wrong. Podman doesn’t do that. Podman adheres to the “Unix philosophy.” Meaning, briefly, if Podman doesn’t feel like doing something, then it doesn’t. And therefore expects you to use systemd for starting your containers on boot. Which is all good and well in theory, until you realize that means Podman wants you to manage your containers entirely with systemd. So… running each container with a systemd service, using those services to stop/start/manage your containers, etc.

Which, if you ask me, is totally bananasland. I don’t know about you, but I don’t want to individually manage my containers with systemd. I want to use my good old trusty Docker Compose. The good news is you can use good old trusty Docker Compose with Podman! Just run a compatibility daemon (tiny and minimal and rootless… don’t you worry) to present a Docker-like socket to Compose and boom everything works. Except your containers still don’t actually start on boot. You still need systemd for that. But if you make systemd run Docker Compose, problem solved!

This isn’t the “Podman Way” though, and any real Podman user will be happy to tell you that. The Podman Way is either the aforementioned systemd-running-the-show approach or something called Quadlet or even a Kubernetes compatibility feature. Briefly, about those: Quadlet is “just” a tighter integration between systemd and Podman so that you can declaratively define Podman containers and volumes directly in a sort of systemd service file. (Well, multiple.) It’s like Podman and Docker Compose and systemd and Windows 3.1 INI files all had a bastard love child—and it’s about as pretty as it sounds. IMO, you’d do well to stick with Docker Compose.

The Kubernetes compatibility feature lets you write Kubernetes-style configuration files and run them with Podman to start/manage your containers. It doesn’t actually use a Kubernetes cluster; it lets you pretend you’re running a big boy cluster because your command has the word “kube” in it, but in actuality you’re just running your lowly Podman containers instead. It also has the feel of being a dev toy intended for local development rather than actual production use.^5^ For instance, there’s no way to apply a change in-place without totally stopping and starting a container with two separate commands. What is this, 2003?

Lastly, there’s Podman Compose. It’s a third-party project (not produced by the Podman devs) that’s intended to support Docker Compose configuration files while working more “natively” with Podman. My brief experience using it (with all due respect to the devs) is that it’s total amateur hour and/or just not ready for prime time. Again, stick with Docker Compose, which works great with Podman.

Anyway, that’s all I’ve got! Use Podman if you want. Don’t use it if you don’t want. I’m not the boss of you. But you said you wanted content on Lemmy, and now you’ve got content on Lemmy. This is all your fault!

^1^ Where “full” is defined as: Not actually full.

^2^ Newer versions of Docker also have some rootless capabilities. But they’ve still got that stinky ol’ daemon.

^2.5^ It’s maybe not quite this simple in practice, because you’ll probably want to run multiple containers under the same Unix account unless you’re really OCD about security and/or have a hatred of the convenience of container networking.

^3^ You can run Podman as root and have many of the same properties as root Docker, but then what’s the point? One less daemon, I guess?

^4^ Where “solution” is defined as: Something that solves the problem while creating five new ones.

^5^ Spoiler: Red Hat’s whole positioning with Podman is like they see it is as a way for buttoned-up corporate devs to run containers locally for development while their “production” is running K8s or whatever. Personally, I don’t care how they position it as long as Podman works well to run my self-hosting shit…

markstos,

I see it as a feature that Podman containers are run via systemd. This makes their management consistent with the other systemd-managed services. Also, Docker does it own things with logs, while with systemd, the logs are managed in a consistent way as well.

Maybe you missed podman generate systemd? Podman will generate the systemd unit files for you.

For me, the two big benefits of podman are being able to run containers via systemd and improved security by being able to run them rootless.

deepdive,

This makes me anxious… How do you cope with all these different technologies… I mean everything is evolving so fast and everyone wants to have his OWN way of doing things… This is messed up ! Right now IT seems a big maze of technologies and nobody seems to be in sync with each other… specially in devOP and Networking…

I don’t know about Podman, but it’s baffling how much you need to know and understand in IT… And If every 3 years you have to relearn everything, it’s a never ending chase of dying and abandoned technologies and a wast of time :/

Just my 2cent, nothing special !

cardes,
@cardes@social.pi.vaduzz.de avatar

@deepdive @witten I think the more you dig the more you find you could learn, probably like every other topic with enough people on it. If you want to keep it simple you mostly still have the chance to just use a little linux machine and put everything there the "old" way. For example: I spend some 3-4 months building a kubernetes stack for my homelab, getting everything to run perfectly, then scrapped everything to rewrite it again with a bit of ansible and a single machine because it justworks

deepdive,

I think the more you dig the more you find you could learn

True, but it’s really frustrating to spend time to learn something that’s maybe going to be useless ? Just look at networking in linux distros between networkd, NetworkManager, netplan, nmtui, nmcli, networkctl, ifupdown… all working in different locations and all having their own way of doing things… This is is fucked up :/

Imagine learning docker’s all subtilities and next year it’s deprecated in favor of another technology with his own flavors and commands… :/

cardes,
@cardes@social.pi.vaduzz.de avatar

@deepdive Yes this can get frustrating if you let it get to you. I‘m 25 years into this and all i learned is how to look stuff up and forgot the rest. I don‘t learn technologies, i try to reduce them to some basic knowledge so i can handle them well enough. Things change all the time and i‘m too lazy to keep track of all that stuff, docker is dead. Its especially true in my actual playground at work where we are using kubernetes. Some of the most complex and fast paced stuff i ever worked with.

lambalicious,

And this is why the trick is learning and focusing the technologies that stick at a “lower level” of the stack, and that have been battle-tested by years or even decades so it’s understood that they won’t just “go away”. Like eg.: learning C or Fortran instead of learning ${niche_language_of_year_20xx}. For the docker bracket for example the near equivalent would be hmmm I’d say (s)chroot.

Then again from here to around 5 years docker will the the schroot of its tech bracket.

witten,

I dunno, I think part of the trick is not learning every single new technology that comes your way. So much of tech these days is just fashion, and you can safely ignore most stuff until there’s a deafening drumbeat bashing down your door. And even then, you should ask if the drumbeat really suits your use cases or if everyone’s in such a fervor over it because it’s fashionable and they’re using it for things it’s not suited for.

Don’t give into the FOMO. Use your judgment. And don’t worry about Podman if what you’re doing now is working!

Scribbd,

I work somewhere that doesn’t have licensing with Docker Inc. And I work on a Mac. With Docker desktop out of the picture, I got some experience with the alternatives. I know this post is about the native implementation and not the VM one, but I just wanted to add my 2 cents:

Alternatives run by me: Podman, Rancher Desktop, Finch

Results:

  • Podman uses a lot more energy on idle than Finch and Rancher. On AVG 4 more Wats on an M1. (Normal idle is about 5W, so 9 almost doubles it cutting greatly in my battery life)
  • Podman and Finch are not compatible with some tools that expect a full docker sock. In my case the AWS CDK and SAM CLI have issues. (Which is fun as Finch is also made by AWS)
  • Finch does not offer a sock at all
  • Finch requires you to recreate the full VM when updated.
  • If you really want to have a drop-in replacement for Docker Desktop, use Rancher Desktop. Rancher lacks in UI and the extension feature. But I never had issues with the sock, as I can run it with containerd.
  • Finch has no UI
  • Podman’s VM has clock drift if you put your machine in sleep. Only solution I found is to reboot the podman VM.
  • Podman allows you to log in the VM with a command. I haven’t found a way on the others.
Avoid8822,

The clock drift issue has been resolved recently: github.com/containers/podman/issues/11541

Scribbd,

That is awesome. I prefer podman, despite what my list might suggest.

stevedave,

I’ve used podman on an RHEL server at work because it works nicely with selinux. I had a hell of a time with rootless containers and network throughput when using an nginx reverse proxy. Made the site painfully slow. Turned out it was due to the slirp4netns rootless networking and MTU size. Just decided to say screw the rootless thing and went rootfull. Next time honestly would just use docker since it’s more common

xtremeownage,

Honestly, I had to use podman at work due to… issues.

Its close enough to docker, that most docker commands will work just fine. You can even alias docker as podman, and things will for the most part, just work.

HOWEVER, there are some big changes and difference. First- podmon creates a systemctl for your containers, for starting them. This- is different, and if you forget to tell it to create the service- then your containers won’t start.

My personal opinion after using it for a few years? I strongly prefer docker. It gives me very few issues. I have spent too much time troubleshooting odd things podman does.

Dark_Arc,
@Dark_Arc@lemmy.world avatar

I’ve tried to switch in the past, but tripped over the differences in Podman vs Docker networking. IIRC Docker is better for creating an isolated network.

I have noticed that Docker doesn’t do the best job at graceful shutdowns (say for automatic installation of updates). I suspect Podman with systemd integration could do much much beter.

kat,

At least podman does not circumvent my firewall (ufw) like docker did. Had to use a workaround to get it to work with docker.

witten,

Podman respects UFW?? That’s awesome to hear.

burningquestion,

I’ve been interested in alternatives to docker for a while now, but considering the way everything is going, tbh, I feel like I’d be more likely to just finally learn kubernetes.

jeena,
@jeena@jemmy.jeena.net avatar

I guess this: “you run a “root” container in your completely unprivileged Unix user and everything just works” sounds like chroot. Also managing your container starts with systemd sounds pretty good to me because this is what systemd is designed for, dependencies between services, etc.

magicsaifa,

Do you think podman could replace Docker Desktop on my Dev machine? Its become so bloated…

witten,

I honestly don’t have any experience with Docker Desktop. Maybe someone else here can weigh in.

EDIT: There is this: podman-desktop.io

lightree,

had a knowledge sharing meeting at work recently on container security. the guy was using podman like a docker cli, but it said "this is only an emulation of docker" - is there any downsides to running podman like this? im very familiar with docker on the command line

loren,

I think calling it an emulation downplays podman. Docker and podman are both container runtimes. Docker came first and is known synonymously with containers, whereas podman is newer and attempts to fix docker’s problems.

One outcome of this is podman chose to match docker’s cli very closely so nobody needs to learn a new cli. You can even put podman on the docker socket so “docker [command]” runs with podman.

amp,

I've switched over my own server last week, using ansible to generate the systemd files, and it worked great. It's just a dozen containers or so.

The only problems I had were with container interdependencies (network-mode=container:x). That didn't work so well with systemd, restarting and updating, but when I used a pod instead these problems all went away.

So I can't say I regret my experience so far. Now I'll be starting to use it at work too, where the user-namespace problem rears its head, but only because we have this very specific, very dumb big lamp dev container that houses apache, sql, redis, and more under one supervisord. That's why we have more than one user in it and frankly that's our own damn fault! When you make proper containers they shouldn't have more than one user in it and then userns=keep-id should work just fine.

So far, I fully recommend podman.

witten,

Using Ansible to spew out systemd service boilerplate seems like a good idea. I’ll have to try that if I can ever give up my Docker Compose security blanket. And I wish you luck with your mega-container Podman conversion. That one sounds like it’ll be… a learning experience.

amp,

I understand very well wanting to stay with the declarative nature of docker-compose. Someone should really build a better podman-compose. (or sooner or later I'll do it myself >_<)

witten,

Do it! I think there’s a market there. Although the “Podman Compose” name is taken and you’ll have to think of something else…

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • uselessserver093
  • Food
  • aaaaaaacccccccce
  • [email protected]
  • test
  • CafeMeta
  • testmag
  • MUD
  • RhythmGameZone
  • RSS
  • dabs
  • Socialism
  • KbinCafe
  • TheResearchGuardian
  • oklahoma
  • feritale
  • SuperSentai
  • KamenRider
  • All magazines