Podman (an alternative to Docker) supports "rootless containers" since around 2019.

what are rootless containers?

• containers that created/run/managed by users w/o admin rights
• protect against engine/runtime/orchestrator compromise

drawbacks?

• slirp4netns (not needed?)

$HOME/.local/share/containers/storage vs /var/lib/containers

Questions

what does conmon do

do they use distinct user namespaces?

I'm running two user-namespace ikiwiki containers. what namespace do they use

├─conmon,2031027 --api-version 1 -c 3353fef59670f6f12de3cbb9094d009fe3cc78796376e28c5313013f777806fd -u... │ └─lighttpd,2031042 -Df /etc/lighttpd/lighttpd.conf ├─conmon,2031121 --api-version 1 -c 55abef9846fb0130ef094c6fe4012317816b4b251d1099dc06c3db2d066bdd23 -u... │ └─lighttpd,2031135 -Df /etc/lighttpd/lighttpd.conf

they seem to be mapped to the same uuid outside

100999 2031042 2031027 0 14:26 ? 00:00:00 /usr/sbin/lighttpd -Df /etc/lighttpd/lighttpd.conf 100999 2031135 2031121 0 14:26 ? 00:00:00 /usr/sbin/lighttpd -Df /etc/lighttpd/lighttpd.conf

the conmon instances are running as uid=jon

containers I am running:

container-calibreweb.service all running as root container-freshrss.service runs apache, cron, sensitive-log.sh as root container-graphite.service whole thing running as root container-navidrome.service

container-nextcloud.service container-paperless.service container-privwiki.service container-redis.service

does podman use a uidmap by default? Can I inspect it?

graphite: USER HUSER COMMAND root root entrypoint root root runsvdir root root runsv root root runsv root root runsv root root runsv root root runsv root root runsv root root runsv root root runsv root root runsv root root runsv root root runsv root root nginx root root crond root root gunicorn root root tee root root tee root root tee root root python3 root root statsd /opt/sta root root tee root root python3 nginx systemd-network nginx nginx systemd-network nginx nginx systemd-network nginx nginx systemd-network nginx root root gunicorn root root gunicorn root root gunicorn root root gunicorn navidrome: USER HUSER COMMAND 1000 jon navidrome freshrss: USER HUSER COMMAND root root apache2 root root cron root root sensitive-log.s root root sed www-data www-data apache2 www-data www-data apache2 www-data www-data apache2 www-data www-data apache2 www-data www-data apache2 www-data www-data apache2 redis: USER HUSER COMMAND redis systemd-coredump redis-server privwiki: USER HUSER COMMAND ikiwiki jon lighttpd paperless: USER HUSER COMMAND root root supervisord paperless jon gunicorn: maste paperless jon [celeryd: celer paperless jon [celery beat] - paperless jon python3 paperless jon gunicorn: worke paperless jon [celeryd: celer calibreweb: USER HUSER COMMAND root root s6-svscan root root s6-supervise root root s6-linux-init-s root root s6-supervise root root s6-supervise root root s6-supervise root root s6-supervise root root s6-ipcserverd abc 911 python3 root root bash root root sleep nextcloud: USER HUSER COMMAND root root apache2 www-data www-data apache2 www-data www-data apache2 www-data www-data apache2 www-data www-data apache2 www-data www-data apache2 www-data www-data apache2 www-data www-data apache2 www-data www-data apache2 www-data www-data apache2 www-data www-data apache2

no uidmapping argh

simplifying my investigations down to uid mapping and the like.

sudo podman top freshrss user,pid,huser,hpid USER PID HUSER HPID root 1 root 3632

sudo ls -l /proc/3632/ns

sudo lsns -t user
        NS TYPE  NPROCS    PID USER  COMMAND
4026531837 user     272      1 root  /sbin/init
4026532989 user       1 223410 uuidd └─/usr/sbin/uuidd --socket-activation

there's only two user namespaces on the system, so podman is not using them yet.

what's uuidd? a local daemon, systemd has PrivateUsers=yes hence the namespace!

which container to experiment on? either privwiki or paperless, neither important

privwiki /var/lib/containers/storage/volumes/privwiki/_data

if I use --userns=auto, the container thinks its running as uid=1000 (can't be sure what is actually chosen) but it cannot read the volume (mapped to nobody)

using a fresh volume, it gets initialised to a big uid (2147484647), but is this consistent? can I rely on it?

second problem: /dev/stderr not writeable without -t, so lighttpd dies

== 2025-05-29 ==

the /dev/stderr thing, that's weird. is it a phobos specific issue? can I recreate it?

there's no root entry in /etc/subuid, but there is (now) a containers entry containers:2147483647:2147483648

podman run -d --replace --name privwiki --userns=auto quay.io/jdowland/opinionated-ikiwiki:latest bash -c 'date > /dev/stderr'

privwiki 9b610a64e032207eed690f18dd4ce59546fe8d810c694ccdd0f74ccdfb1225d0 root@phobos:~# podman logs 9b610a bash: line 1: /dev/stderr: Permission denied

reproduced locally:

podman run -d --replace --name date --userns=auto date bash -c 'id -u; stat /proc/self/fd/2; date >> /proc/self/fd/2'

2e6fc313bb49d19222e45447a561942e79f1288baaecc50b765e562838351a40

carbyne ▶;podman logs 2e6fc

1000

File: /proc/self/fd/2 -> pipe:[24066627]

Size: 64 Blocks: 0 IO Block: 1024 symbolic link

Device: 0,111 Inode: 24070420 Links: 1

Access: (0300/l-wx------) Uid: ( 1000/ jon) Gid: ( 0/ root)

Access: 2025-05-29 13:47:31.763178125 +0000

Modify: 2025-05-29 13:47:31.763178125 +0000

Change: 2025-05-29 13:47:31.763178125 +0000

Birth: -

bash: line 1: /proc/self/fd/2: Permission denied

carbyne ▶;podman run -d --replace --name date --userns=auto date bash -c 'id -u; stat /proc/self/fd/2; date >&2'

617daffb70ec0e4e1f314407530e72723f448d1de9fd7781b4013da545a80599

carbyne ▶;podman logs 617daff 1000 File: /proc/self/fd/2 -> pipe:[24011709] Size: 64 Blocks: 0 IO Block: 1024 symbolic link Device: 0,110 Inode: 24071393 Links: 1 Access: (0300/l-wx------) Uid: ( 1000/ jon) Gid: ( 0/ root) Access: 2025-05-29 13:47:55.659164875 +0000 Modify: 2025-05-29 13:47:55.659164875 +0000 Change: 2025-05-29 13:47:55.659164875 +0000 Birth: - Thu May 29 13:47:55 UTC 2025

== 2025-05-30 ==

should a httpd container log to stderr/stdout by default?

apache: (SIGWINCH!) ErrorLog /proc/self/fd/2 CustomLog /proc/self/fd/1 common

yet seems to work as user with default usermapping stuff

;podman top apache huser user hpid pid comm HUSER USER HPID PID COMMAND 1000 root 2174688 1 httpd 100032 www-data 2174702 8 httpd 100032 www-data 2174703 9 httpd 100032 www-data 2174705 11 httpd

refuses to start with --userns=auto. with userns=host:

podman top apache huser user hpid pid comm HUSER USER HPID PID COMMAND 1000 root 2176081 1 httpd 100032 www-data 2176094 8 httpd 100032 www-data 2176095 9 httpd 100032 www-data 2176097 11 httpd 100999 1000 2176355 92 bash

how to interpret this: container user 0 is mapped to host user 1000, so the root process has my user privileges; my uid 1000 in container context is mapped to something 'safe'

is the issue mapping the default container-user (e.g. root) to not-host-me, so it hasn't permissions on /proc/dev/fd/2?

lighttpd:

== 2025-05-30 ==

need to take a step back. Given quadlet is around the corner, does it invoke containers as root? does it do user mapping?

I want to invoke as root (unless quadlet advises against) so stick to testing uid stuff invoked as root

podman quadlet doesn't do any --userns stuff by default you can put UserNS=, User= etc. into the .container (service) file

== 2025-05-30 ==

OK so root-invoked podman, userns=auto, lighttpd breaks: perms oin /dev/setderr. let's try and debug that