Discussion:
tmux/systemd: tmux user service exits on detach/logout
(too old to reply)
David Runge
2017-07-07 13:24:39 UTC
Permalink
Raw Message
Hey all,

I got a very strange behavior with tmux and systemd on my server and
maybe someone knows a way around it.

The way I start tmux is as follows:
Setting TMUX_TMPDIR to my user's XDG_RUNTIME_DIR and starting tmux in a
systemd user service [1]. I do the same for other things, such as
weechat [2]. They both use the same tmux.conf [3].
Lingering is activated for my user, so the enabled user services will
start with the system and stay running, even when it's disconnected...
in theory (and this used to work very well, without a problem for
years).

With some changes in systemd however, this seems to fail now, and I
can't seem to find the reason why. Tmux just exits with "server lost"
once I do a detach.
At first I could only observe this behavior when I had a shared ssh
session to the server (so I thought), that I then quit, or when I
attached and detached the tmux session quickly.
Now I'm getting this nearly every time I detach from the tmux session!

I tried several things by now, and it doesn't seem to make any
difference:
- using legacy cgroups [4], because of maybe being affected by this [5]
- switching to single connections instead of shared connections for ssh
(lowered the amount of random tmux quits!)
- using hardened kernel (just to try another one)
- logging a tmux session, without getting any output, that would
explain, why this happens
- crossing fingers (actually helps, because the bug seems completely
random ;/ )
- using a system scope unit (***@username) [6], getting the same results
- downgrading tmux to >=2.3 (no difference)
- downgrading systemd to <=233-7 (seems to happen less frequently)

For some awkward reason, it seems to have worked again for a short time,
when Arch first switched to systemd 233 a few days back [6], but ever
since it moved to systemd-stable [7], the problem is back (and even
worse than before). Me thinking that might only be due to a "less
frequent occurence" though.

I'm just wondering, how to solve this. Maybe someone has a great idea to
share? It's getting beyond annoying by now and I can't seem to find out,
if it's tmux, systemd or some kernel problem... or all of them together.

David


P.S.: Doesn't seem to happen with screen. I don't want to adapt
everything to screen now though ;)


[1] https://git.sleepmap.de/software/uenv.git/tree/user/tmux.service
[2] https://git.sleepmap.de/software/uenv.git/tree/user/weechat.service
[3] https://git.sleepmap.de/config/dotfiles.git/tree/.tmux.conf
[4] https://www.freedesktop.org/software/systemd/man/systemd.html#systemd.legacy_systemd_cgroup_controller
[5] https://github.com/systemd/systemd/issues/3388
[6] https://wiki.archlinux.org/index.php/Tmux#Autostart_with_systemd
[7] https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/systemd&id=e0ee38b0c90d0a0da7baad4f472f38dd451326a6
[8] https://github.com/systemd/systemd-stable
--
https://sleepmap.de
André Vitor de Lima Matos via arch-general
2017-07-07 13:53:32 UTC
Permalink
Raw Message
Not sure if related, but try setting KillUserProcesses=no in
/etc/systemd/logind.conf

https://askubuntu.com/questions/802189/how-to-run-tmux-screen-with-systemd-230
--
André Vitor de Lima Matos
David Runge
2017-07-07 14:15:39 UTC
Permalink
Raw Message
Hey Andre Vitor,
Post by André Vitor de Lima Matos via arch-general
Not sure if related, but try setting KillUserProcesses=no in
/etc/systemd/logind.conf
https://askubuntu.com/questions/802189/how-to-run-tmux-screen-with-systemd-230
That is an Arch default setting (commented variables in the systemd
config files reflect compile-time defaults).

Check with:
$ loginctl show-session|grep Kill

Also: The issue at hand is not a logind problem (at least not a
login/logout issue), as far as I can see, as it happens on detach from
tmux (but only sometimes).
That one often times disconnects from the machine after a tmux detach
doesn't help to figure out why it happens unfortunately :>

Thanks anyhow!

Best,
David
--
https://sleepmap.de
David Runge
2017-07-12 09:24:31 UTC
Permalink
Raw Message
Hey again,
Post by David Runge
I got a very strange behavior with tmux and systemd on my server and
maybe someone knows a way around it.
Setting TMUX_TMPDIR to my user's XDG_RUNTIME_DIR and starting tmux in a
systemd user service [1]. I do the same for other things, such as
weechat [2]. They both use the same tmux.conf [3].
Lingering is activated for my user, so the enabled user services will
start with the system and stay running, even when it's disconnected...
in theory (and this used to work very well, without a problem for
years).
With some changes in systemd however, this seems to fail now, and I
can't seem to find the reason why. Tmux just exits with "server lost"
once I do a detach.
At first I could only observe this behavior when I had a shared ssh
session to the server (so I thought), that I then quit, or when I
attached and detached the tmux session quickly.
Now I'm getting this nearly every time I detach from the tmux session!
I tried several things by now, and it doesn't seem to make any
- using legacy cgroups [4], because of maybe being affected by this [5]
- switching to single connections instead of shared connections for ssh
(lowered the amount of random tmux quits!)
- using hardened kernel (just to try another one)
- logging a tmux session, without getting any output, that would
explain, why this happens
- crossing fingers (actually helps, because the bug seems completely
random ;/ )
- downgrading tmux to >=2.3 (no difference)
- downgrading systemd to <=233-7 (seems to happen less frequently)
After a week of testing: tmux - in above mentioned setup - works much
more reliable with systemd 233-7.

Maybe any of the package maintainers can comment on it?
Post by David Runge
For some awkward reason, it seems to have worked again for a short time,
when Arch first switched to systemd 233 a few days back [6], but ever
since it moved to systemd-stable [7], the problem is back (and even
worse than before). Me thinking that might only be due to a "less
frequent occurence" though.
So, in conclusion, the issue is most likely related to some changes
applied to the 233 branch in systemd-stable.
Post by David Runge
[1] https://git.sleepmap.de/software/uenv.git/tree/user/tmux.service
[2] https://git.sleepmap.de/software/uenv.git/tree/user/weechat.service
[3] https://git.sleepmap.de/config/dotfiles.git/tree/.tmux.conf
[4] https://www.freedesktop.org/software/systemd/man/systemd.html#systemd.legacy_systemd_cgroup_controller
[5] https://github.com/systemd/systemd/issues/3388
[6] https://wiki.archlinux.org/index.php/Tmux#Autostart_with_systemd
[7] https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/systemd&id=e0ee38b0c90d0a0da7baad4f472f38dd451326a6
[8] https://github.com/systemd/systemd-stable
Best,
David
--
https://sleepmap.de
David Runge
2017-08-11 08:17:57 UTC
Permalink
Raw Message
Hey all,

as follow up on this (probably not so interesting thread for too many),
I have tested systemd 234 successfully.
The weechat instance within tmux (or tmux itself) still ends
sporadically, but not the main tmux user session.
So, in conclusion, whatever has been broken in systemd 233, was probably
fixed again in 234.

Best,
David
--
https://sleepmap.de
Loading...