Discussion:
tmux/systemd: tmux user service exits on detach/logout
(too old to reply)
David Runge
2017-07-07 13:24:39 UTC
Permalink
Raw Message
Hey all,

I got a very strange behavior with tmux and systemd on my server and
maybe someone knows a way around it.

The way I start tmux is as follows:
Setting TMUX_TMPDIR to my user's XDG_RUNTIME_DIR and starting tmux in a
systemd user service [1]. I do the same for other things, such as
weechat [2]. They both use the same tmux.conf [3].
Lingering is activated for my user, so the enabled user services will
start with the system and stay running, even when it's disconnected...
in theory (and this used to work very well, without a problem for
years).

With some changes in systemd however, this seems to fail now, and I
can't seem to find the reason why. Tmux just exits with "server lost"
once I do a detach.
At first I could only observe this behavior when I had a shared ssh
session to the server (so I thought), that I then quit, or when I
attached and detached the tmux session quickly.
Now I'm getting this nearly every time I detach from the tmux session!

I tried several things by now, and it doesn't seem to make any
difference:
- using legacy cgroups [4], because of maybe being affected by this [5]
- switching to single connections instead of shared connections for ssh
(lowered the amount of random tmux quits!)
- using hardened kernel (just to try another one)
- logging a tmux session, without getting any output, that would
explain, why this happens
- crossing fingers (actually helps, because the bug seems completely
random ;/ )
- using a system scope unit (***@username) [6], getting the same results
- downgrading tmux to >=2.3 (no difference)
- downgrading systemd to <=233-7 (seems to happen less frequently)

For some awkward reason, it seems to have worked again for a short time,
when Arch first switched to systemd 233 a few days back [6], but ever
since it moved to systemd-stable [7], the problem is back (and even
worse than before). Me thinking that might only be due to a "less
frequent occurence" though.

I'm just wondering, how to solve this. Maybe someone has a great idea to
share? It's getting beyond annoying by now and I can't seem to find out,
if it's tmux, systemd or some kernel problem... or all of them together.

David


P.S.: Doesn't seem to happen with screen. I don't want to adapt
everything to screen now though ;)


[1] https://git.sleepmap.de/software/uenv.git/tree/user/tmux.service
[2] https://git.sleepmap.de/software/uenv.git/tree/user/weechat.service
[3] https://git.sleepmap.de/config/dotfiles.git/tree/.tmux.conf
[4] https://www.freedesktop.org/software/systemd/man/systemd.html#systemd.legacy_systemd_cgroup_controller
[5] https://github.com/systemd/systemd/issues/3388
[6] https://wiki.archlinux.org/index.php/Tmux#Autostart_with_systemd
[7] https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/systemd&id=e0ee38b0c90d0a0da7baad4f472f38dd451326a6
[8] https://github.com/systemd/systemd-stable
--
https://sleepmap.de
André Vitor de Lima Matos via arch-general
2017-07-07 13:53:32 UTC
Permalink
Raw Message
Not sure if related, but try setting KillUserProcesses=no in
/etc/systemd/logind.conf

https://askubuntu.com/questions/802189/how-to-run-tmux-screen-with-systemd-230
--
André Vitor de Lima Matos
David Runge
2017-07-07 14:15:39 UTC
Permalink
Raw Message
Hey Andre Vitor,
Post by André Vitor de Lima Matos via arch-general
Not sure if related, but try setting KillUserProcesses=no in
/etc/systemd/logind.conf
https://askubuntu.com/questions/802189/how-to-run-tmux-screen-with-systemd-230
That is an Arch default setting (commented variables in the systemd
config files reflect compile-time defaults).

Check with:
$ loginctl show-session|grep Kill

Also: The issue at hand is not a logind problem (at least not a
login/logout issue), as far as I can see, as it happens on detach from
tmux (but only sometimes).
That one often times disconnects from the machine after a tmux detach
doesn't help to figure out why it happens unfortunately :>

Thanks anyhow!

Best,
David
--
https://sleepmap.de
David Runge
2017-07-12 09:24:31 UTC
Permalink
Raw Message
Hey again,
Post by David Runge
I got a very strange behavior with tmux and systemd on my server and
maybe someone knows a way around it.
Setting TMUX_TMPDIR to my user's XDG_RUNTIME_DIR and starting tmux in a
systemd user service [1]. I do the same for other things, such as
weechat [2]. They both use the same tmux.conf [3].
Lingering is activated for my user, so the enabled user services will
start with the system and stay running, even when it's disconnected...
in theory (and this used to work very well, without a problem for
years).
With some changes in systemd however, this seems to fail now, and I
can't seem to find the reason why. Tmux just exits with "server lost"
once I do a detach.
At first I could only observe this behavior when I had a shared ssh
session to the server (so I thought), that I then quit, or when I
attached and detached the tmux session quickly.
Now I'm getting this nearly every time I detach from the tmux session!
I tried several things by now, and it doesn't seem to make any
- using legacy cgroups [4], because of maybe being affected by this [5]
- switching to single connections instead of shared connections for ssh
(lowered the amount of random tmux quits!)
- using hardened kernel (just to try another one)
- logging a tmux session, without getting any output, that would
explain, why this happens
- crossing fingers (actually helps, because the bug seems completely
random ;/ )
- downgrading tmux to >=2.3 (no difference)
- downgrading systemd to <=233-7 (seems to happen less frequently)
After a week of testing: tmux - in above mentioned setup - works much
more reliable with systemd 233-7.

Maybe any of the package maintainers can comment on it?
Post by David Runge
For some awkward reason, it seems to have worked again for a short time,
when Arch first switched to systemd 233 a few days back [6], but ever
since it moved to systemd-stable [7], the problem is back (and even
worse than before). Me thinking that might only be due to a "less
frequent occurence" though.
So, in conclusion, the issue is most likely related to some changes
applied to the 233 branch in systemd-stable.
Post by David Runge
[1] https://git.sleepmap.de/software/uenv.git/tree/user/tmux.service
[2] https://git.sleepmap.de/software/uenv.git/tree/user/weechat.service
[3] https://git.sleepmap.de/config/dotfiles.git/tree/.tmux.conf
[4] https://www.freedesktop.org/software/systemd/man/systemd.html#systemd.legacy_systemd_cgroup_controller
[5] https://github.com/systemd/systemd/issues/3388
[6] https://wiki.archlinux.org/index.php/Tmux#Autostart_with_systemd
[7] https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/systemd&id=e0ee38b0c90d0a0da7baad4f472f38dd451326a6
[8] https://github.com/systemd/systemd-stable
Best,
David
--
https://sleepmap.de
David Runge
2017-08-11 08:17:57 UTC
Permalink
Raw Message
Hey all,

as follow up on this (probably not so interesting thread for too many),
I have tested systemd 234 successfully.
The weechat instance within tmux (or tmux itself) still ends
sporadically, but not the main tmux user session.
So, in conclusion, whatever has been broken in systemd 233, was probably
fixed again in 234.

Best,
David
--
https://sleepmap.de
Ismael Bouya
2017-09-25 08:16:26 UTC
Permalink
Raw Message
Post by David Runge
as follow up on this (probably not so interesting thread for too many),
I have tested systemd 234 successfully.
The weechat instance within tmux (or tmux itself) still ends
sporadically, but not the main tmux user session.
So, in conclusion, whatever has been broken in systemd 233, was probably
fixed again in 234.
Hey David and all,
I'm in 234.11-8 and I still have this issue. What did you end up with
during your tests?
I do have KillUserProcesses=no and lingering for my user (it was working
for ages without issues before this summer for me too), and made other
various tests among the ones you mention, but it seems like you have the
right combination.

Kind regards,
--
Ismael
David Runge
2017-09-25 12:39:42 UTC
Permalink
Raw Message
Hi Ismael,
Post by Ismael Bouya
Post by David Runge
as follow up on this (probably not so interesting thread for too many),
I have tested systemd 234 successfully.
The weechat instance within tmux (or tmux itself) still ends
sporadically, but not the main tmux user session.
So, in conclusion, whatever has been broken in systemd 233, was probably
fixed again in 234.
Hey David and all,
I'm in 234.11-8 and I still have this issue. What did you end up with
during your tests?
I do have KillUserProcesses=no and lingering for my user (it was working
for ages without issues before this summer for me too), and made other
various tests among the ones you mention, but it seems like you have the
right combination.
Well, I'm sad to tell you, that I'm experiencing the issues again with
234.11-9, but didn't with 234.11-8 or 234.11-6.

This is getting pretty random, untracable and annoying... :(
Especially, as there is no "real change"[1] between the last pkgrels...

Best,
David

[1] https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/systemd&id=dd32accff1ee7e6dfad0874ec04732ff4f695fa6
--
https://sleepmap.de
Ismael Bouya
2017-09-25 13:33:31 UTC
Permalink
Raw Message
Post by David Runge
Well, I'm sad to tell you, that I'm experiencing the issues again with
234.11-9, but didn't with 234.11-8 or 234.11-6.
This is getting pretty random, untracable and annoying... :(
Especially, as there is no "real change"[1] between the last pkgrels...
Thanks. I searched a little longer and found that:
https://github.com/tmux/tmux/issues/1001

So we can now cross fingers and wait until 2.6...

In the meantime, if you keep at least one session open all the time, the
problem is much less frequent (In my case, I have a mosh session open
all the time from my phone to the server, which kind of helps). Still
annoying, but less frequent.

Kind regards,
--
Ismael
Eli Schwartz
2017-09-25 15:28:07 UTC
Permalink
Raw Message
Post by Ismael Bouya
Post by David Runge
Well, I'm sad to tell you, that I'm experiencing the issues again with
234.11-9, but didn't with 234.11-8 or 234.11-6.
This is getting pretty random, untracable and annoying... :(
Especially, as there is no "real change"[1] between the last pkgrels...
https://github.com/tmux/tmux/issues/1001
So we can now cross fingers and wait until 2.6...
In the meantime, if you keep at least one session open all the time, the
problem is much less frequent (In my case, I have a mosh session open
all the time from my phone to the server, which kind of helps). Still
annoying, but less frequent.
Bugreport opened on our tracker, so this fix may get backported. ;)

https://bugs.archlinux.org/task/55746
--
Eli Schwartz
Ismael Bouya
2017-09-25 15:36:07 UTC
Permalink
Raw Message
Post by Eli Schwartz
Bugreport opened on our tracker, so this fix may get backported. ;)
https://bugs.archlinux.org/task/55746
I didn’t know this was "possible", thanks for reporting it :)
--
Ismael
Eli Schwartz
2017-09-25 16:01:51 UTC
Permalink
Raw Message
Post by Ismael Bouya
Post by Eli Schwartz
Bugreport opened on our tracker, so this fix may get backported. ;)
https://bugs.archlinux.org/task/55746
I didn’t know this was "possible", thanks for reporting it :)
What... I didn't do anything, eworm opened that. :p
I do of course watch all new bugs (it's my job!) and therefore find out
about these things.

Yes, it is possible to request a backport for bugs if the fix is
upstream, though the maintainer may not always agree that the
"something" in question merits doing a backport especially if a release
is expected soon.
--
Eli Schwartz
Loading...