Discussion:
Arch pkg user and group IDs?
(too old to reply)
Hauke Fath
2016-11-27 12:58:00 UTC
Permalink
Hi,

what are Arch's conventions for the numeric ranges of system (package)
user and grop IDs? And can the range be limited?

I am working up to switching a few dozen Debian clients to Arch. They
have user homes on NFS, and authenticate through NIS. So far, nis
distributes user and group IDs > 500, making anything lower a local /
system ID. After installing Arch, I see

% awk -F: '{ if ( $3 > 500 ) print }' < /etc/passwd
systemd-journal-remote:x:999:999:systemd Journal Remote:/:/sbin/nologin
systemd-journal-upload:x:998:998:systemd Journal Upload:/:/sbin/nologin
systemd-coredump:x:997:997:systemd Core Dumper:/:/sbin/nologin
% awk -F: '{ if ( $3 > 500 ) print }' < /etc/group
systemd-journal-remote:x:999:
systemd-journal-upload:x:998:
systemd-coredump:x:997:
lightdm:x:620:
%

which collide with NIS users.[1]

My question: Do Arch packages come with hardcoded numerical user/group
IDs? If not, is there a way to limit the numeric range of system IDs? I
guess I would still have to reinstall the distribution.

The alternative - a flag day to change NIS user and group IDs as well
as on 4 TB file storage and local storage on 50 client machines -
wouldn't be pretty. And I don't really have the time, either.

Cheerio,
hauke


[1] lightdm appears to be a pathological case: It hardcodes both name
and ID of its user and group, and installation plows on witjout
flagging an error even after failing to register the user ID.
--
The ASCII Ribbon Campaign Hauke Fath
() No HTML/RTF in email Institut für Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
Respect for open standards Ruf +49-6151-16-21344
Martin Kühne via arch-general
2016-11-27 13:52:25 UTC
Permalink
First of all, any UID less than 1000 is generally considered a system
user/group, I would suspect this tradition or a variation thereof
comes from elder days.
Yes, arch system users are numbered automatically, but the groups and
users you listed are a bit of special snowflakes for systemd. I'm not
sure whether remapping them would be possible.

What I would recommend to you is to use another range further beyond
1000 for your network users and groups because with regard to what we
established in the first paragraph, those don't seem to qualify as
system users IMO. And, just to mention it for completness, there is
/etc/login.defs where you can theoretically adjust those numbers.

cheers!
mar77i
Hauke Fath
2016-11-27 17:27:17 UTC
Permalink
Post by Martin Kühne via arch-general
What I would recommend to you is to use another range further beyond
1000 for your network users and groups because with regard to what we
established in the first paragraph, those don't seem to qualify as
system users IMO.
Well, since we are talking about an existing system with a few dozen
users and more machines, plus a few servers, moving existing users and
groups is a last resort, and one that will create unhappiness and bad
fallout. I would rather not go there.
Post by Martin Kühne via arch-general
And, just to mention it for completness, there is
/etc/login.defs where you can theoretically adjust those numbers.
This

<snip>
# Min/max values for automatic uid selection in useradd
#
UID_MIN 1000
UID_MAX 60000
# System accounts
SYS_UID_MIN 500
SYS_UID_MAX 999
</snip>

might actually help, depending on how well it is enforced. Thanks!

Cheerio,
hauke
--
The ASCII Ribbon Campaign Hauke Fath
() No HTML/RTF in email Institut für Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
Respect for open standards Ruf +49-6151-16-21344
Eli Schwartz via arch-general
2016-11-27 20:49:33 UTC
Permalink
Post by Hauke Fath
Post by Martin Kühne via arch-general
And, just to mention it for completness, there is
/etc/login.defs where you can theoretically adjust those numbers.
This
<snip>
# Min/max values for automatic uid selection in useradd
#
UID_MIN 1000
UID_MAX 60000
# System accounts
SYS_UID_MIN 500
SYS_UID_MAX 999
</snip>
might actually help, depending on how well it is enforced. Thanks!
"enforced"? It is the configuration file for useradd. Anything not
explicitly hardcoded in the UID/GID database (or hardcoded but not in
the database) will respect the useradd configuration (when you reinstall
Arch and all those users are created from scratch again).

Although really, whatever distribution was running on your NFS server
shouldn't be configuring for users with UIDs below 1000 -- a network is
exactly the wrong place to be allowing UIDs that can clash with other
distros' UID reservations.
It might not be a bad idea to report that as a bug.
--
Eli Schwartz
Hauke Fath
2016-11-27 21:46:07 UTC
Permalink
Post by Eli Schwartz via arch-general
[login.defs] might actually help, depending on how well it is
enforced. Thanks!
"enforced"? It is the configuration file for useradd. Anything not
explicitly hardcoded in the UID/GID database (or hardcoded but not in
the database)
... like lightdm?
Post by Eli Schwartz via arch-general
will respect the useradd configuration (when you reinstall
Arch and all those users are created from scratch again).
Although really, whatever distribution was running on your NFS server
shouldn't be configuring for users with UIDs below 1000 -- a network is
exactly the wrong place to be allowing UIDs that can clash with other
distros' UID reservations.
Well, there's
<http://unix.stackexchange.com/questions/101313/what-are-the-dangers-of-creating-a-normal-user-with-uid-500>
which discusses OS distributions' conventions of where to start
non-system uids/gids.

The installation in question is about 13 years old, and it has been
merged a while back with a database that was even older. So the uid
1000 border that Arch uses (and also Debian, according to the link
above, although at least Debian 7 is counting upward, and the highest
system uid on our systems is 118), is by no means universal.

I guess the average distro maintainer doesn't work in a larger,
historically grown network...
Post by Eli Schwartz via arch-general
It might not be a bad idea to report that as a bug.
It was an administrative decision, back more than ten years ago, when
500 IDs appeared to be enough for everyone. ;)

[
systemd-journal-remote:x:999:
systemd-journal-upload:x:998:
systemd-coredump:x:997:
]
Post by Eli Schwartz via arch-general
AFAIK those systemd users/groups are generated by sysusers.d
Aahh! Thanks for the missing puzzle piece. I guess I can find my way
from there.

Cheerio,
Hauke
--
The ASCII Ribbon Campaign Hauke Fath
() No HTML/RTF in email Institut für Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
Respect for open standards Ruf +49-6151-16-21344
Eli Schwartz via arch-general
2016-11-27 21:04:22 UTC
Permalink
Post by Martin Kühne via arch-general
Yes, arch system users are numbered automatically, but the groups and
users you listed are a bit of special snowflakes for systemd. I'm not
sure whether remapping them would be possible.
AFAIK those systemd users/groups are generated by sysusers.d, which
doesn't declare specific UIDs. So I assume that is read from
/etc/login.defs, besides which I don't see how that would otherwise work
on distros that use lower max values.
--
Eli Schwartz
Eli Schwartz via arch-general
2016-11-28 00:03:02 UTC
Permalink
Post by Eli Schwartz via arch-general
Post by Martin Kühne via arch-general
Yes, arch system users are numbered automatically, but the groups
and users you listed are a bit of special snowflakes for systemd.
I'm not sure whether remapping them would be possible.
AFAIK those systemd users/groups are generated by sysusers.d, which
doesn't declare specific UIDs. So I assume that is read from
/etc/login.defs, besides which I don't see how that would otherwise
work on distros that use lower max values.
Actually, the systemd "eevulz maintainers" meme strikes again. Or at
least, a rather bizarre design choice:
https://github.com/systemd/systemd/issues/3855
Post by Eli Schwartz via arch-general
I am strongly against this. Sorry. These defines should be chosen by
the distro vendor, not by users. And it's not realistic to change
these definitions during runtime anyway (as you cannot reasonably
move the IDs allocated during installation).
Quite frankly, these settings should never have been in /etc as this
suggests all would be fine with randomly changing the values for
this, but this cannot work. Sorry.
Completely missing the forest for the trees...

So clearly, the systemd way is to install an Arch system/chroot, change
the defaults, and re-roll your systemd packages for use in a brand-new
installation, then repeat *in perpetuity* , if you intend to deviate *at
install time* from the user-configurable choices of the distro
maintainers. So much for end-user configurability of systems, and here I
thought it would be as simple as initially populating that file on a
blank partition.

I don't know if that is evil, I simply do not know what the logic is at
all. This breaks systemd-sysuser's promised holy grail of declarative
system users -- the "imperative" is still there, but moved, and hidden
inside the binary that translates those files.
The user/package wants certain users, the system announces what a user
should look like, and then systemd stomps all over that by memorizing
how it created users the first time (regardless of which hardware that
was), and then just doing that again.
--
Eli Schwartz
Eli Schwartz via arch-general
2016-11-27 13:55:48 UTC
Permalink
Post by Hauke Fath
Hi,
what are Arch's conventions for the numeric ranges of system (package)
user and grop IDs? And can the range be limited?
I am working up to switching a few dozen Debian clients to Arch. They
have user homes on NFS, and authenticate through NIS. So far, nis
distributes user and group IDs > 500, making anything lower a local /
system ID. After installing Arch, I see
By convention, Arch expects user UIDs to be greater than 1000.

There is a list of explicitly reserved system UIDs by repo packages
here: https://wiki.archlinux.org/index.php/DeveloperWiki:UID_/_GID_Database
--
Eli Schwartz
Hauke Fath
2016-11-27 17:31:01 UTC
Permalink
Post by Eli Schwartz via arch-general
By convention, Arch expects user UIDs to be greater than 1000.
I understand that.
Post by Eli Schwartz via arch-general
There is a list of explicitly reserved system UIDs by repo packages
here: https://wiki.archlinux.org/index.php/DeveloperWiki:UID_/_GID_Database
Which, apart from qmail which is easily avoided, lists only IDs below
500.
But it looks like /etc/login.defs additionally reserves a range of
system IDs.

Cheerio,
hauke
--
The ASCII Ribbon Campaign Hauke Fath
() No HTML/RTF in email Institut für Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
Respect for open standards Ruf +49-6151-16-21344
Leonid Isaev
2016-11-28 02:16:56 UTC
Permalink
Post by Hauke Fath
Post by Eli Schwartz via arch-general
By convention, Arch expects user UIDs to be greater than 1000.
I understand that.
There is no such think as "arch expects". It is a default setting in
/etc/login.defs that is being used by systemd because it is likely that a devs
don't change login.defs before building systemd...
Post by Hauke Fath
Post by Eli Schwartz via arch-general
There is a list of explicitly reserved system UIDs by repo packages
here: https://wiki.archlinux.org/index.php/DeveloperWiki:UID_/_GID_Database
Which, apart from qmail which is easily avoided, lists only IDs below
500.
But it looks like /etc/login.defs additionally reserves a range of
system IDs.
I strongly suspect that the above wikipage is deprecated. Most likely all arch
packages will move to sysuser.d sooner or later.

BTW, have you read the manpage of sysuser.d? You can override those "default"
IDs by specifying your own, just need to change a few files (on our NIS master
there are only snippets supplied with systemd)...

But out of curiosity, why is it difficult to change user IDs on all files? I
assume that you control the storage? Isn't it just a chown -R away? For
example, for our NIS passwd/shadow map we use 6-digit IDs...

Cheers,
--
Leonid Isaev
GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4
C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
Eli Schwartz via arch-general
2016-11-28 02:50:22 UTC
Permalink
Post by Leonid Isaev
There is no such think as "arch expects". It is a default setting in
/etc/login.defs that is being used by systemd because it is likely that a devs
don't change login.defs before building systemd...
Well, packages can have files that need to have a specific system user
ownership. That is why the UID/GID database exists, right? Because the
UID baked into the *.pkg.tar.xz has to match /etc/passwd, and
systemd-sysusers can't inherently do anything that repetitive useradd +
getent scripting wasn't always capable of.
Post by Leonid Isaev
I strongly suspect that the above wikipage is deprecated. Most likely all arch
packages will move to sysuser.d sooner or later.
I don't see why, again, systemd-sysusers was designed as a declarative
alternative to `useradd -r` and the Wikipage was never deprecated by
useradd...
Post by Leonid Isaev
BTW, have you read the manpage of sysuser.d? You can override those "default"
IDs by specifying your own, just need to change a few files (on our NIS master
there are only snippets supplied with systemd)...
Do you mean, by manually specifying a hardcoded UID for each user???
--
Eli Schwartz
Leonid Isaev
2016-11-28 03:03:32 UTC
Permalink
Post by Eli Schwartz via arch-general
Post by Leonid Isaev
There is no such think as "arch expects". It is a default setting in
/etc/login.defs that is being used by systemd because it is likely that a devs
don't change login.defs before building systemd...
Well, packages can have files that need to have a specific system user
ownership. That is why the UID/GID database exists, right? Because the
UID baked into the *.pkg.tar.xz has to match /etc/passwd, and
systemd-sysusers can't inherently do anything that repetitive useradd +
getent scripting wasn't always capable of.
For example, dnsmasq ships /usr/lib/sysusers.d/dnsmasq.conf which contains 'u
dnsmasq - "dnsmasq daemon" /' and on my system the user dnsmasq has
(randomly-generated) ID = 997. Such packages won't have any files owned by a
non-root user because they don't know the UID.
Post by Eli Schwartz via arch-general
Post by Leonid Isaev
I strongly suspect that the above wikipage is deprecated. Most likely all arch
packages will move to sysuser.d sooner or later.
I don't see why, again, systemd-sysusers was designed as a declarative
alternative to `useradd -r` and the Wikipage was never deprecated by
useradd...
Maybe, but currently most config snippets in /usr/lib/sysusers.d/ do not
specify UID (except qemu.conf shipped with qemu). So I assume those get
assigned randomly.
Post by Eli Schwartz via arch-general
Post by Leonid Isaev
BTW, have you read the manpage of sysuser.d? You can override those "default"
IDs by specifying your own, just need to change a few files (on our NIS master
there are only snippets supplied with systemd)...
Do you mean, by manually specifying a hardcoded UID for each user???
Yes, why not? You can override files in /usr/lib/sysusers.d with files in
/etc/sysusers.d having identical names, no? For example, on my workstation,
there are only 23 lines in total where UID need to be changed below 500.

Of course, this needs to be done for all Arch machines. That is why I think
that changing ownership in NFS share is a better idea...

Cheers,
--
Leonid Isaev
GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4
C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
Eli Schwartz via arch-general
2016-11-28 03:32:38 UTC
Permalink
Post by Leonid Isaev
Post by Eli Schwartz via arch-general
Well, packages can have files that need to have a specific system
user ownership. That is why the UID/GID database exists, right?
Because the UID baked into the *.pkg.tar.xz has to match
/etc/passwd, and systemd-sysusers can't inherently do anything that
repetitive useradd + getent scripting wasn't always capable of.
For example, dnsmasq ships /usr/lib/sysusers.d/dnsmasq.conf which
contains 'u dnsmasq - "dnsmasq daemon" /' and on my system the user
dnsmasq has (randomly-generated) ID = 997. Such packages won't have
any files owned by a non-root user because they don't know the UID.
Let's rephrase that. Some packages contain software which requires
runtime files owned by a non-root user associated with the software.

The fact that they don't know the UID during makepkg is precisely the
reason why the UID/GID database exists...

dnsmasq is an irrelevant example of software which has absolutely
nothing to do with the topic at hand.
Post by Leonid Isaev
Maybe, but currently most config snippets in /usr/lib/sysusers.d/ do
not specify UID (except qemu.conf shipped with qemu). So I assume
those get assigned randomly.
Again, that is the whole point. useradd assigns them randomly, and
sysusers assigns them randomly. But there is a range of UIDs reserved
for stuff that needs a non-random UID (because the UID needs to be known
during makepkg for packaging). So yes, Arch expects hardcoded UIDs to be
available below 500, which means system user UIDs can be and should be
allocated from 500-999 (as opposed to, say, preemptively clobbering any
potentially reserved UIDs below that).
Post by Leonid Isaev
Yes, why not? You can override files in /usr/lib/sysusers.d with
files in /etc/sysusers.d having identical names, no? For example, on
my workstation, there are only 23 lines in total where UID need to be
changed below 500.
Because that is the classic definition of working around buggy software?
useradd/sysusers was carefully designed to be able to dynamically
allocate UIDs and now you have to copy and override every single
sysusers definition because systemd won't let you configure the
allocation range...
Post by Leonid Isaev
Of course, this needs to be done for all Arch machines.
Plus that.
Post by Leonid Isaev
That is why I think that changing ownership in NFS share is a better
idea...
Well, depending on the number of files...
Anyway, modifying a single file in a very predictable manner on multiple
*new* machines is an even better idea. Plus, Hauke Fath indicated that
there were also plenty of files scattered across 50 other existing machines.
--
Eli Schwartz
Leonid Isaev
2016-11-28 04:09:59 UTC
Permalink
Post by Eli Schwartz via arch-general
Post by Leonid Isaev
Post by Eli Schwartz via arch-general
Well, packages can have files that need to have a specific system
user ownership. That is why the UID/GID database exists, right?
Because the UID baked into the *.pkg.tar.xz has to match
/etc/passwd, and systemd-sysusers can't inherently do anything that
repetitive useradd + getent scripting wasn't always capable of.
For example, dnsmasq ships /usr/lib/sysusers.d/dnsmasq.conf which
contains 'u dnsmasq - "dnsmasq daemon" /' and on my system the user
dnsmasq has (randomly-generated) ID = 997. Such packages won't have
any files owned by a non-root user because they don't know the UID.
Let's rephrase that. Some packages contain software which requires
runtime files owned by a non-root user associated with the software.
The fact that they don't know the UID during makepkg is precisely the
reason why the UID/GID database exists...
Yes, if such packages exist, then of course, they would require hardcoded UID.
Post by Eli Schwartz via arch-general
Post by Leonid Isaev
Yes, why not? You can override files in /usr/lib/sysusers.d with
files in /etc/sysusers.d having identical names, no? For example, on
my workstation, there are only 23 lines in total where UID need to be
changed below 500.
Because that is the classic definition of working around buggy software?
useradd/sysusers was carefully designed to be able to dynamically
allocate UIDs and now you have to copy and override every single
sysusers definition because systemd won't let you configure the
allocation range...
I kinda agree with Lennart Poettering here as to why systemd shouldn't read
login.defs at runtime. Still, dealing with systemd ppl sucks...

I mean once you do change ranges in login.defs, you'll have to fix user
memberships as well, yes?
Post by Eli Schwartz via arch-general
Post by Leonid Isaev
Of course, this needs to be done for all Arch machines.
Plus that.
Not only that, but also updates must be carefully monitored...
Post by Eli Schwartz via arch-general
Post by Leonid Isaev
That is why I think that changing ownership in NFS share is a better
idea...
Well, depending on the number of files...
Anyway, modifying a single file in a very predictable manner on multiple
*new* machines is an even better idea. Plus, Hauke Fath indicated that
there were also plenty of files scattered across 50 other existing machines.
Hmm, I thought all /home's were on an NFS share... But I still don't understand
why is it such a difficult task?

Sure, migrating 10 year old installations this way is not pleasant, but at
least this will be future-proof. Because how do you avoid clashes with
hard-coded UID in packages now?

Cheers,
--
Leonid Isaev
GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4
C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
Hauke Fath
2016-11-28 10:04:53 UTC
Permalink
Post by Leonid Isaev
But out of curiosity, why is it difficult to change user IDs on all files? I
assume that you control the storage? Isn't it just a chown -R away? For
example, for our NIS passwd/shadow map we use 6-digit IDs...
Because... users have files

- on their NFS home
- on public NFS shares
- on a partition of the local harddrive (and not necessarily limited to
one machine)
- on their home on the web server
- on their home on the mailserver
- on their home on the computing cluster

all of which makes a change of user and group id slightly more involved
than a 'chmod -R'. Nothing that couldn't be done, mind you, given
enough round tuits - both for me and my users.

As I said, it would have to be either a flag day (deploy a script with
old-new mapping to all machines involved, lock out users, shut down
services, run script), or piecemeal change (negociate time slot with
user, log them out, annoy other users because you have to temporarily
disable imap and smtp services, run said script). Both would need to be
planned, communicated and negociated, and so take more time than I have.

Cheerio,
Hauke
--
The ASCII Ribbon Campaign Hauke Fath
() No HTML/RTF in email Institut für Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
Respect for open standards Ruf +49-6151-16-21344
Martin Kühne via arch-general
2016-11-28 11:28:23 UTC
Permalink
Hmm, you could do the move per-server, though, at least for the
network services that publicly can report a different UID/GID pair
than is advertised on the file system, which is at least true for NFS.
Did you look into that, already?

cheers!
mar77i
Hauke Fath
2016-11-28 11:46:35 UTC
Permalink
Post by Martin Kühne via arch-general
Hmm, you could do the move per-server, though, at least for the
network services that publicly can report a different UID/GID pair
than is advertised on the file system, which is at least true for NFS.
NFSv4, that is. We use v3, which by design cannot remap IDs - the
server takes the client's information as gospel.
Post by Martin Kühne via arch-general
Did you look into that, already?
Yes... the NFS server is actually the least of my worries.

Cheerio,
hauke
--
The ASCII Ribbon Campaign Hauke Fath
() No HTML/RTF in email Institut für Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
Respect for open standards Ruf +49-6151-16-21344
Leonid Isaev
2016-11-28 22:25:50 UTC
Permalink
Post by Hauke Fath
Post by Leonid Isaev
But out of curiosity, why is it difficult to change user IDs on all files? I
assume that you control the storage? Isn't it just a chown -R away? For
example, for our NIS passwd/shadow map we use 6-digit IDs...
Because... users have files
- on their NFS home
- on public NFS shares
- on a partition of the local harddrive (and not necessarily limited to
one machine)
- on their home on the web server
- on their home on the mailserver
- on their home on the computing cluster
all of which makes a change of user and group id slightly more involved
than a 'chmod -R'. Nothing that couldn't be done, mind you, given
enough round tuits - both for me and my users.
As I said, it would have to be either a flag day (deploy a script with
old-new mapping to all machines involved, lock out users, shut down
services, run script), or piecemeal change (negociate time slot with
user, log them out, annoy other users because you have to temporarily
disable imap and smtp services, run said script). Both would need to be
planned, communicated and negociated, and so take more time than I have.
OK, if this is not an option, then I you have few options, but all of the suck:
1. Just go over your /etc/passwd, /etc/group etc. and manually assign UID/GID
to systemd* users. The hope is that packages won't install new users. Also,
put /usr/lib/sysusers.d in a version control, so you can track changes. And
of course, change your login.defs appropriately. Systemd won't read it, but
other programs might.
2. If this is too hackish for you, then override files in /usr/lib/sysusers.d/
by copying them to /etc/sysusers.d and putting fixed UIDs there. Of course,
/etc/passwd et all need to be fixed manually, as above.
3. Rebuild systemd with a proper login.defs... Keep in mind though, that new
stable releases of systemd are almost always broken in one way or another...
4. Reopen the bugreport mentioned in this thread and try to bring LP back to
earth. Good luck with that though, in my experience it is impossible :)

If I were you, I'd choose (1) but create every new user in a high-numbered IDs,
so hopefully in some time, you can drop the hack.

HTH,
--
Leonid Isaev
GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4
C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
Continue reading on narkive:
Loading...