Discussion:
Arch Linux on servers?
(too old to reply)
M Saunders
2013-07-09 10:13:08 UTC
Permalink
Hi all,

I'm writing a feature about Arch for Linux Format, a UK-based
newsstand Linux magazine. I've been using Arch myself for a while for
testing new app releases, and it's brilliant for that purpose.

I'm still left wondering though: who uses it on production servers? I
mean, the distro's overall simplicity and trimmed-down base
installation are plus points here, but surely a rolling release poses
problems. After installation you just want security and critical bug
fix updates for software, and not major version bumps, right?

www.archserver.org seems to be on hold, and I've also seen this page:
https://wiki.archlinux.org/index.php/Enhancing_Arch_Linux_Stability

which has some useful tips. But it'd be interesting to hear from
people running Arch on production servers, how well it works for them
and what (if any) problems they've faced.

Thanks!
Mike
Chris Down
2013-07-09 10:43:28 UTC
Permalink
Hi Mike,
Post by M Saunders
I'm writing a feature about Arch for Linux Format, a UK-based
newsstand Linux magazine. I've been using Arch myself for a while for
testing new app releases, and it's brilliant for that purpose.
I'm still left wondering though: who uses it on production servers? I
mean, the distro's overall simplicity and trimmed-down base
installation are plus points here, but surely a rolling release poses
problems. After installation you just want security and critical bug
fix updates for software, and not major version bumps, right?
I only use it to manage small production environments (although these are not
corporate deployments). IMO it is suitable for servers in limited cases, where
neither of the following are true:

- The server will be running obscure services with limited eyes-on
- You will be running a lot of services

I ran my entire personal development infrastructure on Arch Linux for a good
while, and only stopped because I've outsourced it all now so there's no need
for the installation in the first place -- that being, a CI, git hosting, HTTP
server, a few other things.
Post by M Saunders
https://wiki.archlinux.org/index.php/Enhancing_Arch_Linux_Stability
The only thing I did was have linux-lts installed in addition to the linux
package. I never had a problem, and I ran that server for years.
Post by M Saunders
which has some useful tips. But it'd be interesting to hear from
people running Arch on production servers, how well it works for them
and what (if any) problems they've faced.
I never had a problem that was due to the packaging, which limits the
opportunities for breakage to upstream (mainly).

That just means you have to have an eye on things. I was subscribed to the
announcement mailing lists for all the stuff I was using (Jenkins, nginx, git,
cgit, et al). If you're running a very complex server then it can become a bit
complicated to go down this road, especially if you're used to your distribution
providing deprecation guidance for you. Generally things don't just "break" on
Arch, there is [testing] after all. If things break, it's usually because people
didn't pay attention to configuration changes or important details prior to
upgrading. If you aren't willing to keep an eye on upstream and on the Arch
mailing lists, it will not end well.

Good luck with the article. I'm not living in the UK any more, else I would
still be buying Linux Format.

Chris
Paladin
2013-07-09 11:18:29 UTC
Permalink
Post by M Saunders
I'm writing a feature about Arch for Linux Format, a UK-based
newsstand Linux magazine. I've been using Arch myself for a while for
testing new app releases, and it's brilliant for that purpose.
I'm still left wondering though: who uses it on production servers? I
mean, the distro's overall simplicity and trimmed-down base
installation are plus points here, but surely a rolling release poses
problems. After installation you just want security and critical bug
fix updates for software, and not major version bumps, right?
Hi,
I was considering using Archlinux on my VPS, but I run into
problems with OpenVZ compatibility :/ It seems that archlinux
is not currently fully supported in OpenVZ which was a dead stop
for me.

I could image to run Arch on testing/staging server, but for production
I would grab Scientific Linux. It's assembled in CERN, so stability and
consistency come with it. Plus it's based on RHEL, so there is lot of
packages available for it (to be fair, I think archlinux + aur may
have more). It's not so cutting edge as arch (php 5.3 instead 5.4, ...)
but for production I prefer stability which Arch can give, but it
definitely costs more (in term of time) then in SL6.

P.

Disclaimer: Arch on both laptops though.
Gaetan Bisson
2013-07-09 11:00:55 UTC
Permalink
Obviously, I am biased, but...
Post by M Saunders
After installation you just want security and critical bug
fix updates for software, and not major version bumps, right?
If you are prepared to stick to a given feature set, maybe. Then, you
might be able to achieve near-absolute stability with a Debian-stable-
like system. Now, in my experience, that makes any upgrade extremely
painful, so I would definitely not recommend this approach to anyone
with an interest in recent software.

On the other hand, Arch's continuous flow of upgrades proves to be quite
reliable, so long as the system is updated with a minimum degree of
attention. Besides, it avoids duplicating upstream work at the distro
level (such as backporting security fixes): such work can never be
perfect, and has indeed been the cause of major problems in the past
(Debian's openssl entropy issue likely being the most famous).

I run two small-scale servers with Arch, which I only upgrade with care
and when I have available time to fix potential problems. My upgrading
frequency goes from a couple of times a week to once a month. About once
a year, I hit an upgrade that is not straightforward, and it takes me an
hour or two to fix arising issue and perform it; that is about it.
--
Gaetan
Paul Gideon Dann
2013-07-09 11:22:23 UTC
Permalink
I'm writing a feature about Arch for Linux Format, a UK-based newsstand
Linux magazine. I've been using Arch myself for a while for testing new
app releases, and it's brilliant for that purpose.
I'm still left wondering though: who uses it on production servers? I
mean, the distro's overall simplicity and trimmed-down base installation
are plus points here, but surely a rolling release poses problems. After
installation you just want security and critical bug fix updates for
software, and not major version bumps, right?
https://wiki.archlinux.org/index.php/Enhancing_Arch_Linux_Stability
which has some useful tips. But it'd be interesting to hear from people
running Arch on production servers, how well it works for them and what
(if any) problems they've faced.
Hi Mike! I am subscribed to Linux Format, and also listen to the TuxRadar
podcasts. Keep up the good work! I am chronically one issue behind, so tell
Graham to feel free to skip a month to save me the pang of guilt I feel
when a new issue lands on our mat.

Now to business:

I run Arch on two servers at work. One is a general-purpose server running
some internal web-apps, server-room monitoring, and source code repository.
The other is a beowulf cluster server that has 36 diskless nodes connected
to it for heavy number-crunching. I haven't seen any significant downtime
for a long time. The recent /usr switch was a little hairy, but went off
without a hitch in the end, although it meant I had to finally switch from
Legacy Grub to Syslinux. I tend to reboot them every month or two, to keep
the kernel up-to-date, but mostly they just keep going.

I've found Arch fantastic for tweaking. Its flexibility and simplicity
was a great help in getting the cluster working (in a way that I actually
understand and can trouble-shoot). Over the last few years, I've invested
quite some time writing maintenance scripts, automation, backup system,
etc. to keep everything ticking over smoothly, and to send me notifications
when things don't seem quite right. It would be a major pain to port all of
that to a new install, or to deal with the breakage of a big upgrade every
year or so.

The downside is that it does take a decent chunk of time to keep everything
well-maintained and up-to-date.

To be fair, the only servers I've ever set up myself have been Arch
servers. I inherited a production FreeBSD server for a while when I
worked briefly for a small web design company, and the strategy there was
basically not to bother updating it at all, as far as I could tell. I have
started worrying more recently that if I move on at some point, Arch may be
too much for whoever takes over.

I'd probably try Debian in future on a new server if the server's task
isn't too exotic. I don't think I'd enjoy it so much myself, but it
would certainly be much more install-and-forget, which sometimes is more
desirable.

Paul

P.S.: Apologies to Gaetan for top-posting on my first attempt. I'm picking
up bad habits from a non-technical mailing list that I recently joined :)
Florian Pritz
2013-07-09 12:24:04 UTC
Permalink
Post by M Saunders
I'm still left wondering though: who uses it on production servers? I
mean, the distro's overall simplicity and trimmed-down base
installation are plus points here, but surely a rolling release poses
problems. After installation you just want security and critical bug
fix updates for software, and not major version bumps, right?
I've seen at least 2 or 3 kernel exploits that were mitigated by newer
kernel versions (which we had, debian didn't). Obviously there have been
other issues which could only be exploited in more recent kernel
versions which didn't affect debian.

Then there are those issues where there is a patch but no new release so
it might not get fixed in arch until the next release (no security team
nor policy for such patches).

In terms of updating breakage it doesn't matter what you use, updating
will eventually result in breakage, but if you know the system well
enough you will have a much easier time fixing it.

I had a case where a few debian servers got upgraded after something
like 1.5 years and spamassassin suddenly used a lot more resources.
Since basically every package jumped lots of versions finding the
package responsible for that was kind of impossible so they just bought
a bunch more servers to deal with the higher load.

On arch you could probably narrow it down and fix the software. Might
not be cheaper and might not be what you want (cool new feature causing
the issue maybe), but at least you aren't left in the dark.

I'm not sure if either distro is more time intensive, I think you will
just spend your time differently. Also investing time in anything will
result in knowledge so I'm not sure if that's a bad thing.

If you don't know what you are doing, don't run a server with arch. But
then you shouldn't be running a server in that case anyway. As Allan
once said: "If you have to ask, then no".

I'd say neither solution (rolling-release vs "stable and secure") is
better, they are just different. Get to know your tool (distro) and
decide for yourself.
Florian Dejonckheere
2013-07-09 17:27:43 UTC
Permalink
Post by Florian Pritz
Post by M Saunders
I'm still left wondering though: who uses it on production servers? I
mean, the distro's overall simplicity and trimmed-down base
installation are plus points here, but surely a rolling release poses
problems. After installation you just want security and critical bug
fix updates for software, and not major version bumps, right?
I've seen at least 2 or 3 kernel exploits that were mitigated by newer
kernel versions (which we had, debian didn't). Obviously there have been
other issues which could only be exploited in more recent kernel
versions which didn't affect debian.
Then there are those issues where there is a patch but no new release so
it might not get fixed in arch until the next release (no security team
nor policy for such patches).
In terms of updating breakage it doesn't matter what you use, updating
will eventually result in breakage, but if you know the system well
enough you will have a much easier time fixing it.
I had a case where a few debian servers got upgraded after something
like 1.5 years and spamassassin suddenly used a lot more resources.
Since basically every package jumped lots of versions finding the
package responsible for that was kind of impossible so they just bought
a bunch more servers to deal with the higher load.
On arch you could probably narrow it down and fix the software. Might
not be cheaper and might not be what you want (cool new feature causing
the issue maybe), but at least you aren't left in the dark.
I'm not sure if either distro is more time intensive, I think you will
just spend your time differently. Also investing time in anything will
result in knowledge so I'm not sure if that's a bad thing.
If you don't know what you are doing, don't run a server with arch. But
then you shouldn't be running a server in that case anyway. As Allan
once said: "If you have to ask, then no".
I'd say neither solution (rolling-release vs "stable and secure") is
better, they are just different. Get to know your tool (distro) and
decide for yourself.
I have ran (home) servers on both Arch Linux and Debian, and found that the
Arch Linux ones require more work to keep it up to date, but offer way more
software (and closer to upstream). Stability is not garantueed however, and
you are responsible for keeping each and every feature working.

Debian, on the other hand, is more stable out of the box and requires less
updating. Its software is nowhere near upstream, though. For example
systemd (if you don't opt for the default outdated sysvinit) is still at
version 88, missing a lot of crucial functionality from the later versions.

Arch can be used as a server distro, but if you prefer low maintenance, use
something else.
Don deJuan
2013-07-09 22:53:29 UTC
Permalink
Post by Florian Dejonckheere
Post by Florian Pritz
Post by M Saunders
I'm still left wondering though: who uses it on production servers? I
mean, the distro's overall simplicity and trimmed-down base
installation are plus points here, but surely a rolling release poses
problems. After installation you just want security and critical bug
fix updates for software, and not major version bumps, right?
I've seen at least 2 or 3 kernel exploits that were mitigated by newer
kernel versions (which we had, debian didn't). Obviously there have been
other issues which could only be exploited in more recent kernel
versions which didn't affect debian.
Then there are those issues where there is a patch but no new release so
it might not get fixed in arch until the next release (no security team
nor policy for such patches).
In terms of updating breakage it doesn't matter what you use, updating
will eventually result in breakage, but if you know the system well
enough you will have a much easier time fixing it.
I had a case where a few debian servers got upgraded after something
like 1.5 years and spamassassin suddenly used a lot more resources.
Since basically every package jumped lots of versions finding the
package responsible for that was kind of impossible so they just bought
a bunch more servers to deal with the higher load.
On arch you could probably narrow it down and fix the software. Might
not be cheaper and might not be what you want (cool new feature causing
the issue maybe), but at least you aren't left in the dark.
I'm not sure if either distro is more time intensive, I think you will
just spend your time differently. Also investing time in anything will
result in knowledge so I'm not sure if that's a bad thing.
If you don't know what you are doing, don't run a server with arch. But
then you shouldn't be running a server in that case anyway. As Allan
once said: "If you have to ask, then no".
I'd say neither solution (rolling-release vs "stable and secure") is
better, they are just different. Get to know your tool (distro) and
decide for yourself.
I have ran (home) servers on both Arch Linux and Debian, and found that the
Arch Linux ones require more work to keep it up to date, but offer way more
software (and closer to upstream). Stability is not garantueed however, and
you are responsible for keeping each and every feature working.
Debian, on the other hand, is more stable out of the box and requires less
updating. Its software is nowhere near upstream, though. For example
systemd (if you don't opt for the default outdated sysvinit) is still at
version 88, missing a lot of crucial functionality from the later versions.
Arch can be used as a server distro, but if you prefer low maintenance, use
something else.
I honestly do not see Arch on a server as really that much "more work".
I run quite a few production servers on Amazon's Ec2 using the arch
image uplinklabs.net provides.

After successful use of a couple not as critical systems, I have now
moved 99% of my servers on there to Arch. The rest have not only due to
me not being done yet, not other reason. The only time it has bitten me
was when there was a change that I knew and read about that I did not
execute properly. So the update failed, image was borked. But that was
my fault for being in a rush and not thinking before pressing enter. The
fix was simple and pretty quick, even made me learn more things about Ec2.

While I see the merit in using more LTS solutions, like Chester I love
the fact that I am always up to date and have the fine grained control.
While I do update more frequently than he does, I am on a once a week
schedule unless there is something major that comes before then.
Chester Wisniewski
2013-07-09 15:16:15 UTC
Permalink
Post by M Saunders
Hi all,
I'm writing a feature about Arch for Linux Format, a UK-based
newsstand Linux magazine. I've been using Arch myself for a while for
testing new app releases, and it's brilliant for that purpose.
I'm still left wondering though: who uses it on production servers? I
mean, the distro's overall simplicity and trimmed-down base
installation are plus points here, but surely a rolling release poses
problems. After installation you just want security and critical bug
fix updates for software, and not major version bumps, right?
https://wiki.archlinux.org/index.php/Enhancing_Arch_Linux_Stability
which has some useful tips. But it'd be interesting to hear from
people running Arch on production servers, how well it works for them
and what (if any) problems they've faced.
Thanks!
Mike
I run two production Arch server boxes. I update them each about twice a
month unless I hear about a vulnerability in a critical internet facing
component on the system (WordPress, etc).

I just make sure to do it on Fridays and Saturdays. If something goes
wrong, it can often take a little more time to back out than a
traditional distribution. As a security professional, I appreciate
always being up to date and having more detailed control over exactly
what code is on my system. I have massively reduced my exposure to flaws
in packages included by the distro that I never needed or even wanted.

Get Linux? Get Arch Linux. Not so sure about Linux? Call Rackspace.

Chester
kachelaqa
2013-07-09 15:11:45 UTC
Permalink
Post by M Saunders
But it'd be interesting to hear from
people running Arch on production servers, how well it works for them
and what (if any) problems they've faced.
There are quite a few threads on the Arch Forum on this topic:

https://bbs.archlinux.org/viewtopic.php?id=162434
https://bbs.archlinux.org/viewtopic.php?pid=1272825#p1272825
David Benfell
2013-07-10 01:27:48 UTC
Permalink
Post by M Saunders
which has some useful tips. But it'd be interesting to hear from
people running Arch on production servers, how well it works for
them and what (if any) problems they've faced.
Mine is pretty small scale as "production" goes. But my overall
experience running Arch on a server (I currently actually have two of
them) for at least a couple years now is that Arch tends to be stable
without doing anything special.

The one time I got burned was on the latest consolidation to /usr/bin,
where some critical postfix file permissions were not preserved--and
the nature of the problem was such that, at least in my situation, it
didn't show up for hours after. The fix was easy enough; postfix has a
command option just for this situation, and a new version of the
postfix package was available within a few hours that I assume would
have fixed the problem by itself had I waited just that length of time
before doing the upgrade.

So the one bit of advice I would offer probably won't be shocking:
whenever an upgrade comes down that seems critical, it is probably
prudent to just wait a day.

- --
David Benfell / ***@parts-unknown.org
Please see https://parts-unknown.org/node/2 for GnuPG information (or
the attachment you don't understand)
Genes Lists
2013-07-10 11:17:51 UTC
Permalink
I have run Redhat, Fedora and Arch servers. HAnds down arch wins for a
server setup.

Change happens - always. Wigth Arch the changes are fed to me in small
chunks - I get to deal with one change at a time (e.g. changing to
systemd) Any given change might be smaller or larger but with Arch I can
focus on dealing with that single change.

With non-rolling releases, every couple of years (or 6 - 9 months for
Fedora) I have to do a fresh install containing all the changes.

Rolling release model is far superior - and as others have pointed out
Arch stays close to upstream and you can keep your systems current which
of course means fewer exploit opportunities for the bad guys.

gene
Genes Lists
2013-07-10 11:21:15 UTC
Permalink
Forgot to say - for server setups, I always test updates on a shadow
machine before applying them to server itself - something that is
prudent policy regardless of which distro you're using.
Sébastien Luttringer
2013-07-10 11:59:07 UTC
Permalink
Post by M Saunders
which has some useful tips. But it'd be interesting to hear from
people running Arch on production servers, how well it works for them
and what (if any) problems they've faced.
I've 9 personal servers running Archlinux (previously under debian)
and I plan to move about ~250 professional hypervisor under Arch this
year.
Let me share the following experiences with you.

1) Use the minimum set of packages
This will save you from updating useless packages and give you a
better view of what your server use.
As there is only few packages, don't rush to update them when there is
major change on it.

2) Do your sysadmin homework
Before updating, check archlinux.org for announcements.
During update read pacman output.[1]
After updating, look for pacsave/pacorig/pacnew files.

Supervise your packages. I use munin with the following plugins[2][3]

3) Use a versioned kernel
One of the most wanted expectation on a server is to avoid reboot.
Arch official kernel is too often updated for a server _and_ cannot be
installed without breaking the running kernel (modules mismatch).
To workaround this I build custom kernels, with the version in the
name[4] and I use a meta package[5] which push new version
automatically and clean the old one. So I can update my server, and at
the next reboot the last kernel will be selected.

4) Detect daemon upgrade
When you update your system, some libraries or binaries can be updated
and your running programs still use the old version.
This give the bad feeling that your software are up-to-date. But it's false.
Of course you can reboot your server to be sure after each update.
It's too long and give the feeling to hunt fly with a tank.

I use the following script[6] which list services (systemd speaking)
which need to be restarted.

# checkservcies -l # list services to restart
# checkservices -r # restart it

5) Detect server reboot
I track my server reboot with the following software[7]. Btw, this is
not a solution for 250 servers.

6) Use your repository to spread your custom packages
For personal packages or taken from AUR, using a custom repository[8}
will simplify your job.
You compile your soft in one place, no need to have gcc or base-devel
on your servers.
Update is automatically propagate as official repository.
You can easily override official package (not recommended).
You could use a base meta package[9] to have all the basics software
on all your servers.
This will prepare you to become an archlinux TU or Dev.

7) Security
Debian is not more secure because their softwares are old. It's a lie.
Check the number of open flaw in the security bug tracker[10].
If you want to be in a secure environment stay up-to-date, don't use
debian stable, use debian sid. So Archlinux is a good alternative.

Regards,

[1] Please note, that is not a pleasure for a package maintainer to
add a message in his package. So read it.
[2] https://github.com/seblu/archutils/blob/master/archlinux-pacfiles
[3] https://github.com/seblu/archutils/blob/master/archlinux-packages
[4] https://github.com/seblu/archpkg/blob/master/linux-seblu/PKGBUILD
[5] https://github.com/seblu/archpkg/blob/master/linux-seblu-meta/PKGBUILD
[6] https://github.com/seblu/archutils/blob/master/checkservices
[7] https://github.com/seblu/mailboot
[8] https://seblu.net/a/seblu/x86_64/
[9] https://github.com/seblu/archpkg/blob/master/base-seblu/PKGBUILD
[10] https://security-tracker.debian.org/tracker/status/release/stable

--
Sébastien "Seblu" Luttringer
https://www.seblu.net
GPG: 0x2072D77A
Paul Gideon Dann
2013-07-10 12:23:28 UTC
Permalink
Post by Sébastien Luttringer
3) Use a versioned kernel
One of the most wanted expectation on a server is to avoid reboot.
Arch official kernel is too often updated for a server _and_ cannot be
installed without breaking the running kernel (modules mismatch).
To workaround this I build custom kernels, with the version in the
name[4] and I use a meta package[5] which push new version
automatically and clean the old one. So I can update my server, and at
the next reboot the last kernel will be selected.
4) Detect daemon upgrade
When you update your system, some libraries or binaries can be updated
and your running programs still use the old version.
This give the bad feeling that your software are up-to-date. But it's false.
Of course you can reboot your server to be sure after each update. It's too
long and give the feeling to hunt fly with a tank.
I use the following script[6] which list services (systemd speaking)
which need to be restarted.
# checkservcies -l # list services to restart
# checkservices -r # restart it
6) Use your repository to spread your custom packages
For personal packages or taken from AUR, using a custom repository[8}
will simplify your job.
You compile your soft in one place, no need to have gcc or base-devel
on your servers.
Update is automatically propagate as official repository.
You can easily override official package (not recommended).
You could use a base meta package[9] to have all the basics software
on all your servers.
This will prepare you to become an archlinux TU or Dev.
The above three points in particular are incredibly insightful; thank you.
This is definitely helpful to have in the back of my head for if the stakes
are raised for the machines I'm responsible for.

FYI, for monitoring my servers I use a combination of Monit and Ganglia, which
I'm pretty happy with. I wish Ganglia had the ability to notify on certain
conditions as some other monitoring tools can, but that's not a biggie, as
that can be dealt with with cron or Monit if necessary.

Some of the functionality I used Monit for became redundant when Systemd came
along: most importantly ensuring that processes were still running and hadn't
crashed. Systemd is capable of re-starting failed services, so long as it's
enabled in the unit file, but it doesn't (yet) have any notification
capability, which is a shame. I also use a cron job to check for any systemd
units that are in a failed state. (In case something unusual happens that
Monit isn't looking for.)

Paul
Karol Babioch
2013-07-10 14:19:42 UTC
Permalink
Hi,
Post by Sébastien Luttringer
7) Security
Debian is not more secure because their softwares are old. It's a lie.
Check the number of open flaw in the security bug tracker[10].
If you want to be in a secure environment stay up-to-date, don't use
debian stable, use debian sid. So Archlinux is a good alternative.
Nevertheless they have a policy as well as a team dedicated to these
issues in place. Coming along with this is a well accredited mailing
list informing you about current issues. Other "server distros" such as
RHEL (and/or centos) have something very similar.

As already pointed out Arch might not push all minor security releases
into the official repositories. Especially in case of a new major kernel
release, minor versions didn't always make it into the repositories in
the past. I can totally live with this on my PC, but on a server I
expect a little bit more on this front.

I don't think that you can seriously consider something to be a "server
distro" without a dedicated security policy and/or team, which will
follow the known databases and/or mailing lists making absolutely sure
that any security patches make it into the appropriate packages.

One reason we all love Arch is because it doesn't heavily patch any
packages. Therefore I'm not sure whether it is suited as a "server
distro" at all.

That said I'm using it myself on a couple of servers. However they are
not publicly accessible, but are only serving their local networks. As
pointed out the experience is a little bit different compared to
"conservative" distributions like Debian, but not necessarily worse.
There were updates in the past that broke a few things here and there,
but generally speaking updates work just fine. And when upgrading
packages to new versions, you will always run into problems. With Arch
you can tackle them one by one, whereas with Debian and its derivatives
you have to tackle them all at once with the next "dist-upgrade".

Best regards,
Karol Babioch
Eduardo Machado
2013-07-10 14:27:03 UTC
Permalink
Post by Sébastien Luttringer
Post by M Saunders
which has some useful tips. But it'd be interesting to hear from
people running Arch on production servers, how well it works for them
and what (if any) problems they've faced.
I've 9 personal servers running Archlinux (previously under debian)
and I plan to move about ~250 professional hypervisor under Arch this
year.
Let me share the following experiences with you.
1) Use the minimum set of packages
This will save you from updating useless packages and give you a
better view of what your server use.
As there is only few packages, don't rush to update them when there is
major change on it.
2) Do your sysadmin homework
Before updating, check archlinux.org for announcements.
During update read pacman output.[1]
After updating, look for pacsave/pacorig/pacnew files.
Supervise your packages. I use munin with the following plugins[2][3]
3) Use a versioned kernel
One of the most wanted expectation on a server is to avoid reboot.
Arch official kernel is too often updated for a server _and_ cannot be
installed without breaking the running kernel (modules mismatch).
To workaround this I build custom kernels, with the version in the
name[4] and I use a meta package[5] which push new version
automatically and clean the old one. So I can update my server, and at
the next reboot the last kernel will be selected.
4) Detect daemon upgrade
When you update your system, some libraries or binaries can be updated
and your running programs still use the old version.
This give the bad feeling that your software are up-to-date. But it's false.
Of course you can reboot your server to be sure after each update.
It's too long and give the feeling to hunt fly with a tank.
I use the following script[6] which list services (systemd speaking)
which need to be restarted.
# checkservcies -l # list services to restart
# checkservices -r # restart it
5) Detect server reboot
I track my server reboot with the following software[7]. Btw, this is
not a solution for 250 servers.
6) Use your repository to spread your custom packages
For personal packages or taken from AUR, using a custom repository[8}
will simplify your job.
You compile your soft in one place, no need to have gcc or base-devel
on your servers.
Update is automatically propagate as official repository.
You can easily override official package (not recommended).
You could use a base meta package[9] to have all the basics software
on all your servers.
This will prepare you to become an archlinux TU or Dev.
7) Security
Debian is not more secure because their softwares are old. It's a lie.
Check the number of open flaw in the security bug tracker[10].
If you want to be in a secure environment stay up-to-date, don't use
debian stable, use debian sid. So Archlinux is a good alternative.
Regards,
[1] Please note, that is not a pleasure for a package maintainer to
add a message in his package. So read it.
[2] https://github.com/seblu/archutils/blob/master/archlinux-pacfiles
[3] https://github.com/seblu/archutils/blob/master/archlinux-packages
[4] https://github.com/seblu/archpkg/blob/master/linux-seblu/PKGBUILD
[5] https://github.com/seblu/archpkg/blob/master/linux-seblu-meta/PKGBUILD
[6] https://github.com/seblu/archutils/blob/master/checkservices
[7] https://github.com/seblu/mailboot
[8] https://seblu.net/a/seblu/x86_64/
[9] https://github.com/seblu/archpkg/blob/master/base-seblu/PKGBUILD
[10] https://security-tracker.debian.org/tracker/status/release/stable
--
Sébastien "Seblu" Luttringer
https://www.seblu.net
GPG: 0x2072D77A
This all were valuable lessons, thanks to share.

At this point i am thinking that the initial question was a great start to
discuss and share stability an security techniques for an Arch install.
David C. Rankin
2013-07-19 20:22:31 UTC
Permalink
Post by M Saunders
Hi all,
I'm writing a feature about Arch for Linux Format, a UK-based
newsstand Linux magazine. I've been using Arch myself for a while for
testing new app releases, and it's brilliant for that purpose.
I'm still left wondering though: who uses it on production servers? I
mean, the distro's overall simplicity and trimmed-down base
installation are plus points here, but surely a rolling release poses
problems. After installation you just want security and critical bug
fix updates for software, and not major version bumps, right?
https://wiki.archlinux.org/index.php/Enhancing_Arch_Linux_Stability
which has some useful tips. But it'd be interesting to hear from
people running Arch on production servers, how well it works for them
and what (if any) problems they've faced.
Thanks!
Mike
M,

I have run two offices with Arch production servers. The primary consideration
for looking at Arch for production boxes was the rolling-release model which
promised to eliminate the forced reinstall from version X to version X.1 in
traditional release based distros. Arch was stellar from 2009 through mid 2012
in providing seamless updates without an excessive amount of time-robbing
intervention required. Stability of the distro in this regard was a primary
consideration in remaining with Arch. Beginning with the clib change and
continuing through the initscript->systemd migration, the changes really began
to impact Arch's suitability for use in production. In my case, the last killer
was the loss of dmraid (nvidia based controller) support which was not provided
by mdraid. Whereas before, there had always been a stable upgrade path, it ended
at that point.

Arch is just as suitable for server use as any other distro and is rock-solid.
However, Arch's priority to maintain a cutting-edge distro far outweighs any
thought of providing a broad stable upgrade path for all currently supported
hardware. If Arch has a chance to move forward to the latest "released" version
of a package that will break hardware support for some, then those 'some' are
just out of luck. This is something to be aware of before committing to build a
production server platform around Arch.

Reinstalling a desktop if a change is necessary is painless compared to tuning
a new server.
--
David C. Rankin, J.D.,P.E.
Robert Spahr
2013-07-20 00:31:57 UTC
Permalink
On Tue, 9 Jul 2013 11:13:08 +0100
Post by M Saunders
But it'd be interesting to hear from
people running Arch on production servers, how well it works for them
and what (if any) problems they've faced.
Mike,

As an artist that makes computational and generative art, I use servers
to produce art work on a 24/7 schedule. [http://www.robertspahr.com]

I previously used gentoo, on a couple of servers, but over the last two
years, I transitioned my own personal computers and servers from
gentoo, to arch linux. I very much value the rolling releases of both
distributions.

The arch binaries save me so much time over the gentoo compiling. For
me the important thing is to be conservative on the upgrades. I have a
staging server that I always test upgrades first, before updating the
live server.

Besides that, the arch linux community is quite helpful.


Cheers,

-- Rob
--
Robert Spahr, MFA
Assistant Professor
Department of Cinema & Photography
1100 Lincoln Dr. Rm. 1121E, Mail Code 6610
Southern Illinois University
Carbondale IL 62901

http://www.robertspahr.com

"breathe and be mindful"
Alexander Diana
2013-07-21 06:02:41 UTC
Permalink
Post by Robert Spahr
On Tue, 9 Jul 2013 11:13:08 +0100
Post by M Saunders
But it'd be interesting to hear from
people running Arch on production servers, how well it works for them
and what (if any) problems they've faced.
Mike,
As an artist that makes computational and generative art, I use servers
to produce art work on a 24/7 schedule. [http://www.robertspahr.com]
I previously used gentoo, on a couple of servers, but over the last two
years, I transitioned my own personal computers and servers from
gentoo, to arch linux. I very much value the rolling releases of both
distributions.
The arch binaries save me so much time over the gentoo compiling. For
me the important thing is to be conservative on the upgrades. I have a
staging server that I always test upgrades first, before updating the
live server.
Besides that, the arch linux community is quite helpful.
Cheers,
-- Rob
I find that as long as the deployment doesn't have a need for SELinux,
or doesn't face an outside (public) network, its perfectly fine and
stable. Wrestling SELinux into arch is not something fun to do or keep
alive, and nor is complying with security patches.

TLDR: It has served me very well in production as long as you aren't
dealing in security.
--
Alexander Diana <***@gmail.com>
Continue reading on narkive:
Loading...