Discussion:
End of official PaX and grsecurity support in Arch Linux
(too old to reply)
Daniel Micay via arch-general
2017-04-27 18:19:49 UTC
Permalink
Raw Message
The PaX and grsecurity patches are no longer going to be public, so
official support in Arch Linux has ended:

https://grsecurity.net/passing_the_baton.php
https://grsecurity.net/passing_the_baton_faq.php

I'll be clearing out the AUR packages for PaX and grsecurity soon since
the current 4.10 patches are not accessible. Providing out-of-date
packages with known vulnerabilities and without the current set of
mitigations is infringing on how they want their trademarks used. If
people want to maintain forks, either a 4.9 LTS or porting it forward to
newer kernels, they'll need to make a new project with a new name rather
than using "PaX" or "grsecurity" as the naming. If it's serious and done
by people that know what they're doing, official support for it can be
considered. There are few people that are capable of truly taking over
maintenance of the whole patches and I expect that they won't have time
to do it. Don't be optimistic about this happening.

There are no viable alternatives to PaX and grsecurity. Their focus is
on kernel self-protection i.e. protecting the kernel from exploits, and
we don't have anything for users to migrate to from these. There are
plenty of alternatives to grsecurity RBAC but that's only a small
portion of what the patches provide. Any form of access control (whether
it's MAC, containers, uid/gid separation, ACLs, etc.) can be entirely
bypassed with a single kernel exploit, so the only good way to use other
MAC implementations like TOMOYO, AppArmor or SELinux was with
grsecurity. We don't actually provide official support for any of these
alternatives, but it's all little good without a hardened kernel anyway.

A small subset of the kernel self-protection features have landed
upstream as part of the Kernel Self Protection Project. The core/linux
package doesn't enable the bulk of these features, and it probably
doesn't make sense for it to turn everything on because a subset of them
do have a significant performance cost. It would be possible to provide
a linux-hardened package doing that but it would only offer a tiny
portion of the kernel self-protection that grsecurity does. I may end up
doing that, along with enabling support for all of the LSMs, etc. there
but it's really not at all comparable to what has been lost. The LSMs
are also little good without people working on providing userspace
integration and policies for them.

There are no good answers here. It would be possible to maintain a fork
of grsecurity, especially if all kernel architectures other than x86_64
(and arm64, but it's not currently supported) were dropped along with
grsecurity RBAC and anything that has a proper equivalent upstream. i386
and armv7 can still be supported as userspace archs, since dropping them
as kernel archs is what would save most of the maintainance work and
would eliminate a bunch of complex code only currently fully understood
by the PaX and grsecurity authors. It might also make sense to start
dropping support for old CPUs, beyond just only supporting 64-bit
kernels. Intel Broadwell and later provide SMAP and there's also SMEP,
so those could be used as substitutes for UDEREF and that portion of the
KERNEXEC feature.

The problem is that there aren't capable people with the time and
willingness to dedicate a huge amount of their time to a volunteer
project without pay. I already maintain/develop hardened Android LTS
kernels as a significant but overall quite small part of CopperheadOS.
It's not comparable to this because it's my job. I spend far more than
40 hours a week on CopperheadOS, and it's actually quite a relief to not
need to worry about maintaining PaX and grsecurity for Arch Linux
anymore especially since I had to do it on my own. I've been aware that
the test patches becoming private was a very real possibility even
before spender started talking about it on IRC weeks ago, so it's really
not a surprise.
Carsten Mattner via arch-general
2017-04-27 19:11:10 UTC
Permalink
Raw Message
This is an undesirable situation for users, but I want to offer a
positive outlook on this. Ever since KSPP started, some of the
dynamics started to shift and I wager that closing off grsec will
motivate more users and developers to consider supporting efforts that
are in mainline linux. Short-term this is a problem and may require
relying and hoping 4.9-lts-grsec will be available and functioning.
When Bitkeeper licensing was revoked from the community, it didn't
take long for git to emerge. I see a similar pattern and high
potential for repetition of the same dynamics here. No grsec will
force people to either subscribe ala RHEL and hope spender is able to
fulfill his end of the contract or supporting KSPP and seamless LSM
integration in major distro packages.

I must admit that spender may have started a process that will result
in arriving quicker at mainline kernel having a comparable set of
protections. Because as long as grsec was there and offered for
relatively recent kernels, there wasn't much motivation or arguments
to make to support a mainline reimplementation.

I believe this will light a fire under KSPP and related community
driven projects.

I faintly remember when there was OpenGrsec because grsec was dead or
zombie but that was at least a decade ago and my memory is probably
incomplete.

I mean some grsec users might consider fleeing to HardenedBSD since
they provide a whole system like Hardened Gentoo, especially those
using grsec on hosting servers where the availability of jails,
capsicum, zfs, dtrace, ports and hardenedbsd may have already looked
enticing.
Daniel Micay via arch-general
2017-04-27 19:37:03 UTC
Permalink
Raw Message
Post by Carsten Mattner via arch-general
This is an undesirable situation for users, but I want to offer a
positive outlook on this. Ever since KSPP started, some of the
dynamics started to shift and I wager that closing off grsec will
motivate more users and developers to consider supporting efforts that
are in mainline linux. Short-term this is a problem and may require
relying and hoping 4.9-lts-grsec will be available and functioning.
When Bitkeeper licensing was revoked from the community, it didn't
take long for git to emerge. I see a similar pattern and high
potential for repetition of the same dynamics here. No grsec will
force people to either subscribe ala RHEL and hope spender is able to
fulfill his end of the contract or supporting KSPP and seamless LSM
integration in major distro packages.
It's primarily not a technical issue, it's a political ones. As part
landing mitigations upstream, they end up being watered down into
slower, incomplete and/or weaker implementations. Lots has been outright
rejected. Software implementations of SMEP and SMAP are not happening
for i386 and x86_64. Proper slab sanitization was rejected. Proper page
sanitization was rejected. RAP and SIZE_OVERFLOW upstream are pipe
dreams. The REFCOUNT mitigation was rejected and is going to need to be
done as opt-in, but they also blocked an efficient implementation like
PaX and opt-in usage was rejected in the network stack, etc. due to the
performance cost of their crappier implementation. A new refcount_t
implementation may land, but there are a lot of politics involved and
landing a migration from atomic_t to refcount_t across the kernel is
going to be insanely slow and difficult. It has to be submitted bit by
bit to different maintainers... and that's only the tip of the iceberg
for these mitigations.

It's realistically going to take 5+ years for KSPP to land everything
other than RAP and SIZE_OVERFLOW and that's assuming it's extremely
successful and the political issues are overcome. UDEREF/KERNEXEC will
never land in their entirety and PaX and grsecurity are quickly moving
targets. RAP didn't exist until recently, and it's now the flagship
mitigation they offer. KERNSEAL/STRUCTGUARD won't have public code to
copy... and neither will all of the other new stuff. No one else is
doing compelling new kernel security research... so what happens once
there's no longer public code to copy? It's literally a copy-paste job
right now with bikeshedding of naming and kernel politics, and yet it's
still going poorly. KSPP is quite useful and is improving things, sure.
It's a joke to claim that it's going to be comparable to grsecurity
though.
Post by Carsten Mattner via arch-general
I must admit that spender may have started a process that will result
in arriving quicker at mainline kernel having a comparable set of
protections. Because as long as grsec was there and offered for
relatively recent kernels, there wasn't much motivation or arguments
to make to support a mainline reimplementation.
I believe this will light a fire under KSPP and related community
driven projects.
I can't see it going any faster than it already is, which is slow
progress towards landing some things, with lots of setbacks and
crippling of the features to make them acceptable upstream. I don't
think there's any hope of landing mitigations like SIZE_OVERFLOW and RAP
any time in the near future. RAP requires sweeping fixes of undefined
behavior across the kernel, along with changes to calls/returns in all
of the assembly code. SIZE_OVERFLOW is similar, but also requires many
fixes of non-bugs. The only changes that have been successful landed
upstream are those that are quite contained, or bits and pieces of ones
that make wider changes in areas where the maintainers are active and
see the value of it. Familiarity with what's actually happening and has
happened already is needed to make any useful predictions about it.
Post by Carsten Mattner via arch-general
I faintly remember when there was OpenGrsec because grsec was dead or
zombie but that was at least a decade ago and my memory is probably
incomplete.
I can't find anything about that via searching for it. The grsecurity
patches are also a lot different in terms of what they contain now
compared to a decade ago. PaX started from NX emulation, inventing ASLR
and other userspace mitigations but then ended up being focused on
kernel self-protection and grsecurity similarly has a much different
focus than it did in the past. They're much bigger today and they have a
lot of complex, arcane code for architecture-specific mitigations.
Post by Carsten Mattner via arch-general
I mean some grsec users might consider fleeing to HardenedBSD since
they provide a whole system like Hardened Gentoo, especially those
using grsec on hosting servers where the availability of jails,
capsicum, zfs, dtrace, ports and hardenedbsd may have already looked
enticing.
HardenedBSD doesn't provide most of the grsecurity mitigations,
including some of the most important / strongest mitigations like RAP.
Carsten Mattner via arch-general
2017-04-27 19:12:55 UTC
Permalink
Raw Message
Is CopperheadOS using grsec or something derived from it?
Carsten Mattner via arch-general
2017-04-27 19:46:39 UTC
Permalink
Raw Message
On Thu, Apr 27, 2017 at 7:12 PM, Carsten Mattner
Post by Carsten Mattner via arch-general
Is CopperheadOS using grsec or something derived from it?
Found the technical details, it seems to be select grsec features
ported to AOSP but not a full port of grsec, which together with the
other hardening looks reasonable since it's a whole system/distro approach.
Daniel Micay via arch-general
2017-04-27 20:16:20 UTC
Permalink
Raw Message
Post by Carsten Mattner via arch-general
Is CopperheadOS using grsec or something derived from it?
It starts from the baseline provided by Google and ports features from
PaX and grsecurity as needed to the kernels. It used to use a full PaX
port on ARM devices but that hasn't made sense for quite some time.

Android is much a different target. CopperheadOS is starting from a base
system that's already quite hardened. PaX / grsecurity kernel self
protection features are applicable to Android but the bulk of them don't
support arm64 in grsecurity at the moment so it's no use beyond as a
source for porting / inspiration. The features that landed upstream
ended up being ported to arm64. The grsecurity test patches becoming
private has no impact on CopperheadOS. There's a significant indirect
impact in that the source of 95% of Linux kernel security research /
innovations is now no longer publicly available. KSPP no longer has an
incredible source to copy ideas and code from and neither do we.

grsecurity RBAC and all but a few of the miscellaneous features aren't
relevant to Android. It already has the baseline per-user, per-app
uid/gid separation reinforced with a full system SELinux policy that's
far beyond what RHEL / Fedora provide, since there's a well-defined base
system (no system administrator choosing packages / configuration) and a
well-defined app sandbox for all third party code. Devices handle their
drivers and hardware-specific services/libraries via device-specific
policy extensions. For example, SELinux ioctl filtering is used for
kernel attack surface reduction by whitelisting ioctl commands per-
device including whitelisting only the set of GPU driver ioctl commands
used by the device in practice. They're also making increasingly good
use of seccomp-bpf.

Even some of the most basic security features like full verified boot
for the whole OS and always enabled encryption are pipe dreams for
traditional distributions. Android's linker doesn't support non-PIE and
text relocations are only supported for legacy API level 32-bit apps.
Full RELRO, strong SSP and _FORTIFY_SOURCE=2 (beyond what glibc
provides) are globally enabled.

The AOSP kernels are very minimal, i.e. no modules enabled and only a
small set of drivers, etc. needed for the platform. Features like System
V IPC and now AIO aren't enabled. Google already has KSPP features
backported and enabled in their LTS kernels like HARDENED_USERCOPY
(incomplete KSPP implementation of PaX USERCOPY), __ro_after_init
(incomplete KSPP implementation of PaX __read_only) and PAN (UDEREF
equivalent) emulation for ARM and ARMv8.0 where PAN isn't yet available.
They also have kernel security features that were rejected upstream like
perf_event_paranoid=3 which we upstreamed to AOSP.

In addition to that much different starting point, CopperheadOS also has
full control of the entire base system. There's a unified build system
and all of the sources are in one tree like a *BSD OS. It's so much
different than trying to deal with bringing desktop Linux security on
par with 2008 security standards, which is still so far away as a goal.
One day perhaps flatpak / wayland will have caught up to providing a
basic security model for desktop Linux and some day Arch will finally
get PIE enabled by default but all of those kind of things are already
provided by the baseline OS that CopperheadOS starts from.
Alexander Harrigan
2017-04-27 20:45:08 UTC
Permalink
Raw Message
It would be great if you can provide linux-hardened kernel with everything
what KSPP has enabled by default. Even in AUR so you won't have to rebuild it
constantly and random stack option would have more sense.

Two questions:

1\. Do you think maintaining 4.9 lts grsec kernel would be doable until it
gets EOL from upstream? Are there many incompatible changes with minor kernel
releases? I heard gentoo (maybe debian?) are considering this.

2\. Why hidepid was removed? I saw "lack of time" comment but...there wasn't
much to maintain as it worked flawles for long time.

Thank you for your great effort in maintaining grsec for all that years. I'm
looking forward to being your customer (Copperhead) when it will be avalaible
overseas :)

On On Thu, Apr 27, 2017 at 08:17 PM, Daniel Micay via arch-general <arch-
***@archlinux.org> wrote:

> On Thu, 2017-04-27 at 19:12 +0000, Carsten Mattner wrote:

> > Is CopperheadOS using grsec or something derived from it?

>

> It starts from the baseline provided by Google and ports features from

> PaX and grsecurity as needed to the kernels. It used to use a full PaX

> port on ARM devices but that hasn't made sense for quite some time.

>

> Android is much a different target. CopperheadOS is starting from a base

> system that's already quite hardened. PaX / grsecurity kernel self

> protection features are applicable to Android but the bulk of them don't

> support arm64 in grsecurity at the moment so it's no use beyond as a

> source for porting / inspiration. The features that landed upstream

> ended up being ported to arm64. The grsecurity test patches becoming

> private has no impact on CopperheadOS. There's a significant indirect

> impact in that the source of 95% of Linux kernel security research /

> innovations is now no longer publicly available. KSPP no longer has an

> incredible source to copy ideas and code from and neither do we.

>

> grsecurity RBAC and all but a few of the miscellaneous features aren't

> relevant to Android. It already has the baseline per-user, per-app

> uid/gid separation reinforced with a full system SELinux policy that's

> far beyond what RHEL / Fedora provide, since there's a well-defined base

> system (no system administrator choosing packages / configuration) and a

> well-defined app sandbox for all third party code. Devices handle their

> drivers and hardware-specific services/libraries via device-specific

> policy extensions. For example, SELinux ioctl filtering is used for

> kernel attack surface reduction by whitelisting ioctl commands per-

> device including whitelisting only the set of GPU driver ioctl commands

> used by the device in practice. They're also making increasingly good

> use of seccomp-bpf.

>

> Even some of the most basic security features like full verified boot

> for the whole OS and always enabled encryption are pipe dreams for

> traditional distributions. Android's linker doesn't support non-PIE and

> text relocations are only supported for legacy API level 32-bit apps.

> Full RELRO, strong SSP and _FORTIFY_SOURCE=2 (beyond what glibc

> provides) are globally enabled.

>

> The AOSP kernels are very minimal, i.e. no modules enabled and only a

> small set of drivers, etc. needed for the platform. Features like System

> V IPC and now AIO aren't enabled. Google already has KSPP features

> backported and enabled in their LTS kernels like HARDENED_USERCOPY

> (incomplete KSPP implementation of PaX USERCOPY), __ro_after_init

> (incomplete KSPP implementation of PaX __read_only) and PAN (UDEREF

> equivalent) emulation for ARM and ARMv8.0 where PAN isn't yet available.

> They also have kernel security features that were rejected upstream like

> perf_event_paranoid=3 which we upstreamed to AOSP.

>

> In addition to that much different starting point, CopperheadOS also has

> full control of the entire base system. There's a unified build system

> and all of the sources are in one tree like a *BSD OS. It's so much

> different than trying to deal with bringing desktop Linux security on

> par with 2008 security standards, which is still so far away as a goal.

> One day perhaps flatpak / wayland will have caught up to providing a

> basic security model for desktop Linux and some day Arch will finally

> get PIE enabled by default but all of those kind of things are already

> provided by the baseline OS that CopperheadOS starts from.

\-- Sent using MsgSafe.io's Free Plan Private, encrypted, online communication
For everyone. https://www.msg
Daniel Micay via arch-general
2017-04-27 23:21:20 UTC
Permalink
Raw Message
Post by Alexander Harrigan
It would be great if you can provide linux-hardened kernel with everything
what KSPP has enabled by default. Even in AUR so you won't have to rebuild it
constantly and random stack option would have more sense.
1\. Do you think maintaining 4.9 lts grsec kernel would be doable until it
gets EOL from upstream?
That could be done, but it can't be called grsecurity or PaX anymore.

There will be conflicts when applying the new patches. There will be
real bugs that are caught by the mitigations and also some cases that
are false positives or just benign issues that upstream doesn't consider
to be a bug from features like SIZE_OVERFLOW. There is no longer an
upstream (i.e. PaX and grsecurity) where these issues can be reported.

For it to be officially packaged again, there needs to be a very
competent upstream taking responsibility for everything as there was
before.

I don't really think it makes sense to track 4.9 until it dies. It's a
dead end and new security features will be landing in mainline.

I think it makes sense to have a linux-hardened package but I can't
currently commit to doing that. I need to think about it over the next
few days.

It would also make sense to have a substantially stripped down fork,
dropping everything that can be provided via upstream features, modern
hardware (SMAP on Broadwell or later) or SELinux and just hoping that
people work on having great SELinux support as an orthogonal project. I
really don't have time for that though. It needs to be a collaborative
effort, and I really do mean collaborative i.e. *multiple* people doing
a substantial amount of difficult work and making hard decisions about what should be dropped to keep it maintainable. For example, some may see dropping UDEREF for x86_64 and just assuming SMAP is present to be too unfriendly to people with legacy hardware, but NOT doing that as much as possible means so much extra very complex code to maintain. I have no interest in maintaining code for legacy hardware that I don't use or being blocked by porting it to new kernel versions.
Post by Alexander Harrigan
Are there many incompatible changes with minor kernel
releases? I heard gentoo (maybe debian?) are considering this.
Plenty. It's very realistic to do that, but there's still work. The main
issue is that there isn't going to be an end to upstream bugs and false
positives being found, just fewer if it's frozen at 4.9 and destined to
die off as it ages.
Post by Alexander Harrigan
2\. Why hidepid was removed? I saw "lack of time" comment but...there
wasn'tmuch to maintain as it worked flawles for long time.
A bug was reported where it caused problems booting. Multiple people
stated that they hit it, so I believe that it's real. I don't have time
to look into it and no one else was doing that, so I replaced it with
documentation on doing it manually on the wiki:

https://wiki.archlinux.org/index.php/Security#hidepid

It was nice having the package, but if there's something broken and no
one is addressing that I'm not going to keep it around.
ITwrx.org
2017-04-28 01:58:11 UTC
Permalink
Raw Message
Post by Daniel Micay via arch-general
The PaX and grsecurity patches are no longer going to be public, so
this is highly disappointing but not completely unexpected. thanks for
your work all this time.
Alexander Harrigan
2017-04-29 17:03:15 UTC
Permalink
Raw Message
I found someone from opensuse started to maintain grsec patches for 4.9 kernel
series [1]. Maybe it will be possible to add linux-lts-grsec package to AUR
based on Daniel's PKGBUILD and config with RANDSTRUCT enabled linked to new
upstream source.

[1] https://github.com/kdave/grsecurity-patches/tree/master/wip

\-- Sent using MsgSafe.io's Free Plan Private, encrypted, online communication
For everyone. https://www.msgsafe.io
Daniel Micay via arch-general
2017-04-29 19:19:48 UTC
Permalink
Raw Message
Post by Alexander Harrigan
I found someone from opensuse started to maintain grsec patches for 4.9 kernel
series [1]. Maybe it will be possible to add linux-lts-grsec package to AUR
based on Daniel's PKGBUILD and config with RANDSTRUCT enabled linked to new
upstream source.
[1] https://github.com/kdave/grsecurity-patches/tree/master/wip
As I mentioned, it can't be called PaX or grsecurity. I also don't think
it makes sense to expend time on this. It won't support new hardware and
systemd will probably increase the minimum kernel version before the 4.9
LTS is end-of-life. Effort spent on 4.9 is effort not spent on anything
that will actually last. If someone decides to do this, they'll also be
taking responsibility for maintaining PaX exceptions, etc. and handling
any bugs caught by the features or false positives. There will be new
issues introduced as the LTS gets changes backported to it.
Alexander Harrigan
2017-04-29 20:03:55 UTC
Permalink
Raw Message
Name doesn't matter. When it breaks with systemd then it will be stopped but
that won't be in months. Paxd is avalaible and it always be assumed to
maintain it by yourself. Also I see contradiction when you say many critical
grsecurity features won't be avalaible in mainline kernels anytime soon and
same time you say there is no sense to use last grsec kernel as long as we
can. I'm not asking you to maintain it. There is a chance that
gentoo|opensuse|debian guys step in to do it. In that case it will be
beneficial to Arch having it avalaible in AUR.

Anyway I'll be using grsec kernel myself until something like linux-hardened
be avalaible.

On On Sat, Apr 29, 2017 at 07:20 PM, Daniel Micay via arch-general <arch-
***@archlinux.org> wrote:

> On Sat, 2017-04-29 at 17:03 +0000, Alexander Harrigan wrote:

> > I found someone from opensuse started to maintain grsec patches for

> > 4.9 kernel

> > series [1]. Maybe it will be possible to add linux-lts-grsec package

> > to AUR

> > based on Daniel's PKGBUILD and config with RANDSTRUCT enabled linked

> > to new

> > upstream source.

> >

> > [1] https://github.com/kdave/grsecurity-patches/tree/master/wip

>

> As I mentioned, it can't be called PaX or grsecurity. I also don't think

> it makes sense to expend time on this. It won't support new hardware and

> systemd will probably increase the minimum kernel version before the 4.9

> LTS is end-of-life. Effort spent on 4.9 is effort not spent on anything

> that will actually last. If someone decides to do this, they'll also be

> taking responsibility for maintaining PaX exceptions, etc. and handling

> any bugs caught by the features or false positives. There will be new

> issues introduced as the LTS gets changes backported to it.

\-- Sent using MsgSafe.io's Free Plan Private, encrypted, online communication
For everyone. https://www.msgsafe.io
Daniel Micay via arch-general
2017-04-29 21:14:51 UTC
Permalink
Raw Message
It isn't a contradiction. If the focus is on an LTS, then it's a dead
end and there will be nothing to show for it in the future. The easiest
time to start deciding what to drop and porting forward is now while
it's only one kernel version behind.
Alexander Harrigan
2017-05-01 12:34:55 UTC
Permalink
Raw Message
It looks Gentoo's Hardened Kernel Project oficially started.

https://wiki.gentoo.org/wiki/Hardened/Hardened_Kernel_Project

\-- Sent using MsgSafe.io's Free Plan Private, encrypted, online communication
For everyone. https://www.msgsafe.io
Daniel Micay via arch-general
2017-05-01 17:31:08 UTC
Permalink
Raw Message
Post by Alexander Harrigan
It looks Gentoo's Hardened Kernel Project oficially started.
https://wiki.gentoo.org/wiki/Hardened/Hardened_Kernel_Project
Gentoo wiki page != Gentoo project.

Loading...