Discussion:
Updating linux-lts initcpios... error: command failed to execute correctly
(too old to reply)
Ralph Corderoy
2018-11-07 12:27:29 UTC
Permalink
Hi,

This morning's `pacman -Su' had a problem building the fallback initcpio
image for 4.14.79-1-lts. Yesterday's /var/log/pacman.log shows

[2018-11-06 10:45] [ALPM] upgraded linux-lts (4.14.78-1 -> 4.14.79-1)
[2018-11-06 10:45] [ALPM] running '60-linux-lts.hook'...
[2018-11-06 10:45] [ALPM] running '90-linux-lts.hook'...

that all went well.

Today, package `linux-lts' wasn't upgraded but 90-linux-lts.hook was
triggered. 60-linux-lts.hook wasn't. pacman's TTY output complained
`error: command failed to execute correctly' and the list of `Running'
build hooks didn't complete. The same steps for package `linux'
immediately after didn't have the same problem.

:: Running post-transaction hooks...
( 1/12) Updating linux-lts initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default'
-> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img
==> Starting build: 4.14.79-1-lts
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [block]
-> Running build hook: [filesystems]
-> Running build hook: [keyboard]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-lts.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'fallback'
-> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts-fallback.img -S autodetect
==> Starting build: 4.14.79-1-lts
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: aic94xx
→ error: command failed to execute correctly
( 2/12) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 4.18.16-arch1-1-ARCH
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [block]
-> Running build hook: [filesystems]
-> Running build hook: [keyboard]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 4.18.16-arch1-1-ARCH
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: aic94xx
-> Running build hook: [filesystems]
-> Running build hook: [keyboard]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img
==> Image generation successful
( 3/12) Updating icon theme caches...

I was hoping pacman.log might have more detail, but it doesn't even
record that there was an error. I think that's a bug.

[2018-11-07 11:18] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'fallback'
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts-fallback.img -S autodetect
[2018-11-07 11:18] [ALPM-SCRIPTLET] ==> Starting build: 4.14.79-1-lts
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2018-11-07 11:18] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: wd719x
[2018-11-07 11:18] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2018-11-07 11:19] [ALPM] running '90-linux.hook'...
[2018-11-07 11:19] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'

Where should I look for clues why the command failed?
And after that, is attempting another build of the image as simple as
`/usr/bin/mkinitcpio -p linux-lts' as root?
--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy
Eli Schwartz via arch-general
2018-11-07 14:03:07 UTC
Permalink
Post by Ralph Corderoy
Hi,
This morning's `pacman -Su' had a problem building the fallback initcpio
image for 4.14.79-1-lts. Yesterday's /var/log/pacman.log shows
[2018-11-06 10:45] [ALPM] upgraded linux-lts (4.14.78-1 -> 4.14.79-1)
[2018-11-06 10:45] [ALPM] running '60-linux-lts.hook'...
[2018-11-06 10:45] [ALPM] running '90-linux-lts.hook'...
that all went well.
Today, package `linux-lts' wasn't upgraded but 90-linux-lts.hook was
triggered. 60-linux-lts.hook wasn't.
The 60- hook is:
Description = Updating linux-lts module dependencies...

and its purpose is to run depmod whenever you update modules.

The 90- hook is:
Description = Updating linux-lts initcpios...

and its purpose is to run mkinitcpio whenever you update either the
kernel, or else any file which is part of the mkinitcpio configuration.

$ pacman -Qo /usr/lib/initcpio/
/usr/lib/initcpio/ is owned by archiso 39-1
/usr/lib/initcpio/ is owned by btrfs-progs 4.19-1
/usr/lib/initcpio/ is owned by cryptsetup 2.0.5-1
/usr/lib/initcpio/ is owned by device-mapper 2.02.182-1
/usr/lib/initcpio/ is owned by dmraid 1.0.0.rc16.3-11
/usr/lib/initcpio/ is owned by lvm2 2.02.182-1
/usr/lib/initcpio/ is owned by mdadm 4.0-1
/usr/lib/initcpio/ is owned by mkinitcpio 25-1
/usr/lib/initcpio/ is owned by mkinitcpio-busybox 1.29.3-1
/usr/lib/initcpio/ is owned by systemd 239.300-1

As you can see, there's lots of things which can do so!

This explains why one ran and the other did not.
Post by Ralph Corderoy
pacman's TTY output complained
`error: command failed to execute correctly' and the list of `Running'
build hooks didn't complete. The same steps for package `linux'
immediately after didn't have the same problem.
This means mkinitcpio itself got borked, but rerunning it with a
different kernel preset works okay... why, remains to be seen.
Post by Ralph Corderoy
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-lts.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'fallback'
-> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts-fallback.img -S autodetect
==> Starting build: 4.14.79-1-lts
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: aic94xx
→ error: command failed to execute correctly
So, the fallback image for linux-lts seems to have simply died halfway
through (but the main image generated okay).
Post by Ralph Corderoy
I was hoping pacman.log might have more detail, but it doesn't even
record that there was an error. I think that's a bug.
[2018-11-07 11:18] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'fallback'
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts-fallback.img -S autodetect
[2018-11-07 11:18] [ALPM-SCRIPTLET] ==> Starting build: 4.14.79-1-lts
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2018-11-07 11:18] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: wd719x
[2018-11-07 11:18] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2018-11-07 11:19] [ALPM] running '90-linux.hook'...
[2018-11-07 11:19] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
Well, the log only contains a copy of the output from the hook which you
can look up after the fact, so it definitely won't have *more* input. As
for logging the failure of the hook to complete, I guess you could open
a ticket on the bugtracker for this (and/or post to the pacman-dev
mailing list if that floats your boat).
Post by Ralph Corderoy
Where should I look for clues why the command failed?
And after that, is attempting another build of the image as simple as
`/usr/bin/mkinitcpio -p linux-lts' as root?
This looks like it is another occurrence of
https://bugs.archlinux.org/task/60396

To test, you could try reinstalling a package that provides a mkinitcpio
configuration, and test resizing the terminal to see if you get the same
error.

Then, try installing pacman-git from the AUR or from my unofficial repo
hosted at
https://wiki.archlinux.org/index.php/Unofficial_user_repositories#eschwartz

The fix should be in pacman-git, and this should no longer crash.
--
Eli Schwartz
Bug Wrangler and Trusted User
Ralph Corderoy
2018-11-07 15:06:37 UTC
Permalink
Hi Eli,
Post by Eli Schwartz via arch-general
$ pacman -Qo /usr/lib/initcpio/
/usr/lib/initcpio/ is owned by archiso 39-1
...

Thanks, it hadn't occurred to me to use -Qo on a directory before.
Post by Eli Schwartz via arch-general
Well, the log only contains a copy of the output from the hook which
you can look up after the fact, so it definitely won't have *more*
input.
Understood. I was thinking along the lines of how a ./configure writes
to an agreed file descriptor other than stdout/stderr for more verbose
detail, that kind of thing, but there weren't extra messages to record
anyway given below suggests it's a SIGPIPE. Would the exit status of
the hook have reflected that signal by being 128+13?
Post by Eli Schwartz via arch-general
As for logging the failure of the hook to complete, I guess you could
open a ticket on the bugtracker for this
Okay. https://bugs.archlinux.org/task/60735
Post by Eli Schwartz via arch-general
This looks like it is another occurrence of
https://bugs.archlinux.org/task/60396
It's very likely. I know I did maximise the X terminal part-way through
pacman's run to unwrap lines.
Post by Eli Schwartz via arch-general
To test...
I'll skip that, sorry. Way behind with life as it is at the moment.
I've looked at the bug report and the patch and it seems sound.
Post by Eli Schwartz via arch-general
Post by Ralph Corderoy
And after that, is attempting another build of the image as simple
as `/usr/bin/mkinitcpio -p linux-lts' as root?
I did this and it seems to have gone well.

Thanks for the prompt answers.
--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy
Eli Schwartz via arch-general
2018-11-07 15:19:09 UTC
Permalink
Post by Ralph Corderoy
Hi Eli,
Post by Eli Schwartz via arch-general
$ pacman -Qo /usr/lib/initcpio/
/usr/lib/initcpio/ is owned by archiso 39-1
...
Thanks, it hadn't occurred to me to use -Qo on a directory before.
It's pretty nifty, yeah. Manpage and --help output does say "file(s)",
though -- maybe we should change that.
Post by Ralph Corderoy
Post by Eli Schwartz via arch-general
Well, the log only contains a copy of the output from the hook which
you can look up after the fact, so it definitely won't have *more*
input.
Understood. I was thinking along the lines of how a ./configure writes
to an agreed file descriptor other than stdout/stderr for more verbose
detail, that kind of thing, but there weren't extra messages to record
anyway given below suggests it's a SIGPIPE. Would the exit status of
the hook have reflected that signal by being 128+13?
Yes, pacman just checks if the exit status of the hook was something
indicating an error.
Post by Ralph Corderoy
Post by Eli Schwartz via arch-general
As for logging the failure of the hook to complete, I guess you could
open a ticket on the bugtracker for this
Okay. https://bugs.archlinux.org/task/60735
Thanks. <3
Post by Ralph Corderoy
Post by Eli Schwartz via arch-general
This looks like it is another occurrence of
https://bugs.archlinux.org/task/60396
It's very likely. I know I did maximise the X terminal part-way through
pacman's run to unwrap lines.
Post by Eli Schwartz via arch-general
To test...
I'll skip that, sorry. Way behind with life as it is at the moment.
I've looked at the bug report and the patch and it seems sound.
Sure, I wasn't expecting problems -- it's been tested already. :)

But of course you are free to use my git repo either way -- it's
verified by my packaging key already, it contains cool new features, and
I am good about keeping it up to date.

Anyway, it's planned to be added to a maintenance release for pacman
5.1.2, hopefully soon.
Post by Ralph Corderoy
Post by Eli Schwartz via arch-general
Post by Ralph Corderoy
And after that, is attempting another build of the image as simple
as `/usr/bin/mkinitcpio -p linux-lts' as root?
I did this and it seems to have gone well.
Forgot to mention, but yeah, running the command that the hook was
supposed to run, will usually work. :D

If it doesn't work, it would be because the hook specifies NeedsTargets
and therefore expects to receive a list of files on standard input,
listing the files which triggered the hook.
Post by Ralph Corderoy
Thanks for the prompt answers.
No problem! :)
--
Eli Schwartz
Bug Wrangler and Trusted User
l***@gmx.fr
2018-11-07 18:50:27 UTC
Permalink
Post by Ralph Corderoy
Hi,
This morning's `pacman -Su' had a problem building the fallback initcpio
Right command is pacman -Syu
Post by Ralph Corderoy
image for 4.14.79-1-lts. Yesterday's /var/log/pacman.log shows
[2018-11-06 10:45] [ALPM] upgraded linux-lts (4.14.78-1 -> 4.14.79-1)
[2018-11-06 10:45] [ALPM] running '60-linux-lts.hook'...
[2018-11-06 10:45] [ALPM] running '90-linux-lts.hook'...
Ralph Corderoy
2018-11-07 22:46:37 UTC
Permalink
Hi leoutation,
Post by l***@gmx.fr
Post by Ralph Corderoy
This morning's `pacman -Su' had a problem building the fallback initcpio
Right command is pacman -Syu
Not in my case as I've previously done a `pacman -Syuw' during `free
bandwidth' hours, thanks to cron. Later on, when I'm at the keyboard
and it's a convenient time for breakage, I upgrade to what's already
been downloaded manually and definitely don't want `-y'. :-)
--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy
Eli Schwartz via arch-general
2018-11-07 23:16:15 UTC
Permalink
Post by Ralph Corderoy
Hi leoutation,
Post by l***@gmx.fr
Post by Ralph Corderoy
This morning's `pacman -Su' had a problem building the fallback initcpio
Right command is pacman -Syu
Not in my case as I've previously done a `pacman -Syuw' during `free
bandwidth' hours, thanks to cron. Later on, when I'm at the keyboard
and it's a convenient time for breakage, I upgrade to what's already
been downloaded manually and definitely don't want `-y'. :-)
That's dangerous because you could forget and run pacman -S pkgname. Not
saying you would, and in fact I've done -Sywu all the time myself...

But, just because I do it doesn't mean I should. ;)

Instead, I eventually did some fiddling with the checkupdates script,
which can be seen here:
https://git.archlinux.org/users/eschwartz/pacman-contrib.git/commit/?id=80be77ce7d3e1ba8eee42c56623e01f0c45235cd

This will eventually make its way to pacman-contrib upstream, but it's
pending some pacman changes which are still in git and only very
recently landed.

In the meantime you can also use
https://github.com/eli-schwartz/dotfiles/blob/master/bin/cacheupdates

Just run cacheupdates the way you'd run checkupdates, but if you specify
the argument "-d" (or "--download") then it will also try to download
any pending updates into the cache, still using the temporary copy of
the syncdbs.
--
Eli Schwartz
Bug Wrangler and Trusted User
Ralph Corderoy
2018-11-08 17:25:13 UTC
Permalink
Hi Eli,
Post by Eli Schwartz via arch-general
Post by Ralph Corderoy
Not in my case as I've previously done a `pacman -Syuw' during `free
bandwidth' hours, thanks to cron. Later on, when I'm at the
keyboard and it's a convenient time for breakage, I upgrade to
what's already been downloaded manually and definitely don't want
`-y'. :-)
That's dangerous because you could forget and run pacman -S pkgname.
AIUI, if I did then the version of pkgname it would try to download to
install would be that snap-shotted when cron ran `pacman -Syuw'. And
that's a problem because that version might not be available for
download any more if a later one has been made available? Similarly,
with packages pkgname depends on. Is there another problem I don't
know?

I actually tend to: sudo -i at 1:23 <<<'pacman -Syw pkgname' as it's
normally not urgent.

Can an attempt to fetch an old, now missing, package still occur even
with `pacman -Sy pkgname' because the remote database has been altered,
and old versions removed, between the -y's refresh and fetching a long
list of packages?
--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy
Eli Schwartz via arch-general
2018-11-08 17:37:30 UTC
Permalink
Post by Ralph Corderoy
Hi Eli,
Post by Eli Schwartz via arch-general
Post by Ralph Corderoy
Not in my case as I've previously done a `pacman -Syuw' during `free
bandwidth' hours, thanks to cron. Later on, when I'm at the
keyboard and it's a convenient time for breakage, I upgrade to
what's already been downloaded manually and definitely don't want
`-y'. :-)
That's dangerous because you could forget and run pacman -S pkgname.
AIUI, if I did then the version of pkgname it would try to download to
install would be that snap-shotted when cron ran `pacman -Syuw'. And
that's a problem because that version might not be available for
download any more if a later one has been made available? Similarly,
with packages pkgname depends on. Is there another problem I don't
know?
-_-

No, the problem is that you'll accidentally do a partial update.

The package itself might very well be cached already, but if not, you'll
get a 404 "cannot find package, I will now error without doing anything".

That's not "dangerous". Dangerous is when you break your system by
installing incompatible packages via a "successful" partial update.
Post by Ralph Corderoy
I actually tend to: sudo -i at 1:23 <<<'pacman -Syw pkgname' as it's
normally not urgent.
Can an attempt to fetch an old, now missing, package still occur even
with `pacman -Sy pkgname' because the remote database has been altered,
and old versions removed, between the -y's refresh and fetching a long
list of packages?
It won't be old and now missing, because you used -Sy and it will
refresh *again*.

But if you don't use -y, then trying to install a package first looks in
the cache, where it may not need to be fetched...
--
Eli Schwartz
Bug Wrangler and Trusted User
Ralph Corderoy
2018-11-08 18:05:58 UTC
Permalink
Hi Eli,
Dangerous is when you break your system by installing incompatible
packages via a "successful" partial update.
Ah, right. I `pacman -S foo'. foo depends on libbar. libbar 1.0 is
already installed because xyzzy needs it. The earlier `pacman -Syuw'
updated the local database to know libbar 1.1 is available thus
installing foo upgrades libbar. xyzzy remains at the old version,
incompatible with libbar 1.1.
Post by Ralph Corderoy
Can an attempt to fetch an old, now missing, package still occur
even with `pacman -Sy pkgname' because the remote database has been
altered, and old versions removed, between the -y's refresh and
fetching a long list of packages?
It won't be old and now missing, because you used -Sy and it will
refresh *again*.
But isn't it multiple stages with a race condition?

1 Update /var/lib/pacman/sync/core.db
2 Update /var/lib/pacman/sync/extra.db
3 Update /var/lib/pacman/sync/community.db
4 Update /var/lib/pacman/sync/multilib.db
5 Create /var/cache/pacman/pkg/foo-1.0.pkg.tar.xz
6 Create /var/cache/pacman/pkg/bar-2.0.pkg.tar.xz
7 Create /var/cache/pacman/pkg/xyzzy-3.0.pkg.tar.xz

Can the remote core.db be updated between 1 and 6 because bar 2.1 is
available? Do bar-2.0.pkg.tar.xz and bar-2.1.pkg.tar.xz ever exist at
the same time? Perhaps the fetch of bar-2.0.pkg.tar.xz might fail
because it's been removed by the time it's attempted.

This is nothing to do with my original question. :-) I guess I'm
asking how is a consistent multi-file database of indexes and `records'
presented? It might be that the rare times this happens, the user just
runs `pacman -Sy pkgname' again.
--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy
Eli Schwartz via arch-general
2018-11-08 19:32:44 UTC
Permalink
Post by Ralph Corderoy
Post by Eli Schwartz via arch-general
It won't be old and now missing, because you used -Sy and it will
refresh *again*.
But isn't it multiple stages with a race condition?
1 Update /var/lib/pacman/sync/core.db
2 Update /var/lib/pacman/sync/extra.db
3 Update /var/lib/pacman/sync/community.db
4 Update /var/lib/pacman/sync/multilib.db
5 Create /var/cache/pacman/pkg/foo-1.0.pkg.tar.xz
6 Create /var/cache/pacman/pkg/bar-2.0.pkg.tar.xz
7 Create /var/cache/pacman/pkg/xyzzy-3.0.pkg.tar.xz
Can the remote core.db be updated between 1 and 6 because bar 2.1 is
available? Do bar-2.0.pkg.tar.xz and bar-2.1.pkg.tar.xz ever exist at
the same time? Perhaps the fetch of bar-2.0.pkg.tar.xz might fail
because it's been removed by the time it's attempted.
This is nothing to do with my original question. :-) I guess I'm
asking how is a consistent multi-file database of indexes and `records'
presented? It might be that the rare times this happens, the user just
runs `pacman -Sy pkgname' again.
Well, most users will have multiple mirrors in their pacman mirrorlist.
So if the package is no longer available on the first mirror, pacman
will keep trying until it finds it somewhere.

Also, the job we use to delete old packages only runs at 3-hour
intervals, which means there's a window of opportunity when multiple
versions are still available, and most mirrors sync roughly hourly but
not on the same minute of that hour. So it would be an odd coincidence
if you managed to mistime your update such that

1) your mirror updates in the seconds between syncing the db and
downloading the package

2) that mirror update is during its one-hour sync cycle that is also the
one-third of the time when the cleanup job runs

3) every one of your fallback mirrors is newer than your first
configured mirror

If you do manage to win this game of roulette, you can just pacman -Syu
again. :) pacman will simply abort the upgrade if it cannot find one of
the package files, so consistency is maintained -- it's just a much more
exotic case of "I haven't done pacman -Syu in a week and cannot install
new packages that have been updated on the mirrors".

...

Source:

ansible repo containing our dbscripts role, and in particular, the job
cleanup.{service,time} :
https://git.archlinux.org/infrastructure.git/tree/roles/dbscripts/files

The dbscripts cleanup script itself:
https://git.archlinux.org/dbscripts.git/tree/cron-jobs/ftpdir-cleanup
--
Eli Schwartz
Bug Wrangler and Trusted User
Loading...