Discussion:
makepkg - any way to recompile only newly patched files in large packages?
Add Reply
David C. Rankin
2017-08-07 03:23:12 UTC
Reply
Permalink
Raw Message
All,

There was a fix for a gtk2 bug I filed regarding the recentchooserdefault.c
file. (it was a one-line fix where a variable needed reset to 0). Is there any
way I can configure makepkg to behave as if the gcc -MP -MD options had been
given to have it only recompile the one patched source without rebuilding the
entire package -- again?

I've been through the wiki and the man page, and other than --repackage,
there doesn't seem to be anything that would work (or I'm just missing it).
--
David C. Rankin, J.D.,P.E.
David C. Rankin
2017-08-07 07:39:54 UTC
Reply
Permalink
Raw Message
Post by David C. Rankin
All,
There was a fix for a gtk2 bug I filed regarding the recentchooserdefault.c
file. (it was a one-line fix where a variable needed reset to 0). Is there any
way I can configure makepkg to behave as if the gcc -MP -MD options had been
given to have it only recompile the one patched source without rebuilding the
entire package -- again?
I've been through the wiki and the man page, and other than --repackage,
there doesn't seem to be anything that would work (or I'm just missing it).
If it wasn't clear, I have already built the gtk2 package yesterday to
--enable-debug=yes so I have all of the files in a state I could call
--repackge on, except for the gtk/gtkrecentchooser.c file with the one line
change. I was wondering if there was a way to avoid the full 15 minute rebuild
of all of gtk2 and just compile gtkrecentchooserdefault.c to object and then
--repackage?
--
David C. Rankin, J.D.,P.E.
LoneVVolf
2017-08-07 09:59:26 UTC
Reply
Permalink
Raw Message
On 07-08-17 09:39, David C. Rankin wrote:
I was wondering if there was a way to avoid the full 15 minute rebuild
Post by David C. Rankin
of all of gtk2 and just compile gtkrecentchooserdefault.c to object and then
--repackage?
I don't know a way to do that, but do you have ccache installed ?

On my system it's set to use max 20G on a HDD .
When building things like mesa or llvm repeatedly with small changes
this can shaves minutes of the build time.

LW

https://wiki.archlinux.org/index.php/Ccache
Christoph Gysin via arch-general
2017-08-07 11:06:52 UTC
Reply
Permalink
Raw Message
On Mon, Aug 7, 2017 at 10:39 AM, David C. Rankin
Post by David C. Rankin
If it wasn't clear, I have already built the gtk2 package yesterday to
--enable-debug=yes so I have all of the files in a state I could call
--repackge on, except for the gtk/gtkrecentchooser.c file with the one line
change. I was wondering if there was a way to avoid the full 15 minute rebuild
of all of gtk2 and just compile gtkrecentchooserdefault.c to object and then
--repackage?
Not the answer to your question, but you might consider configuring
/etc/makepkg.conf to use ccache to speed up rebuilds. Also set
MAKEFLAGS=-j4 or similar to make sure you are using all available
cores.
--
echo mailto: NOSPAM !#$.'<*>'|sed 's. ..'|tr "<*> !#:2" ***@fr33z3
David C. Rankin
2017-08-08 02:06:22 UTC
Reply
Permalink
Raw Message
Post by Christoph Gysin via arch-general
Not the answer to your question, but you might consider configuring
/etc/makepkg.conf to use ccache to speed up rebuilds. Also set
MAKEFLAGS=-j4 or similar to make sure you are using all available
cores.
Thanks Cristoph,

How do you know whether a package is capable of being built in parallel?
I've run into problems before with big builds when building in parallel. Is it
just a 'try and see if it works' issue, or is there some general rule-of-thumb
you can use?
--
David C. Rankin, J.D.,P.E.
Eli Schwartz
2017-08-08 20:29:53 UTC
Reply
Permalink
Raw Message
Post by David C. Rankin
Post by Christoph Gysin via arch-general
Not the answer to your question, but you might consider configuring
/etc/makepkg.conf to use ccache to speed up rebuilds. Also set
MAKEFLAGS=-j4 or similar to make sure you are using all available
cores.
Thanks Cristoph,
How do you know whether a package is capable of being built in parallel?
I've run into problems before with big builds when building in parallel. Is it
just a 'try and see if it works' issue, or is there some general rule-of-thumb
you can use?
Generally, you try and see if it works. If it doesn't, the build rules
don't properly specify dependencies, and you file an upstream bug
report, which may or may not get fixed, and you make sure the PKGBUILD
maintainer specifically resets MAKEFLAGS back to -j1 (this is the only
time it is okay to specify the number of make jobs to run in parallel,
if you are resetting back to the defaults because a package doesn't
compile with custom MAKEFLAGS in makepkg.conf).
--
Eli Schwartz
respiranto
2017-08-07 12:35:05 UTC
Reply
Permalink
Raw Message
Post by David C. Rankin
Post by David C. Rankin
All,
There was a fix for a gtk2 bug I filed regarding the recentchooserdefault.c
file. (it was a one-line fix where a variable needed reset to 0). Is there any
way I can configure makepkg to behave as if the gcc -MP -MD options had been
given to have it only recompile the one patched source without rebuilding the
entire package -- again?
I've been through the wiki and the man page, and other than --repackage,
there doesn't seem to be anything that would work (or I'm just missing it).
If it wasn't clear, I have already built the gtk2 package yesterday to
--enable-debug=yes so I have all of the files in a state I could call
--repackge on, except for the gtk/gtkrecentchooser.c file with the one line
change. I was wondering if there was a way to avoid the full 15 minute rebuild
of all of gtk2 and just compile gtkrecentchooserdefault.c to object and then
--repackage?
That should be precisely the job of make. All the other preparation
steps such as ./configure or patching should not harm either, likely
rather be necessary.

Hence, you could just try to rerun makepkg, providing that the $srcdir
is still in the state after the last successful build.

In fact, I already had problems doing this. Suppose a sed script
replacing 'x' by 'xx', inline. In such cases one might consider using
the --noprepare flag.

In any case, it is a good idea to read the PKGBUILD (as always) and
possibly modify it or even do all the work manually.
Eli Schwartz
2017-08-07 14:36:37 UTC
Reply
Permalink
Raw Message
Post by David C. Rankin
Post by David C. Rankin
All,
There was a fix for a gtk2 bug I filed regarding the recentchooserdefault.c
file. (it was a one-line fix where a variable needed reset to 0). Is there any
way I can configure makepkg to behave as if the gcc -MP -MD options had been
given to have it only recompile the one patched source without rebuilding the
entire package -- again?
I've been through the wiki and the man page, and other than --repackage,
there doesn't seem to be anything that would work (or I'm just missing it).
If it wasn't clear, I have already built the gtk2 package yesterday to
--enable-debug=yes so I have all of the files in a state I could call
--repackge on, except for the gtk/gtkrecentchooser.c file with the one line
change. I was wondering if there was a way to avoid the full 15 minute rebuild
of all of gtk2 and just compile gtkrecentchooserdefault.c to object and then
--repackage?
I think you're misunderstanding what -MP does.

As for rebuilding only the one source, makepkg source extraction should
result in extracting files with the same initial timestamp as the last
time they were extracted. Although if you are using the gtk2 PKGBUILD
from the ABS, it uses git anyway... So Make should consider all objects
to be up-to-date, except the one you have sed'ed.

You could also use `makepkg --noextract` which assumes you applied the
fix by hand rather than in the PGBUILD, which is probably safer as it
skips prepare(), and in theory re-running configure should not cause
Make to expire all object files (though with autotools who really knows...)

Aside: repackage does not do what you want, since it doesn't rebuild
anything. Granted, Makefiles usually have the install target depend on
the default target, so it would probably end up being built too... in
fakeroot...
--
Eli Schwartz
David C. Rankin
2017-08-08 02:12:53 UTC
Reply
Permalink
Raw Message
Post by Eli Schwartz
You could also use `makepkg --noextract` which assumes you applied the
fix by hand rather than in the PGBUILD, which is probably safer as it
skips prepare(), and in theory re-running configure should not cause
Make to expire all object files (though with autotools who really knows...)
That is one thing I didn't consider. It has always been beaten into my head
that you don't touch the original source and you take the time to create a
.patch to be applied though prepare().

I guess this is one instance where it would make sense to actually edit the
files in the source-tree directly in order to skip prepare.

Thanks.
Post by Eli Schwartz
Aside: repackage does not do what you want, since it doesn't rebuild
anything. Granted, Makefiles usually have the install target depend on
the default target, so it would probably end up being built too... in
fakeroot...
Yes, the reference to --repackage was just a way of describing the state of
the build directory. Repackaging the same executables hoping something changed
is as productive as trying to destroy a shot-put with a rubber hammer...
--
David C. Rankin, J.D.,P.E.
Eli Schwartz
2017-08-08 20:33:22 UTC
Reply
Permalink
Raw Message
Post by David C. Rankin
Post by Eli Schwartz
You could also use `makepkg --noextract` which assumes you applied the
fix by hand rather than in the PGBUILD, which is probably safer as it
skips prepare(), and in theory re-running configure should not cause
Make to expire all object files (though with autotools who really knows...)
That is one thing I didn't consider. It has always been beaten into my head
that you don't touch the original source and you take the time to create a
.patch to be applied though prepare().
I guess this is one instance where it would make sense to actually edit the
files in the source-tree directly in order to skip prepare.
Thanks.
You should do both. Set it in prepare() to save yourself the effort in
the future, and for your local run you can do it by hand to save time
when not doing a clean rebuild.

prepare() is not idempotent, its value lies in making sure you can
replicate what you did next week when running `makepkg -C`.
--
Eli Schwartz
ProgAndy
2017-08-07 14:42:41 UTC
Reply
Permalink
Raw Message
Post by David C. Rankin
If it wasn't clear, I have already built the gtk2 package yesterday to
--enable-debug=yes so I have all of the files in a state I could call
--repackge on, except for the gtk/gtkrecentchooser.c file with the one line
change. I was wondering if there was a way to avoid the full 15 minute rebuild
of all of gtk2 and just compile gtkrecentchooserdefault.c to object and then
--repackage?
You can use makepkg with the -e (--noextract) option to rebuild from the
existing source tree. If the configure step forces a full rebuild, you
can comment it out in the PKGBUILD after the initial build.


--
Andy
Loading...