Discussion:
how long does a make buildworld normally take?
Christoph P.U. Kukulies
2016-11-18 09:38:45 UTC
Permalink
how long does a make buildworld normally take?
Has make buildworld time constant over the years?


In 1995 "make world" on a Pentium Pro about 18 hours
On my 4GB, CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
(1995.64-MHz 686-class CPU)
make buildworld took about 18 hours. Well that CPU isn't the latest
either (10 Years or so)


Reminds me of a joke, saying that all bycycles weight the same (10lbs).

Light carbon fiber bike 3lbs. need 7 lbs. lock

Heavy steel bike 7lbs. need 3 lbs. lock

As hardware grew faster over the years, software exploded the same way
and eats up resources.


Anyway, how could one speed up the make buildworld process?
Parallelizing (make -j 8 or sth.)?

I will try next on my 16GB notebook with i7-4720 (8core) CPU 1TB SSD and
see what it gives.

--

Christoph
Matt Smith
2016-11-18 10:17:52 UTC
Permalink
Post by Christoph P.U. Kukulies
how long does a make buildworld normally take?
Has make buildworld time constant over the years?
In 1995 "make world" on a Pentium Pro about 18 hours
On my 4GB, CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
(1995.64-MHz 686-class CPU)
make buildworld took about 18 hours. Well that CPU isn't the latest
either (10 Years or so)
18 hours?! On my current home-server which is an Intel Atom D525 1.8ghz
it only takes 5 hours. And these are not known for being fast at all. In
1996 ish I was doing it on something like a pentium 75 and I think it
still took about 5 hours. So you're right there! 18 hours seems way too
long though.

These days it compiles things like clang which take a very long time.
--
Matt
Polytropon
2016-11-18 10:58:01 UTC
Permalink
Post by Matt Smith
Post by Christoph P.U. Kukulies
how long does a make buildworld normally take?
Has make buildworld time constant over the years?
In 1995 "make world" on a Pentium Pro about 18 hours
On my 4GB, CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
(1995.64-MHz 686-class CPU)
make buildworld took about 18 hours. Well that CPU isn't the latest
either (10 Years or so)
18 hours?! On my current home-server which is an Intel Atom D525 1.8ghz
it only takes 5 hours. And these are not known for being fast at all.
Allow me to quote from a message I wrote in 2008, in order to
illustrate technical progress by collectingg several build times
for different world and kernel configurations.

FreeBSD 5 on Pentium 4 with 2 GHz and 1 GB RAM:

b.world+b.kern: 17494.415u 2562.134s 5:46:42.25 96.4% (with CFLAGS)
17474.169u 2481.368s 5:46:30.40 95.9% (without CLFAGS)
5608.712u 1595.130s 2:13:18.67 90.0%
6382.185u 1788.433s 2:26:36.06 92.8%
buildworld: 5086.993u 1431.086s 1:58:16.33 91.8%
11457.047u 2151.158s 3:54:15.31 96.8%
buildkernel 2326.380u 234.457s 43:42.15 97.6%
1102.491u 278.194s 25:18.58 90.9%
1182.203u 294.622s 26:12.71 93.9%
1518.402u 310.741s 34:16.96 88.9%
3289.368u 529.669s 1:05:25.90 97.2%
installkernel: 5.718u 6.898s 0:30.97 40.6%
6.655u 7.389s 0:32.08 43.7%
6.994u 7.734s 0:33.19 44.3%

(...software advance happens here...)

FreeBSD 7 on Pentium 4 with 2 GHz and 1 GB RAM:

b.world+b.kern: 16574.070u 2516.128s 6:06:03.90 86.9% (with debug)
18232.967u 2427.404s 7:19:49.24 78.2% (with debug)
18992.839u 2569.146s 9:12:00.28 65.1%
buildworld: 11457.047u 2151.158s 3:54:15.31 96.8%
buildkernel: 3289.368u 529.669s 1:05:25.90 97.2%
3503.732u 524.399s 1:11:05.53 94.4%
4032.019u 572.636s 1:58:29.08 64.7% (with debug)
installkernel: 17.396u 12.587s 0:46.89 63.9%
18.890u 12.131s 1:11.85 43.1%

As you can see, 5 hours was a possible value on a single-core
single-threat slow-as-ass CPU. But then the system became more
advanced, and 7 - 9 hours compile time became possible. :-)
Post by Matt Smith
In
1996 ish I was doing it on something like a pentium 75 and I think it
still took about 5 hours. So you're right there! 18 hours seems way too
long though.
On my first FreeBSD machine, compiling world and kernel needed
approx. 24 hours. The kernel itself could be rebuilt in more or
less 6 hours. Today we have loadable modules. :-)
Post by Matt Smith
These days it compiles things like clang which take a very long time.
But it also introduces optimization which the "old compilers" did
not provide, so by investing more build time, you get a better
runtime experience (programs run faster). Especially on limited
hardware, this is worth considering (like in ye olde days setting
several "strange" CFLAGS to make mplayer perform better).
--
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
Eduardo Lemos de Sa
2016-11-18 13:03:47 UTC
Permalink
Dear
Post by Polytropon
Post by Matt Smith
Post by Christoph P.U. Kukulies
how long does a make buildworld normally take?
Has make buildworld time constant over the years?
In 1995 "make world" on a Pentium Pro about 18 hours
On my 4GB, CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
(1995.64-MHz 686-class CPU)
make buildworld took about 18 hours. Well that CPU isn't the latest
either (10 Years or so)
18 hours?! On my current home-server which is an Intel Atom D525 1.8ghz
it only takes 5 hours. And these are not known for being fast at all.
Allow me to quote from a message I wrote in 2008, in order to
illustrate technical progress by collectingg several build times
for different world and kernel configurations.
b.world+b.kern: 17494.415u 2562.134s 5:46:42.25 96.4% (with CFLAGS)
17474.169u 2481.368s 5:46:30.40 95.9% (without CLFAGS)
5608.712u 1595.130s 2:13:18.67 90.0%
6382.185u 1788.433s 2:26:36.06 92.8%
buildworld: 5086.993u 1431.086s 1:58:16.33 91.8%
11457.047u 2151.158s 3:54:15.31 96.8%
buildkernel 2326.380u 234.457s 43:42.15 97.6%
1102.491u 278.194s 25:18.58 90.9%
1182.203u 294.622s 26:12.71 93.9%
1518.402u 310.741s 34:16.96 88.9%
3289.368u 529.669s 1:05:25.90 97.2%
installkernel: 5.718u 6.898s 0:30.97 40.6%
6.655u 7.389s 0:32.08 43.7%
6.994u 7.734s 0:33.19 44.3%
(...software advance happens here...)
b.world+b.kern: 16574.070u 2516.128s 6:06:03.90 86.9% (with debug)
18232.967u 2427.404s 7:19:49.24 78.2% (with debug)
18992.839u 2569.146s 9:12:00.28 65.1%
buildworld: 11457.047u 2151.158s 3:54:15.31 96.8%
buildkernel: 3289.368u 529.669s 1:05:25.90 97.2%
3503.732u 524.399s 1:11:05.53 94.4%
4032.019u 572.636s 1:58:29.08 64.7% (with debug)
installkernel: 17.396u 12.587s 0:46.89 63.9%
18.890u 12.131s 1:11.85 43.1%
As you can see, 5 hours was a possible value on a single-core
single-threat slow-as-ass CPU. But then the system became more
advanced, and 7 - 9 hours compile time became possible. :-)
Post by Matt Smith
In
1996 ish I was doing it on something like a pentium 75 and I think it
still took about 5 hours. So you're right there! 18 hours seems way too
long though.
On my first FreeBSD machine, compiling world and kernel needed
approx. 24 hours. The kernel itself could be rebuilt in more or
less 6 hours. Today we have loadable modules. :-)
Post by Matt Smith
These days it compiles things like clang which take a very long time.
But it also introduces optimization which the "old compilers" did
not provide, so by investing more build time, you get a better
runtime experience (programs run faster). Especially on limited
hardware, this is worth considering (like in ye olde days setting
several "strange" CFLAGS to make mplayer perform better).
--
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-
I found strange to spend 18, or even, 5 hours to finish a make buildworld.
My desktop is certainly the faster for standart machines:

hw.machine: amd64
hw.model: AMD FX(tm)-8320 Eight-Core Processor
hw.ncpu: 8
hw.machine_arch: amd64

with 16 GB Ram (1333 MHz)

and spend less than 2 hours:

--------------------------------------------------------------
Post by Polytropon
Post by Matt Smith
Post by Christoph P.U. Kukulies
World build started on Wed Nov 16 17:23:35 BRST 2016
--------------------------------------------------------------

--------------------------------------------------------------
Post by Polytropon
Post by Matt Smith
Post by Christoph P.U. Kukulies
Rebuilding the temporary build tree
--------------------------------------------------------------


--------------------------------------------------------------
Post by Polytropon
Post by Matt Smith
Post by Christoph P.U. Kukulies
World build completed on Wed Nov 16 19:07:33 BRST 2016
--------------------------------------------------------------


It is a FreeBSD-10.3-release, with a single processor (no parallel)

I suggest you to check your procedure or even possible problems with your
hard disk.

My best wishes

Eduardo
--
Eduardo Lemos de Sa
Associated Professor Level 4
Dep. Quimica da Universidade Federal do Paraná
fone: +55(41)3361-3300
fax: +55(41)3361-3186
Polytropon
2016-11-18 13:26:46 UTC
Permalink
Post by Eduardo Lemos de Sa
hw.machine: amd64
hw.model: AMD FX(tm)-8320 Eight-Core Processor
hw.ncpu: 8
hw.machine_arch: amd64
with 16 GB Ram (1333 MHz)
--------------------------------------------------------------
Post by Christoph P.U. Kukulies
World build started on Wed Nov 16 17:23:35 BRST 2016
--------------------------------------------------------------
--------------------------------------------------------------
Post by Christoph P.U. Kukulies
Rebuilding the temporary build tree
--------------------------------------------------------------
--------------------------------------------------------------
Post by Christoph P.U. Kukulies
World build completed on Wed Nov 16 19:07:33 BRST 2016
--------------------------------------------------------------
It is a FreeBSD-10.3-release, with a single processor (no parallel)
That sounds possible, given the computing power mentioned,
especially when the resources can be used fully by the
build process (no significant load from other system
activities or services).
Post by Eduardo Lemos de Sa
I suggest you to check your procedure or even possible problems with your
hard disk.
The data I presented is _historical_ data from FreeBSD versions
and processors that do not exist anymore. :-)
--
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
Brian W.
2016-11-18 13:53:54 UTC
Permalink
Using ccache and setting the jobs at 2 gets it to about an hour on my old
amd am2 dual core.

Brian
Post by Polytropon
Post by Eduardo Lemos de Sa
hw.machine: amd64
hw.model: AMD FX(tm)-8320 Eight-Core Processor
hw.ncpu: 8
hw.machine_arch: amd64
with 16 GB Ram (1333 MHz)
--------------------------------------------------------------
Post by Christoph P.U. Kukulies
World build started on Wed Nov 16 17:23:35 BRST 2016
--------------------------------------------------------------
--------------------------------------------------------------
Post by Christoph P.U. Kukulies
Rebuilding the temporary build tree
--------------------------------------------------------------
--------------------------------------------------------------
Post by Christoph P.U. Kukulies
World build completed on Wed Nov 16 19:07:33 BRST 2016
--------------------------------------------------------------
It is a FreeBSD-10.3-release, with a single processor (no parallel)
That sounds possible, given the computing power mentioned,
especially when the resources can be used fully by the
build process (no significant load from other system
activities or services).
Post by Eduardo Lemos de Sa
I suggest you to check your procedure or even possible problems with your
hard disk.
The data I presented is _historical_ data from FreeBSD versions
and processors that do not exist anymore. :-)
--
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-
Warren Block
2016-11-18 15:49:41 UTC
Permalink
Post by Christoph P.U. Kukulies
how long does a make buildworld normally take?
There are too many options to really nail that down. It varies due to
hardware, build options, caching, and the version of FreeBSD. The
fastest machines take minutes, slower ones several hours.

With every go-fast option enabled on this Xeon E3 system, a full
buildworld and kernel takes about twenty minutes (excluding LLDB).
With few or no changes to source, that can be repeated in under two
minutes, because it's all in cache thanks to bdrewery's build system
work (NO_CLEAN and ccache).
Post by Christoph P.U. Kukulies
Has make buildworld time constant over the years?
No, although one version of FreeBSD usually has pretty consistent build
times.
Tim Daneliuk
2016-11-18 16:00:18 UTC
Permalink
Post by Warren Block
Post by Christoph P.U. Kukulies
how long does a make buildworld normally take?
There are too many options to really nail that down.
<SNIP>

I have two example environments - both running 10.3-STABLE, updated and
recompiled nightly:

Quad Core i5 3.2GHz / 8G - About 30 mins for full buildworld (including LLDB) and 2 kernels

Single Core VPS Xeon CPU E5-2650L v3 @ 1.80GHz / 1G - As above, no LLDB - 5 hours

Bear in mind that the VPS is presumably sharing that core with many other instances, hence
the long compile times. Also, it is severely memory constrained and is probably
swapping mightily during these compiles.

FWIW, I have noted that this is very much a CPU-bound problem. Doing the work
on faster drives - like SSDs - seems to not make much difference on a given machine.
--
----------------------------------------------------------------------------
Tim Daneliuk ***@tundraware.com
PGP Key: http://www.tundraware.com/PGP/
Loading...