Discussion:
mayday - new updates killed httpd - make_sock: could not bind to address [::]:443
(too old to reply)
David C. Rankin
2010-08-09 09:19:41 UTC
Permalink
guys,

I updated my test box and it checked out, then I updated my server and apache
will not start. The error given is:

[03:58 nirvana:/etc/httpd/conf] # apachectl start
(98)Address already in use: make_sock: could not bind to address [::]:443
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:443
no listening sockets available, shutting down
Unable to open logs

The updates tonight included:

[2010-08-09 00:41] upgraded consolekit (0.4.1-2 -> 0.4.1-3)
[2010-08-09 00:41] upgraded dhcpcd (5.2.5-1 -> 5.2.7-1)
[2010-08-09 00:41] upgraded dnsutils (9.6.1-3 -> 9.7.1.P2-1)
[2010-08-09 00:41] upgraded libnice (0.0.12-1 -> 0.0.13-1)
[2010-08-09 00:41] upgraded farsight2 (0.0.20-1 -> 0.0.21-1)
[2010-08-09 00:41] upgraded libva (0.31.0_p13-2 -> 1.0.4-1)
[2010-08-09 00:41] upgraded ffmpeg (24460-1 -> 24460-2)
[2010-08-09 00:41] installed iana-etc (2.30-1)
[2010-08-09 00:41] upgraded filesystem (2010.02-4 -> 2010.07-1)
[2010-08-09 00:41] upgraded hwdetect (2010.07-1 -> 2010.08-1)
[2010-08-09 00:42] upgraded kernel26 (2.6.34.2-1 -> 2.6.34.2-2)
[2010-08-09 00:42] upgraded kernel26-headers (2.6.34.2-1 -> 2.6.34.2-2)
[2010-08-09 00:44] upgraded kernel26-lts (2.6.32.16-1 -> 2.6.32.17-2)
[2010-08-09 00:44] upgraded kernel26-lts-headers (2.6.32.16-1 -> 2.6.32.17-2)
[2010-08-09 00:44] upgraded networkmanager (0.8-1 -> 0.8.1-1)
[2010-08-09 00:44] warning: /etc/pacman.d/mirrorlist installed as
/etc/pacman.d/mirrorlist.pacnew
[2010-08-09 00:44] upgraded pacman-mirrorlist (20100621-1 -> 20100808-1)
[2010-08-09 00:44] upgraded python-telepathy (0.15.17-1 -> 0.15.18-1)
[2010-08-09 00:44] upgraded rpcbind (0.2.0-1 -> 0.2.0-2)
[2010-08-09 00:44] upgraded sqlite3 (3.7.0-1 -> 3.7.0.1-1)
[2010-08-09 00:44] upgraded telepathy-haze (0.3.6-1 -> 0.4.0-1)
[2010-08-09 00:44] upgraded thunderbird (3.1.1-1 -> 3.1.2-1)
[2010-08-09 00:44] upgraded util-linux-ng (2.18-2 -> 2.18-3)
[2010-08-09 00:44] upgraded vdpau-video (0.6.9-2 -> 0.6.10-2)
[2010-08-09 00:44] upgraded vlc (1.1.2-1 -> 1.1.2-2)

I have switched to a backup server so I'm not dead, but to have apache just not
start was a bit disconcerting to say the least. This has got to be an issue with
the kernel26 (2.6.34.2-1 -> 2.6.34.2-2) upgrade. My primary and backup servers
are both Arch 64bit boxes with identical software installs. The box with the
problem is an Opteron(tm) Processor 180 running on Tyan Tomcat S2865, the backup
is a Phenom(tm) 9850 on an MSI K9N2 SLI Platinum.

The fact that the software is identical and the primary can't make/bind to
sockets anymore --- tells me what?

Anyway, I need help. Any thoughts on what do check? I'll drop back to LTS and
see how that goes and report back. Thanks.
--
David C. Rankin, J.D.,P.E.
Rankin Law Firm, PLLC
510 Ochiltree Street
Nacogdoches, Texas 75961
Telephone: (936) 715-9333
Facsimile: (936) 715-9339
www.rankinlawfirm.com
Jan de Groot
2010-08-09 09:31:57 UTC
Permalink
Post by David C. Rankin
(98)Address already in use: make_sock: could not bind to address [::]:443
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:443
Maybe you should read the error messages and check out with tools like
lsof what process is using that port.
David C. Rankin
2010-08-09 09:35:33 UTC
Permalink
Post by Jan de Groot
Maybe you should read the error messages and check out with tools like
lsof what process is using that port.
OK,

This is where I need to add some tools to my toolbox. lsof always just gives me
reams of data that doesn't make much sense to me. lsof what here?
--
David C. Rankin, J.D.,P.E.
Rankin Law Firm, PLLC
510 Ochiltree Street
Nacogdoches, Texas 75961
Telephone: (936) 715-9333
Facsimile: (936) 715-9339
www.rankinlawfirm.com
ALAUX, Guillaume
2010-08-09 09:40:25 UTC
Permalink
lsof -i TCP | grep LISTEN

Otherwise the man page of lsof
Post by David C. Rankin
Post by Jan de Groot
Maybe you should read the error messages and check out with tools like
lsof what process is using that port.
OK,
This is where I need to add some tools to my toolbox. lsof always
just gives me reams of data that doesn't make much sense to me. lsof what
here?
--
David C. Rankin, J.D.,P.E.
Rankin Law Firm, PLLC
510 Ochiltree Street
Nacogdoches, Texas 75961
Telephone: (936) 715-9333
Facsimile: (936) 715-9339
www.rankinlawfirm.com
David C. Rankin
2010-08-09 09:45:31 UTC
Permalink
Post by Jan de Groot
Post by David C. Rankin
(98)Address already in use: make_sock: could not bind to address [::]:443
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:443
Maybe you should read the error messages and check out with tools like
lsof what process is using that port.
OK Googling help:

[04:38 nirvana:/home/david] # lsof -i tcp:443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
cupsd 1896 root 8u IPv4 7293 0t0 TCP *:https (LISTEN)
cupsd 1896 root 9u IPv6 7294 0t0 TCP *:https (LISTEN)

cups??

Now the question is what do I do to fix it? Currently, cups is started as one
of the last things:

DAEMONS=(syslog-ng network hal named @netfs @ntpd @sshd @dhcpd @crond
@avahi-daemon @postfix @mysqld @dovecot @httpd @hylafax @samba @cups @sensors
@upsd kdm)

I guess I'll try taking them out of background processing.

The over 'Arching' point (no pun intended) is how can we make sure that
updates, that do not change any packages by adding/deleting, do not kill a
primary part of a box? And, why this box and not the backup?
--
David C. Rankin, J.D.,P.E.
Rankin Law Firm, PLLC
510 Ochiltree Street
Nacogdoches, Texas 75961
Telephone: (936) 715-9333
Facsimile: (936) 715-9339
www.rankinlawfirm.com
Ng Oon-Ee
2010-08-09 11:50:20 UTC
Permalink
Post by David C. Rankin
The over 'Arching' point (no pun intended) is how can we make sure that
updates, that do not change any packages by adding/deleting, do not kill a
primary part of a box? And, why this box and not the backup?
I think I've seen this said to you before, but the responsibility for
your box falls on you. JGC has answered your specific question, but I'd
comment on the broader assumption you make that the package maintainer
is responsible for your system in a specific manner. He is not.

Archers need to know their systems. You seem to, to some extent.
Therefore I don't really understand the request for (as I see it) a more
fool-proof system. You're not a fool.
David C. Rankin
2010-08-09 12:30:51 UTC
Permalink
Post by Ng Oon-Ee
I think I've seen this said to you before, but the responsibility for
your box falls on you. JGC has answered your specific question, but I'd
comment on the broader assumption you make that the package maintainer
is responsible for your system in a specific manner. He is not.
Archers need to know their systems. You seem to, to some extent.
Therefore I don't really understand the request for (as I see it) a more
fool-proof system. You're not a fool.
I agree with everything you said. The backgrounding issue was a bit surprising
having 2 identical boxes behave completely opposite due to a self-imposed bit of
cups Russian-roulette.

The request follows logically from what you have expressed. Archer need to know
their systems. When faced with unexpected behavior, I want to do my part and
understand what happened. Thus the question -- "What Changed?" More specifically
I guess would be "What changed in the updated packages such that cups went from
losing the race to grab port 443 for the past 18 months to suddenly 'winning'
the race for port 443 despite being backgrounded later in the DAEMONS line.

Logically something did it for cups to go from never getting the port to always
getting the port. That's what I was trying to learn and understand. I suspect
the only person that can answer that is the one of a few that know what
influences the port race and what change they made that would affect it.

Regardless, learning has occurred. An obscure cups setting has been changed,
lsof is better understood and my system is more robust as a result. A good start
to the day :p
--
David C. Rankin, J.D.,P.E.
Rankin Law Firm, PLLC
510 Ochiltree Street
Nacogdoches, Texas 75961
Telephone: (936) 715-9333
Facsimile: (936) 715-9339
www.rankinlawfirm.com
David C. Rankin
2010-08-09 09:49:35 UTC
Permalink
Post by David C. Rankin
Anyway, I need help. Any thoughts on what do check? I'll drop back to LTS
and see how that goes and report back. Thanks.
Jan,

Thank you for your prodding in the right direction. It was fricking cups that
was the problem and the work around was to stop cups and start apache and
restart cups:

[04:42 nirvana:/home/david] # lsof -i tcp:443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
cupsd 1896 root 8u IPv4 7293 0t0 TCP *:https (LISTEN)
cupsd 1896 root 9u IPv6 7294 0t0 TCP *:https (LISTEN)
[04:45 nirvana:/home/david] # rccups stop
/etc/rc.d/functions: line 303: .: /etc/rc.d/functions.d/functions.d: is a directory
:: Stopping CUPS Daemon
[DONE]
[04:45 nirvana:/home/david] # lsof -i tcp:443
[04:45 nirvana:/home/david] # rchttpd start
/etc/rc.d/functions: line 303: .: /etc/rc.d/functions.d/functions.d: is a directory
:: Starting Apache Web Server
[DONE]
[04:45 nirvana:/home/david] # rccups start
/etc/rc.d/functions: line 303: .: /etc/rc.d/functions.d/functions.d: is a directory
:: Starting CUPS Daemon
[DONE]

Now how do I fix this in rc.conf to make sure cups doesn't screw me over again?
Take http out of the background?

I've booted this box 15 times or so in the past 1.5 years and this has never
been a problem until the updates tonight. What changed? How many others are
going to get caught?
--
David C. Rankin, J.D.,P.E.
Rankin Law Firm, PLLC
510 Ochiltree Street
Nacogdoches, Texas 75961
Telephone: (936) 715-9333
Facsimile: (936) 715-9339
www.rankinlawfirm.com
Jan de Groot
2010-08-09 10:00:00 UTC
Permalink
Post by David C. Rankin
[04:45 nirvana:/home/david] # rccups start
/etc/rc.d/functions: line 303: .: /etc/rc.d/functions.d/functions.d: is a directory
:: Starting CUPS Daemon
[DONE]
Now how do I fix this in rc.conf to make sure cups doesn't screw me over again?
Take http out of the background?
I've booted this box 15 times or so in the past 1.5 years and this has never
been a problem until the updates tonight. What changed? How many others are
going to get caught?
First of all, why don't you fix that annoying error
with /etc/rc.d/functions.d that pops up on every rc script invocation?

As for cups: the default cupsd.conf contains "Listen localhost:631" and
"Listen /var/run/cups/cups.sock". I think your cupsd.conf contains an
entry for 443 also there. The problem with your setup is that your setup
relies on a failure: apache binds to https before cups can, so cups
doesn't bind to 443 anymore. Because you're using backgrounding, any
update to your system can interfere with timing and cause issues like
these if your configuration is not right.
David C. Rankin
2010-08-09 12:20:28 UTC
Permalink
Post by Jan de Groot
First of all, why don't you fix that annoying error
with /etc/rc.d/functions.d that pops up on every rc script invocation?
OK - you're on, don't keep me in suspense, just go ahead and tell me what notice
I missed that says do X or Y to fix the very annoying /etc/rc.d/functions.d that
pops up with every init...

I thought that was a side effect of the bashification of the init scrips and
that an update would fix it? wrong?
Post by Jan de Groot
As for cups: the default cupsd.conf contains "Listen localhost:631" and
"Listen /var/run/cups/cups.sock". I think your cupsd.conf contains an
entry for 443 also there.
Damn, your clairvoyant:

#
# "$Id: cupsd.conf.in 8805 2009-08-31 16:34:06Z mike $"
#
# Sample configuration file for the CUPS scheduler. See "man cupsd.conf" for a
# complete description of this file.
#

# Log general information in error_log - change "warn" to "debug"
# for troubleshooting...
ServerName nirvana.3111skyline.com
ServerAdmin ***@3111skyline.com
ServerAlias nirvana
ServerAlias www.3111skyline.com
ServerAlias localhost
# Allow remote access
Port 631
SSLPort 443
<snip>

now
#SSLPort 443

I have no recognition of making that change and don't know why I would have done
that. Could that be a host:631 cups auto setting in response to some other
selection?

The problem with your setup is that your setup
Post by Jan de Groot
relies on a failure: apache binds to https before cups can, so cups
doesn't bind to 443 anymore.
(I see that)

Because you're using backgrounding, any
Post by Jan de Groot
update to your system can interfere with timing and cause issues like
these if your configuration is not right.
Yes, I'm using backgrounding and my config hasn't changed since I first started
using Arch and was done in response to a post here pointing out that it was
recommended to bring up hal in the DAEMONS line and then background everything
thereafter up to your display manager. Was that advise incorrect?

I have been running with the following:

DAEMONS=(syslog-ng network hal named @netfs @ntpd @sshd @dhcpd @crond
@avahi-daemon @postfix @mysqld @dovecot @httpd @hylafax @samba @cups @sensors
@upsd kdm)

What needs to be changed? Or -- where is an Arch reference that provides the
pros & cons of backgrouding the various processes?

I guess I have been lucky because Arch has been bullet-proof since April '09 as
my primary server. Last night was the first DAEMONS issue I had run into except
for the little 'display manager respawning too fast' nit.

What say the experts on the backgrounding issue?
--
David C. Rankin, J.D.,P.E.
Rankin Law Firm, PLLC
510 Ochiltree Street
Nacogdoches, Texas 75961
Telephone: (936) 715-9333
Facsimile: (936) 715-9339
www.rankinlawfirm.com
Mauro Santos
2010-08-09 12:39:44 UTC
Permalink
Post by David C. Rankin
Post by Jan de Groot
First of all, why don't you fix that annoying error
with /etc/rc.d/functions.d that pops up on every rc script invocation?
OK - you're on, don't keep me in suspense, just go ahead and tell me
what notice I missed that says do X or Y to fix the very annoying
/etc/rc.d/functions.d that pops up with every init...
I thought that was a side effect of the bashification of the init scrips
and that an update would fix it? wrong?
Post by Jan de Groot
As for cups: the default cupsd.conf contains "Listen localhost:631" and
"Listen /var/run/cups/cups.sock". I think your cupsd.conf contains an
entry for 443 also there.
#
# "$Id: cupsd.conf.in 8805 2009-08-31 16:34:06Z mike $"
#
# Sample configuration file for the CUPS scheduler. See "man
cupsd.conf" for a
# complete description of this file.
#
# Log general information in error_log - change "warn" to "debug"
# for troubleshooting...
ServerName nirvana.3111skyline.com
ServerAlias nirvana
ServerAlias www.3111skyline.com
ServerAlias localhost
# Allow remote access
Port 631
SSLPort 443
<snip>
now
#SSLPort 443
I have no recognition of making that change and don't know why I would
have done that. Could that be a host:631 cups auto setting in response
to some other selection?
The problem with your setup is that your setup
Post by Jan de Groot
relies on a failure: apache binds to https before cups can, so cups
doesn't bind to 443 anymore.
(I see that)
Because you're using backgrounding, any
Post by Jan de Groot
update to your system can interfere with timing and cause issues like
these if your configuration is not right.
Yes, I'm using backgrounding and my config hasn't changed since I first
started using Arch and was done in response to a post here pointing out
that it was recommended to bring up hal in the DAEMONS line and then
background everything thereafter up to your display manager. Was that
advise incorrect?
@avahi-daemon @postfix @mysqld @dovecot @httpd @hylafax @samba @cups
@sensors @upsd kdm)
What needs to be changed? Or -- where is an Arch reference that
provides the pros & cons of backgrouding the various processes?
I guess I have been lucky because Arch has been bullet-proof since April
'09 as my primary server. Last night was the first DAEMONS issue I had
run into except for the little 'display manager respawning too fast' nit.
What say the experts on the backgrounding issue?
I'm not an expert but as I see it you should have seen it coming,
backgrounding the daemons is ok as long as the daemons you background
are independent.

On the listening port issue, you did that to yourself, you must have
explicitly configured cups to listen on port 443 and you configured
something else to listen on the same port, only two outcomes are
possible, either the applications fail gracefully and move on the best
wait they can or they will explode and refuse to work (which is always
lots of fun). Besides you should have noticed before that something
wasn't right, you did test your system to check if all the
functionalities you wanted to be working were actually working, right?

I guess now you know you can't have two programs listening on the same
port(s) and it's the sysadmin responsibility to assure that doesn't
happen or deal with it when the sticky smelly stuff hits the fan.
--
Mauro Santos
Loading...