Discussion:
pacman doesn't show download progress
(too old to reply)
SET
2017-08-15 19:37:23 UTC
Permalink
Raw Message
pacman -Syu show database download progress but not packages being downloaded.
They do get downloaded and installation progress is rightly displayed. This
can be annoying when connection speed is poor. This has been observed on three
machines.

I am obviously requesting a solution and am ready to provide any additional
required information.

Thanks.
Eli Schwartz
2017-08-15 19:43:46 UTC
Permalink
Raw Message
Post by SET
pacman -Syu show database download progress but not packages being downloaded.
They do get downloaded and installation progress is rightly displayed. This
can be annoying when connection speed is poor. This has been observed on three
machines.
I am obviously requesting a solution and am ready to provide any additional
required information.
I'm not sure what to say, other than that I see progress every time I
download packages, unless they are already cached and therefore do not
need to be downloaded.

Have you set an external downloader?
--
Eli Schwartz
SET
2017-08-15 21:04:35 UTC
Permalink
Raw Message
Post by Eli Schwartz
Have you set an external downloader?
No, I'm using the default downloader.

[options]
#RootDir = /
#DBPath = /var/lib/pacman/
#CacheDir = /var/cache/pacman/pkg/
#LogFile = /var/log/pacman.log
#GPGDir = /etc/pacman.d/gnupg/
HoldPkg = pacman glibc
#XferCommand = /usr/bin/curl -C - -f %u > %o
#XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
#CleanMethod = KeepInstalled
#UseDelta = 0.7
Architecture = auto
IgnoreGroup = xorg
IgnorePkg = xf86-input-keyboard xf86-input-mouse xf86-video-vmware
IgnorePkg = qt5-doc minuet ocaml
#NoUpgrade =
#NoExtract =
# Misc options
#UseSyslog
Color
#TotalDownload
CheckSpace
Moses Miller via arch-general
2017-08-15 21:05:44 UTC
Permalink
Raw Message
Can you explain in more detail what the problem is, and show actual and
expected output?
Post by SET
Post by Eli Schwartz
Have you set an external downloader?
No, I'm using the default downloader.
[options]
#RootDir = /
#DBPath = /var/lib/pacman/
#CacheDir = /var/cache/pacman/pkg/
#LogFile = /var/log/pacman.log
#GPGDir = /etc/pacman.d/gnupg/
HoldPkg = pacman glibc
#XferCommand = /usr/bin/curl -C - -f %u > %o
#XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
#CleanMethod = KeepInstalled
#UseDelta = 0.7
Architecture = auto
IgnoreGroup = xorg
IgnorePkg = xf86-input-keyboard xf86-input-mouse xf86-video-vmware
IgnorePkg = qt5-doc minuet ocaml
#NoUpgrade =
#NoExtract =
# Misc options
#UseSyslog
Color
#TotalDownload
CheckSpace
Ismael Bouya
2017-08-16 09:13:26 UTC
Permalink
Raw Message
Post by Moses Miller via arch-general
Can you explain in more detail what the problem is, and show actual and
expected output?
Hello,
I have the same "problem" for a few weeks too.
Here is the output of pacman -Syu I got just now on two machines:
https://paste.ee/p/hqCTu
https://paste.ee/p/eLu01

In the first one, you see download progress on the packages.
In the second one you don't see them. Here you don't see them at all (probably
because there are only a few packages), but it happens that I only see a few of
them. The behavior is completely random, and on both machines: it's just luck
that I could get the two different examples today

The only line that differs between both machines in pacman.conf is IgnorePkg:
----------------------------------
[options]
HoldPkg = pacman glibc
IgnorePkg = (...)
Architecture = auto
Color
CheckSpace
SigLevel = Required DatabaseOptional
LocalFileSigLevel = Optional
[core]
Include = /etc/pacman.d/mirrorlist
[extra]
Include = /etc/pacman.d/mirrorlist
[community]
Include = /etc/pacman.d/mirrorlist
[multilib]
Include = /etc/pacman.d/mirrorlist
------------------------------------

[NB: I "think" it's related to curl (didn't check if pacman uses it though)
since I have a similar experience once with that command in a non-related use.]

Hope it can help.
--
Ismael
Darshit Shah via arch-general
2017-08-16 15:09:11 UTC
Permalink
Raw Message
Could you also share the first few lines of your mirrorlist file?

I don't have much idea about how `pacman` is implemented, so what I'm
saying next is pure guesswork and speculation.

My guess is that there exist some mirror(s) that return a bad HTTP
Response, that is a response without a file-size. A HTTP/1.1 response,
almost always includes a "Content-Length" header which tells the client
program what the total size of the file is. This data is also obviously
important for drawing an accurate progress bar. Now, if the
"Content-Length" header was missing, pacman cannot know the final size
of the file before hand and hence it is not drawing the progress bars.

As a test, I would request that the next time you see something like
this, go through these steps:

1. Delete the just downloaded package from the cache, so that pacman
will download it again
2. Replace the XferCommand line with this:
`XferCommand = /usr/bin/wget --show-progress -q --passive-ftp -c -S
-o %o.log -O %o %u`
3. Run pacman again

This will create a log file containing the Server's response next to the
package in the pacman cache. Take a look at the log file and also please
share the contents here with us. If my guess is correct, we should be
able to see this in the log file, else, we find another way to debug
this
Post by Ismael Bouya
Post by Moses Miller via arch-general
Can you explain in more detail what the problem is, and show actual and
expected output?
Hello,
I have the same "problem" for a few weeks too.
https://paste.ee/p/hqCTu
https://paste.ee/p/eLu01
In the first one, you see download progress on the packages.
In the second one you don't see them. Here you don't see them at all (probably
because there are only a few packages), but it happens that I only see a few of
them. The behavior is completely random, and on both machines: it's just luck
that I could get the two different examples today
----------------------------------
[options]
HoldPkg = pacman glibc
IgnorePkg = (...)
Architecture = auto
Color
CheckSpace
SigLevel = Required DatabaseOptional
LocalFileSigLevel = Optional
[core]
Include = /etc/pacman.d/mirrorlist
[extra]
Include = /etc/pacman.d/mirrorlist
[community]
Include = /etc/pacman.d/mirrorlist
[multilib]
Include = /etc/pacman.d/mirrorlist
------------------------------------
[NB: I "think" it's related to curl (didn't check if pacman uses it though)
since I have a similar experience once with that command in a non-related use.]
Hope it can help.
--
Ismael
--
Thanking You,
Darshit Shah
PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6
n***@netcourrier.com
2017-08-16 15:55:26 UTC
Permalink
Raw Message
Post by Darshit Shah via arch-general
XferCommand = /usr/bin/wget --show-progress -q --passive-ftp -c -S
-o %o.log -O %o %u
Where will be the log file ? current working directory ?
Darshit Shah via arch-general
2017-08-17 04:46:37 UTC
Permalink
Raw Message
Post by n***@netcourrier.com
Post by Darshit Shah via arch-general
XferCommand = /usr/bin/wget --show-progress -q --passive-ftp -c -S
-o %o.log -O %o %u
Where will be the log file ? current working directory ?
No, right next to the packages, in the package cache
(/var/cache/pacman/pkg by default)
--
Thanking You,
Darshit Shah
PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6
SET
2017-08-17 18:23:38 UTC
Permalink
Raw Message
Post by Darshit Shah via arch-general
No, right next to the packages, in the package cache
(/var/cache/pacman/pkg by default)
Though the problem is resolved by choosing another mirror, here is a sample
log file : oxygen-icons-1:5.37.0-1-any.pkg.tar.xz.part.log

HTTP/1.1 200 OK
Server: nginx/1.6.2
Date: Thu, 17 Aug 2017 18:14:56 GMT
Content-Type: application/octet-stream
X-Varnish: 537330300
Age: 0
Via: 1.1 varnish-v4
Transfer-Encoding: chunked
Accept-Ranges: bytes
X-IPLB-Instance: 5865

I don't know how to understand it.
Eli Schwartz
2017-08-17 18:43:33 UTC
Permalink
Raw Message
Post by SET
Post by Darshit Shah via arch-general
No, right next to the packages, in the package cache
(/var/cache/pacman/pkg by default)
Though the problem is resolved by choosing another mirror, here is a sample
log file : oxygen-icons-1:5.37.0-1-any.pkg.tar.xz.part.log
HTTP/1.1 200 OK
Server: nginx/1.6.2
Date: Thu, 17 Aug 2017 18:14:56 GMT
Content-Type: application/octet-stream
X-Varnish: 537330300
Age: 0
Via: 1.1 varnish-v4
Transfer-Encoding: chunked
Accept-Ranges: bytes
X-IPLB-Instance: 5865
I don't know how to understand it.
Well, look at this:

```
[***@arch ~]$ pacman -Sp linux
http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i686.pkg.tar.xz
[***@arch ~]$ wget -P /tmp/
http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i686.pkg.tar.xz
--2017-08-17 14:31:17--
http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i686.pkg.tar.xz
Resolving archlinux.mirrors.ovh.net... 213.32.5.7,
2001:41d0:202:100:213:32:5:7
Connecting to archlinux.mirrors.ovh.net|213.32.5.7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘/tmp/linux-4.12.7-1-i686.pkg.tar.xz’

linux-4.12.7-1-i686.pkg.tar.xz [ <=>
] 1.37M 912KB/s

```

Length is unspecified, and wget does its <=> slidy-back-and-forth thing.
And it seems pacman just gives up on showing progress information and
instead shows nothing (which isn't unreasonable).

As for your log, that's just the HTTP headers, something you can also
see with curl -I http://...

Notice how the Content-Length: header is missing.
--
Eli Schwartz
Eli Schwartz
2017-08-17 18:46:06 UTC
Permalink
Raw Message
Post by Eli Schwartz
```
http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i686.pkg.tar.xz
http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i686.pkg.tar.xz
--2017-08-17 14:31:17--
http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i686.pkg.tar.xz
Resolving archlinux.mirrors.ovh.net... 213.32.5.7,
2001:41d0:202:100:213:32:5:7
Connecting to archlinux.mirrors.ovh.net|213.32.5.7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘/tmp/linux-4.12.7-1-i686.pkg.tar.xz’
linux-4.12.7-1-i686.pkg.tar.xz [ <=>
] 1.37M 912KB/s
```
Length is unspecified, and wget does its <=> slidy-back-and-forth thing.
And it seems pacman just gives up on showing progress information and
instead shows nothing (which isn't unreasonable).
As for your log, that's just the HTTP headers, something you can also
see with curl -I http://...
Notice how the Content-Length: header is missing.
... and it's probably worth mentioning that sometime in the last 10
minutes they seem to have fixed this.
--
Eli Schwartz
Eli Schwartz
2017-08-17 19:48:49 UTC
Permalink
Raw Message
Post by Eli Schwartz
... and it's probably worth mentioning that sometime in the last 10
minutes they seem to have fixed this.
Or not. It's erratically switching between working and not-working.
--
Eli Schwartz
Guus Snijders via arch-general
2017-08-17 19:58:10 UTC
Permalink
Raw Message
Post by Eli Schwartz
... and it's probably worth mentioning that sometime in the last 10
minutes they seem to have fixed this.
Or not. It's erratically switching between working and not-working.


So it's probably one faulty server behind a load balancer? Seems strange
behaviour for a single machine.


Mvg, Guus Snijders
n***@netcourrier.com
2017-08-18 08:11:23 UTC
Permalink
Raw Message
Post by Eli Schwartz
As for your log, that's just the HTTP headers, something you can also
see with curl -I http://...
Notice how the Content-Length: header is missing.
Thanks for the clear explanations, at least I understand what's happening,
it's up to remote to fix.
mpan
2017-08-18 09:55:41 UTC
Permalink
Raw Message
Post by n***@netcourrier.com
Post by Eli Schwartz
Notice how the Content-Length: header is missing.
Thanks for the clear explanations, at least I understand what's happening,
it's up to remote to fix.
Don’t write them to fix that, as nothing is broken: Content-Length is
not required.
n***@netcourrier.com
2017-08-18 10:17:01 UTC
Permalink
Raw Message
Don’t write them to fix that, as nothing is broken: Content-Length is
not required.
Of course I won't, ;-)

SET
2017-08-16 17:20:41 UTC
Permalink
Raw Message
Post by Darshit Shah via arch-general
Could you also share the first few lines of your mirrorlist file?
The first few lines verbatim :

Server = http://archlinux.mirrors.ovh.net/archlinux/$repo/os/$arch
#Server = http://mirror.archlinux.ikoula.com/archlinux/$repo/os/$arch
Server = http://archlinux.polymorf.fr/$repo/os/$arch
##
## Arch Linux repository mirrorlist
## Generated on 2016-03-10
##

## Worldwide
#Server = http://mirror.rackspace.com/archlinux/$repo/os/$arch

## Australia
#Server = http://mirror.aarnet.edu.au/pub/archlinux/$repo/os/$arch
SET
2017-08-16 18:41:41 UTC
Permalink
Raw Message
Post by Darshit Shah via arch-general
My guess is that there exist some mirror(s) that return a bad HTTP
Response, that is a response without a file-size.
You're probably right. pacman has expected output if I change remote mirror.
It had to download 150 MB, I interrupted the download and restarted many
times. No output while downloading with one mirror, and normal output with
another one. At least up to now.

Thanks.
Bartłomiej Piotrowski
2017-08-16 18:48:59 UTC
Permalink
Raw Message
Post by SET
Post by Darshit Shah via arch-general
My guess is that there exist some mirror(s) that return a bad HTTP
Response, that is a response without a file-size.
You're probably right. pacman has expected output if I change remote mirror.
It had to download 150 MB, I interrupted the download and restarted many
times. No output while downloading with one mirror, and normal output with
another one. At least up to now.
Thanks.
Which mirror was it? We will try to contact them about the issue.

B
SET
2017-08-16 19:47:58 UTC
Permalink
Raw Message
Post by Bartłomiej Piotrowski
Which mirror was it? We will try to contact them about the issue.
B
It is :
http://archlinux.mirrors.ovh.net/archlinux/$repo/os/$arch

Today at least, the following mirror is working fine :
http://archlinux.polymorf.fr/$repo/os/$arch

Thx.
Florian Pritz via arch-general
2017-08-17 07:47:04 UTC
Permalink
Raw Message
Post by SET
Post by Bartłomiej Piotrowski
Which mirror was it? We will try to contact them about the issue.
B
http://archlinux.mirrors.ovh.net/archlinux/$repo/os/$arch
I've notified the mirror admin. I'll post here again if I know more.

Florian
Ralf Mardorf
2017-08-15 23:02:02 UTC
Permalink
Raw Message
Post by SET
# Misc options
#UseSyslog
Color
#TotalDownload
CheckSpace
Here a progress bar is displayed, but I'm using the "ILoveCandy" Easter
egg.

[***@archlinux ~]# grep Misc\ options -A8 /etc/pacman.conf
# Misc options
#UseSyslog
#UseDelta
#TotalDownload
CheckSpace
#VerbosePkgLists
Color
ILoveCandy

[***@archlinux ~]# pacman -Q pacman
pacman 5.0.2-2
Loading...