[Feature Suggestion] progress meter (pv subcommand)

[terefang]

hello, again

you may consider this eye-candy, but it is some quality of life improvement.

use case (implementation)

pv

the pv command -- https://linux.die.net/man/1/pv

$ pv --help
Usage: pv [OPTION] [FILE]...
Concatenate FILE(s), or standard input, to standard output,
with monitoring.

  -p, --progress           show progress bar
  -t, --timer              show elapsed time
  -e, --eta                show estimated time of arrival (completion)
  -I, --fineta             show absolute estimated time of arrival 
                           (completion)
  -r, --rate               show data transfer rate counter
  -a, --average-rate       show data transfer average rate counter
  -b, --bytes              show number of bytes transferred
  -T, --buffer-percent     show percentage of transfer buffer in use
  -A, --last-written NUM   show NUM bytes last written
  -F, --format FORMAT      set output format to FORMAT
  -n, --numeric            output percentages, not visual information
  -q, --quiet              do not output any transfer information at all

  -W, --wait               display nothing until first byte transferred
  -D, --delay-start SEC    display nothing until SEC seconds have passed
  -s, --size SIZE          set estimated data size to SIZE bytes
  -l, --line-mode          count lines instead of bytes
  -0, --null               lines are null-terminated
  -i, --interval SEC       update every SEC seconds
  -w, --width WIDTH        assume terminal is WIDTH characters wide
  -H, --height HEIGHT      assume terminal is HEIGHT rows high
  -N, --name NAME          prefix visual information with NAME
  -f, --force              output even if standard error is not a terminal
  -c, --cursor             use cursor positioning escape sequences

  -L, --rate-limit RATE    limit transfer to RATE bytes per second
  -B, --buffer-size BYTES  use a buffer size of BYTES
  -C, --no-splice          never use splice(), always use read/write
  -E, --skip-errors        skip read errors in input
  -S, --stop-at-size       stop after --size bytes have been transferred
  -R, --remote PID         update settings of process PID

  -P, --pidfile FILE       save process ID in FILE

  -d, --watchfd PID[:FD]   watch file FD opened by process PID

  -h, --help               show this help and exit
  -V, --version            show version information and exit

re-use case/synergy

you could reuse the pv implementation for the status output of the dd command

prior art

gnu shtool prop

markdown cannot show it correctly but the propellor will be printed on the same line

$ dd .... | shtool prop -p "processing" 2>&1|tr '\r' '\n'
processing...|
processing.../
processing...-
processing...\
processing...|
processing.../
processing...-
processing...\
processing...|
processing.../
processing...-
.....

dd

modern dd has the status=progress option that print transfer stats every second like this:

$ dd ..... status=progress 2>&1|tr '\r' '\n'
271581184 bytes (272 MB, 259 MiB) copied, 1 s, 271 MB/s
551550976 bytes (552 MB, 526 MiB) copied, 2 s, 276 MB/s
832569344 bytes (833 MB, 794 MiB) copied, 3 s, 277 MB/s
1110441984 bytes (1,1 GB, 1,0 GiB) copied, 4 s, 278 MB/s
1391460352 bytes (1,4 GB, 1,3 GiB) copied, 5 s, 278 MB/s
1671430144 bytes (1,7 GB, 1,6 GiB) copied, 6 s, 279 MB/s
1951399936 bytes (2,0 GB, 1,8 GiB) copied, 7 s, 279 MB/s
2231369728 bytes (2,2 GB, 2,1 GiB) copied, 8 s, 279 MB/s
2509242368 bytes (2,5 GB, 2,3 GiB) copied, 9 s, 279 MB/s
2790260736 bytes (2,8 GB, 2,6 GiB) copied, 10 s, 279 MB/s
3070230528 bytes (3,1 GB, 2,9 GiB) copied, 11 s, 279 MB/s
3351248896 bytes (3,4 GB, 3,1 GiB) copied, 12 s, 279 MB/s
3633315840 bytes (3,6 GB, 3,4 GiB) copied, 13 s, 279 MB/s
3914334208 bytes (3,9 GB, 3,6 GiB) copied, 14 s, 280 MB/s
.....

wget

wget has:

• --progress=dot – traces the retrieval by printing dots on the screen, each dot representing a fixed amount of downloaded data.
• --progress=bar – draws an ASCII progress bar graphics (a.k.a "thermometer" display) indicating the status of retrieval. If the output is not a TTY, the "dot" type will be used by default.

$ wget --progress=dot .....

     0K .......... .......... .......... .......... ..........  1%  194K 17s
    50K .......... .......... .......... .......... ..........  3%  394K 12s
.....
  3200K .......... .......... .......... .......... .......... 99% 11,9M 0s
  3250K .......... .........                                  100%

$ wget --progress=bar .....
FILE             0%[....................]       0  --.-KB/s               
FILE             1%[....................]  43,69K   172KB/s               
FILE             4%[....................] 159,69K   311KB/s               
FILE            18%[==>.................] 595,69K   774KB/s               
FILE            35%[======>.............]   1,13M  1,10MB/s               
FILE            53%[=========>..........]   1,70M  1,38MB/s               
FILE            65%[============>.......]   2,10M  1,47MB/s               
FILE            83%[===============>....]   2,66M  1,63MB/s               
FILE           100%[===================>]   3,19M  1,75MB/s

curl

curl has a simple progress bar

$ curl -# .... 2>&1|tr '\r' '\n'
#=#=-#                                                                                   
##O#- #                                                                                  
#=#=#   #                                                                                
-#O#- #   #                                                                              

                                                                                      0,2%
                                                                                      1,1%
##                                                                                    2,8%
#####                                                                                 6,2%
##########                                                                           13,2%
#####################                                                                25,5%
###############################                                                      38,5%
###########################################                                          52,0%
#####################################################                                64,1%
##########################################################                           69,9%
####################################################################                 82,8%
################################################################################     97,3%
################################################################################### 100,0%

[landley]

So you want toybox's "count -l" to have an -L 8123748 variant so it knows how much data to expect and can do a progress bar and percentage complete?

As for "prior art", one of my first interactions with busybox back in the day was submitting my little "count" toy to them I'd been using since the 90's http://lists.busybox.net/pipermail/busybox/2003-October/043590.html and the immediate suggestion was of course to add a progress bar ala http://lists.busybox.net/pipermail/busybox/2003-October/043613.html and then there was bikeshedding on the list and the IRC channel and a fw weeks later the concept went in as https://git.busybox.net/busybox/commit/?id=e9080c9f4170 and then there was more bikeshedding I largely stayed out of because I'd been happy with "count"...

[terefang]

that is why i did not use the word "request" but rather "suggestion".

and i have to confess that i totally missed "count".

[terefang]

So you want toybox's "count -l" to have an -L 8123748 variant so it knows how much data to expect and can do a progress bar and percentage complete?

lets skip the discussion about a (graphical) progress meter a bit.

i have looked at count -l and it is a wonderful headstart i did not notice.

given that, a -L <expected-total> would be nice to show a "percentage complete".

looking at wc ... would it make sense to have an option to count lines (\n) instead ?
i have a number of use cases where a script reads line-wise from a file, calculating and showing progress.

[landley]

You mean give count a line counting option? I could make -L do line counting and -p do progress bar, sure. Neither is hard, it's just... you listed 33 options for "pv" above and I'd like to know where the edges of the problem space are before extending much further. (Looks like pv uses -s for estimated size, might as well match that.)

Let's see...

-p and maybe -eI are implied by -s, -trab is default, -l you just requested and I guess -0 goes with that.

But -TA are silly, I'm not doing -F, -nwDiBC are micromanaging, what is -q FOR (don't put it in the pipeline then, cat exists), -wH have environment variables that already do that, -f implies that we care whether stderr is a terminal which is not currently the case, -E is similarly out of scope, -c is toybox policy, -RPd are just off in la-la land, and -hV have --long versions as part of default toybox plumbing so aren't an individual command's job.

You could make an argument for adding -LS (which would explain -q) but that's not really this command's job? Ratelimiting is usually bidirectional, so it's more a netcat -f thing? Hmmm. I suppose you could make the case that we've got the plumbing so it's cheap for us to do...

So maybe -sl0LSq options. And maybe test that "zcat blah | COLUMNS=47 count" works right, although defaulting to 80 seems pretty reasonable too...

P.S. I usually try to have this kind of discussion on the mailing list. Microsoft github will eventually go the way of all web services and this history will be lost, with no obvious standard export format to migrate the archive elsewhere.

[terefang]

i will subscribe