When I use zip, how can I display the overall progress without flooding the command line?

I would better use:

cat INPUTFILE.img | pv -bep -s $(du -bs INPUTFILE.img | awk '{print $1}') | zip -q > OUTPUTFILE.zip

In that case you see real percentage of data through "PV" The only trouble now is file name, which can be solved:

echo -e "@ -n@=NEWNAME.img" | zipnote -w OUTPUTFILE.zip

Or both commands in one line:

cat INPUTFILE.img | pv -bep -s $(du -bs INPUTFILE.img | awk '{print $1}') | zip -q > OUTPUTFILE.zip ; echo -e "@ -n@=NEWNAME.img" | zipnote -w OUTPUTFILE.zip

Use (-q)[--quiet] option , like zip -q .... see man zip for more https://linux.die.net/man/1/zip

I would use grep expression

egrep -c '^([-d][rwx-]{9})'

At least for me on Ubuntu, no directory starts with "/" but rather with "d" (sure you could use [-d] only, but I was not sure about the recursive output of 'ls'.

drwxr-xr-x  3 user user 4096 apr 14  2015 gems

Also note we are not counting in hidden files here (or better said, the first instance of the command Lefteris wrote). And due to "." and ".." I would say the counts will be off (the more complex the directory tree, the worse estimate for the pv) if we use switch "-a". So that could also be somehow taken care of in the grep or perhaps in ls depending on your system.

If you're ok with using 7z:

7z a output.zip folder/

This will give you a progress bar like this:

Open archive: test.zip
--
Path = test.zip
Type = zip
Physical Size = 232039663

Scanning the drive:
3 folders, 2401 files, 238122225 bytes (228 MiB)

Updating archive: test.zip

Items to compress: 2404

 16% 279 U folder/file.txt  

I have successfully used the following:

zip -r [target_zip] [folder_to_zip] 2>&1 | 
pv -lep -s $(ls -Rl1 [folder_to_zip] | egrep -c '^[-/]') > /dev/null

And this is explained below:

zip -r [target_zip] [folder_to_zip] 2>&1 |

zip recursively into [target_zip] file the [folder_to_zip] redirecting stderr to stdout. Note that stderr will contain one line for each file and directory being processed.

pv -lep -s $(ls -Ral1 [folder_to_zip] | egrep -c '^[-/]') > /dev/null

pipe into pv the lines with the filenames as they are being output from zip. pv is operated in line mode (counting progress based on lines and size is also in number of lines to expect – see PV man page -l option).

The total size of lines to expect is gathered by recursively listing (ls) the [folder_to_zip] and counting the lines starting with '-' or 'd', i.e., all the files and directories (remember directories are listed starting with '/' ).

The above provides accurate percentage of completion as the 100% is reached when all files and directories have been processed.

The problem with the answer of pedroapero is that progress is calculated on number of bytes processed (compressed) over the total number of bytes to process (uncompressed). As a result the process will complete at around 30% (depending on the compression rate).

zip can compress data to standard output. Hence, you can combine it with other tools like pv:

zip -qr - [folder] | pv -bep -s $(du -bs [folder] | awk '{print $1}') > [file.zip]

Remove one of the -bep options as your convenience.