No fun funzipping

Anything QL Software or Programming Related.
User avatar
pjw
Gold Card
Posts: 379
Joined: Fri Jul 11, 2014 8:44 am
Location: Norway

No fun funzipping

Postby pjw » Mon Jan 29, 2018 10:34 am

In some projects I use zip/funzip to archive certain material on the fly via pipes. But what takes zip about a second to do, it takes funzip 10 seconds to undo! The same material can be unzipped with the normal unzip in about a second or two. Im using the latest known versions of Jonathan Hudson's zip suit V2.3 for QDOS/SMSQ/E.

So my questions are: Why? Can I fix it? Can anyone else fix it? Is there something else I could use that does more or less the same? (I dont need individual files, search, encryption etc, just a pure stream of data going in, being compressed, and coming out - and the reverse, of course). Is there a way of fooling unzip to do the same - without the use of intermediary files?

TIA


Per
For every complex problem there is an answer that is clear, simple, and wrong.
- H. L. Mencken
User avatar
tofro
QL Wafer Drive
Posts: 1372
Joined: Sun Feb 13, 2011 10:53 pm
Location: SW Germany

Re: No fun funzipping

Postby tofro » Mon Jan 29, 2018 12:38 pm

Per,

did you try "unzip -p" instead of "funzip" (which should basically do the same thing?

Tobias


User avatar
pjw
Gold Card
Posts: 379
Joined: Fri Jul 11, 2014 8:44 am
Location: Norway

Re: No fun funzipping

Postby pjw » Mon Jan 29, 2018 3:23 pm

tofro wrote:did you try "unzip -p" instead of "funzip" (which should basically do the same thing?

I was discouraged, as the manual stated "Archives read from standard input are not yet supported,
except with funzip (and then only the first member of the archive can be extracted)."

I did try anyway, but no arrangement of parameters or (pre-opened) channels produced the desired
effect:

Code: Select all

500 id = FEX_M('win2_arc_zip_funzip', fnm$, 'pipe_in_2048')
510 cpi = FOP_IN("pipe_in")
..

BTW: My unzip version was probably 5.2


Per
For every complex problem there is an answer that is clear, simple, and wrong.
- H. L. Mencken
User avatar
pjw
Gold Card
Posts: 379
Joined: Fri Jul 11, 2014 8:44 am
Location: Norway

Re: No fun funzipping

Postby pjw » Tue Jan 30, 2018 7:37 pm

Ok, over to plan B: Does anyone have a current email address for Jonathan Hudson? The last contact I had with him was in 1998, and that address no longer works. Please PM it to me if you can.


Per
For every complex problem there is an answer that is clear, simple, and wrong.
- H. L. Mencken
User avatar
XorA
Gold Card
Posts: 490
Joined: Thu Jun 02, 2011 11:31 am
Location: Shotts, North Lanarkshire, Scotland, UK

Re: No fun funzipping

Postby XorA » Wed Jan 31, 2018 8:24 pm

Is this a streaming speed issue, because zip algorithm needs to read the end of archive first!


User avatar
pjw
Gold Card
Posts: 379
Joined: Fri Jul 11, 2014 8:44 am
Location: Norway

Re: No fun funzipping

Postby pjw » Thu Feb 01, 2018 12:08 am

XorA wrote:Is this a streaming speed issue, because zip algorithm needs to read the end of archive first!

You cracked it! :) Thanks a lot!
Increasing the pipe size to 32768 reduced the time to a fraction, even for archives many times this size. (I tried pipe sizes beyond that but it didnt appear to make any difference..)


Per
For every complex problem there is an answer that is clear, simple, and wrong.
- H. L. Mencken
User avatar
mk79
Over Heated PSU
Posts: 137
Joined: Sun Feb 02, 2014 10:54 am

Re: No fun funzipping

Postby mk79 » Fri Feb 02, 2018 6:33 am

Have a look at bzcat/bzip2, it's specifically for compressing/decompressing streams of data. I've used it to stream huge amounts using SBasic before.


User avatar
pjw
Gold Card
Posts: 379
Joined: Fri Jul 11, 2014 8:44 am
Location: Norway

Re: No fun funzipping

Postby pjw » Fri Feb 02, 2018 11:29 am

mk79 wrote:Have a look at bzcat/bzip2, it's specifically for compressing/decompressing streams of data. I've used it to stream huge amounts using SBasic before.
Thanks for the tip :)
I checked out the version currently on Dilwyn's site. This is what I found:
Sample: 64 files @ 32792b == 2Mb to be squashed into a single archive, tmp$
:
bzip2:

Code: Select all

..
cpo = FOPEN("pipe_out_2048")
id = FEX_M('win2_arc_bzp_bzip2', 'pipe_out', 'nul', tmp$)
<stuff data into pipe>
..
and
..
id = FEX_M('win2_arc_bzp_bzcat', tmp$, 'nul', 'pipe_in_32768')
<get data out of pipe>
..

pipe_out 32k, zip t= 17s, unzip pipe_in 32k, t 1s
pipe_out 2k, zip t= 17s, unzip pipe_in 32k, t 1s
pipe_out 2k, zip t= 17s, unzip pipe_in 10k, t 1s
pipe_out 2k, zip t= 17s, unzip pipe_in 5k, t 1s
pipe_out 2k, zip t= 17s, unzip pipe_in 2k, t >5min or choked

Compressed size: 183440

zip:

Code: Select all

..
cpo = FOPEN("pipe_out_2048")
id = FEX_M('win2_arc_zip_exe', 'pipe_out', 'nul', 'ram1_tst_zip')
..
and
..
id = FEX_M('win2_arc_zip_funzip', tmp$, 'nul', 'pipe_in_32768')
..

pipe_out 32k, zip t= 2s, unzip pipe_in 32k, t 2s
pipe_out 2k, zip t= 2s, unzip pipe_in 32k, t 2s
pipe_out 2k, zip t= 2s, unzip pipe_in 10k, t 4s
pipe_out 2k, zip t= 2s, unzip pipe_in 5k, t 10s

Compressed size: 176478

183440 - 176478 = 6962 in zip's favour

Conclusion: For the kind of data I tested, zip compresses faster and meaner,
but is slow at decompression. bzip is super fast at decompressing. Sadly bzcat
doesnt decompress zipped files!

Different settings may give different results! If you or anyone knows how any of
this may be improved, please post it here! For now Ill stick with zip/funzip


Per
For every complex problem there is an answer that is clear, simple, and wrong.
- H. L. Mencken

Return to “Software & Programming”

Who is online

Users browsing this forum: No registered users and 4 guests