Thursday, December 16, 2021

SD Card File Transfers Done Carefully

 With USB2 SD card readers there really isn't any need to slow them down, they are already not fast.  But if you are like me and you like fast file transfers you probably looked for and found a USB3 SD card reader.  

Unfortunately some combos of SD cards and SD card readers bog down real bad when large transfers are done at high speed. On Linux, this is the solution I use:

tar -cf - {source_files} | pv -q -L {transfer_rate_limit} | tar -C {destination_files} -xvf -

The {transfer_rate_limit} is specified in bytes/sec; a suffix (k, m, g, or t) can be added to the end for specifying KiB, MiB, GiB, or TiB's per second. 

I got this from Matt on stack exchange.

Another use for this: If the transfer is being done between two modern computers, you can speed up the transfer by adding encryption at the source and decryption at the destination, the 'z' is for gzip which is fairly fast:

tar -czf - {source_files} | pv -q -L {transfer_rate_limit} | tar -C {destination_files} -xzvf -

lz4 compresses smaller and much faster, as show by CatchChallenger, but make sure it's installed and supported by your version of tar: 

tar -I lz4 -cf - {source_files} | pv -q -L {transfer_rate_limit} | tar -C {destination_directory} -I lz4 -xvf -

It's a long command with lots of fiddly bits; this is exactly how UNIX was initially designed.  It was supposed to be a system where you can string fairly simple programs together to accomplish complex tasks.  Check out Brian Kernighan talking about it on YouTube.

I haven't figured out what the best speed to transfer at is, I assume it depends on your card and card reader.  The program "time" will measure how long the command takes to run:


anon@grayghost:~$ time tar -I lz4 -cf - ToDo.txt | pv -q -L 4096 | tar -C /home/anon/Sy/ -I lz4 -xvf -
ToDo.txt

real 0m0.015s
user 0m0.009s
sys 0m0.016s
anon@grayghost:~$ time tar -I lz4 -cf - ToDo.txt | pv -q -L 1024 | tar -C /home/anon/Sy/ -I lz4 -xvf -
ToDo.txt

real 0m0.287s
user 0m0.016s
sys 0m0.019s
anon@grayghost:~$

 

Neat!


 
 

No comments:

Post a Comment