SD Card File Transfers Done Carefully

SEE WARNING BELOW, DON'T DO WHAT I SUGGEST IN THIS POST!

 With USB2 SD card readers there really isn't any need to slow them down, they are already not fast.  But if you are like me and you like fast file transfers you probably looked for and found a USB3 SD card reader.  

Unfortunately some combos of SD cards and SD card readers bog down real bad when large transfers are done at high speed. On Linux, this is the solution I use:

tar -cf - {source_files} | pv -q -L {transfer_rate_limit} | tar -C {destination_files} -xvf -

The {transfer_rate_limit} is specified in bytes/sec; a suffix (k, m, g, or t) can be added to the end for specifying KiB, MiB, GiB, or TiB's per second. 

I got this from Matt on stack exchange.

Another use for this: If the transfer is being done between two modern computers, you can speed up the transfer by adding encryption at the source and decryption at the destination, the 'z' is for gzip which is fairly fast:

tar -czf - {source_files} | pv -q -L {transfer_rate_limit} | tar -C {destination_files} -xzvf -

lz4 compresses smaller and much faster, as show by CatchChallenger, but make sure it's installed and supported by your version of tar: 

tar -I lz4 -cf - {source_files} | pv -q -L {transfer_rate_limit} | tar -C {destination_directory} -I lz4 -xvf -

It's a long command with lots of fiddly bits; this is exactly how UNIX was initially designed.  It was supposed to be a system where you can string fairly simple programs together to accomplish complex tasks.  Check out Brian Kernighan talking about it on YouTube.

I haven't figured out what the best speed to transfer at is, I assume it depends on your card and card reader.  The program "time" will measure how long the command takes to run:

anon@grayghost:~$ time tar -I lz4 -cf - ToDo.txt | pv -q -L 4096 | tar -C /home/anon/Sy/ -I lz4 -xvf -
ToDo.txt

real 0m0.015s
user 0m0.009s
sys 0m0.016s
anon@grayghost:~$ time tar -I lz4 -cf - ToDo.txt | pv -q -L 1024 | tar -C /home/anon/Sy/ -I lz4 -xvf -
ToDo.txt

real 0m0.287s
user 0m0.016s
sys 0m0.019s
anon@grayghost:~$

 Neat!

Update; works great with making ISO images with dd:

dd bs=1M if=[image_name] | pv -q -L 10M | dd of=/[your_director_here]
(I haven't tested that dd command YMMV)

I just ran into a problem.  The contents of my SD card was corrupted.  Transferring data slowly off of the card actually heated it up to 164°F, while just letting it run was only 99°F.  I don't yet know if the heat and transfer speed was source of the problem.  Until I can figure it out avoid, or at least be a bit cautious, using this technique!