Use binary diff tools like xdelta, bsdiff/bspatch to transfer large files

Tags: , , ,

What do you do if you have to frequently transfer large files over a slow connection as part of your work? Perhaps new builds produced by your corporate build server need to be deployed at a different location for the QA team? I have seen such scenarios causing huge amounts of wasted time for multiple […]

Continue reading » 4 Comments

/dev/random vs /dev/urandom

Tags: , , ,

If you want random data in a Linux/Unix type OS, the standard way to do so is to use /dev/random or /dev/urandom. These devices are special files. They can be read like normal files and the read data is generated via multiple sources of entropy in the system which provide the randomness. /dev/random will block […]

Continue reading » 1 Comment

How to create a file of arbitrary size with shell script commands

Tags: , , , ,

A few days ago I was working on writing Java code to transfer files via SFTP and FTPS. As part of the test cases, I wanted to try and transfer files of small medium and large sizes like 0 byte, 1 byte, 100 bytes, 1000 bytes, 1MB, 10MB. How would one go about creating files […]

Continue reading » 3 Comments

Check for corruption of files using md5 checksum

Tags: , , , , , ,

You just transferred a large multi-gigabit file from your build server to your QA server. You want to check if the files are exactly the same. Data corruption can happen for many reasons! What do you do? Remember, just because two files are the same size, it doesn’t mean that they are the same. It […]

Continue reading » 2 Comments

How to split a file, process the pieces in multiple threads and combine results using a shell script

Tags: , , , , , ,

Say you are in a situation where you have a file with a huge number of records to be processed and the processing of one record does not need data from the processing of previous records (ie. a perfectly paralellizable situation), what can you do to speed up things? Well, here’s what I did when […]

Continue reading » 4 Comments

Partitioning help for multi booting Ubuntu Linux and Windows XP safely

Tags: , , , ,

A few days back, the Windows installation on my PC at home got corrupted. So I decided to reinstall Windows and multi-boot it with Ubuntu. I backed up whatever data I needed from the desktop hard drive onto my laptop and proceeded to have fun trying out all kinds of acrobatics with no fear of […]

Continue reading » 8 Comments