How to create a file of arbitrary size with shell script commands

Tags: , , , ,

A few days ago I was working on writing Java code to transfer files via SFTP and FTPS. As part of the test cases, I wanted to try and transfer files of small medium and large sizes like 0 byte, 1 byte, 100 bytes, 1000 bytes, 1MB, 10MB. How would one go about creating files […]

Continue reading » 3 Comments

Check for corruption of files using md5 checksum

Tags: , , , , , ,

You just transferred a large multi-gigabit file from your build server to your QA server. You want to check if the files are exactly the same. Data corruption can happen for many reasons! What do you do? Remember, just because two files are the same size, it doesn’t mean that they are the same. It […]

Continue reading » 2 Comments

How to split a file, process the pieces in multiple threads and combine results using a shell script

Tags: , , , , , ,

Say you are in a situation where you have a file with a huge number of records to be processed and the processing of one record does not need data from the processing of previous records (ie. a perfectly paralellizable situation), what can you do to speed up things? Well, here’s what I did when […]

Continue reading » 4 Comments