Copy large files between servers

If you have to copy large files between systems and security is not an issue you could use netcat. Netcat is a tool that has been developed in 1996 but it is still one of the best tools for this type of job. It is the Swiss army knife for stuff that can be done in using a dedicated network connection. This article will use it to copy files over the network in an easy and fast way.
Let’s say that you have a file huge_file.db that you want to copy from server1 to server2.

  1. open a server socket on server2 (file will go to the /temp directory)
    [root@server2 ~]# nc -l -p 11221 > /test/huge_file.db
  2. copy the file to the remote socket
    [root@server1 ~]# dd if=/root/Downloads/huge_file.db | nc server2 11221
    3900979+1 records in
    3900979+1 records out
    1997301556 bytes (2.0 GB) copied, 169.821 s, 11.8 MB/s

You will not see any output until the process has finished. To see the progress you can use pv. PV is a tool to look into a pipe and see the data transfer. It is not installed by default but is included in most of the repositories.
[root@server1 ~]# dd if=/root/Downloads/huge_file.db | pv | nc server2 11221
3900979+1 records inMB/s] [                  ]
3900979+1 records out
1997301556 bytes (2.0 GB) copied, 169.903 s, 11.8 MB/s
1.86GB 0:02:49 [11.2MB/s] [                  ]

About Juergen Caris

I am 54yo, MSc(Dist) and BSc in Computer Science, German and working as a Senior Server Engineer for the NHS Lothian. I am responsible for the patient management system, called TrakCare. I am a UNIX/Linux guy, working in this sector for more than 20 years now. I am also interested in robotics, microprocessors, system monitoring, Home automation and programming.
This entry was posted in Bash, Beginner. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>