Which is the fastest way to copy 400G of files from an ec2 elastic block store volume to s3?

Another good option is peak/s5cmd:

For uploads, s5cmd is 32x faster than s3cmd and 12x faster than aws-cli. For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s), whereas s3cmd and aws-cli can only reach 85 MB/s and 375 MB/s respectively.

Tune AWS CLI S3 Configuration values as per http://docs.aws.amazon.com/cli/latest/topic/s3-config.html.

The below increased an S3 sync speed by at least 8x!

Example:

$ more ~/.aws/config
[default]
aws_access_key_id=foo
aws_secret_access_key=bar
s3 =
   max_concurrent_requests = 100
   max_queue_size = 30000

Try using s3-cli instead of s3cmd. I used it instead of s3cmd to upload files to my s3 bucket and it made my deployment faster almost by 17 minutes (from 21 to 4 minutes)!

Here's the link : https://github.com/andrewrk/node-s3-cli

Try s4cmd instead, it's really faster than s3cmd. Its address: https://github.com/bloomreach/s4cmd

There is also: s3funnel, which seems very old (2008) and some open bugs, but is still listed from Amazon itself: amzn-lnk

I wrote a optimized console application in C# (CopyFasterToS3) to do this. I used in EBS vol, i my case it had 5 folders with more than 2 millions files in a amount of 20Gb. The script executed in less than 30 minutes.

In this article i showed how to using a recursive function with parallel. You can transcripted it to another language.

Good luck!

There are several key factors that determine throughput from EC2 to S3:

• File size - smaller files require a larger number of requests and more overhead and transfer slower. The gain with filesize (when originating from EC2) is negligible for files larger than 256kB. (Whereas, transfering from a remote location, with higher latency, tends to continue showing appreciable improvements until between 1MiB and 2MiB).
• Number of parallel threads - a single upload thread usually has a fairly low throughout - often below 5MiB/s. Throughput increases with the number of concurrent threads, and tends to peak between 64 and 128 threads. It should be noted that larger instances are able to handle a greater number of concurrent threads.
• Instance size - As per the instance specifications, larger instances have more dedicated resources, including a larger (and less variable) allocation of network bandwidth (and I/O in general - including reading from ephemeral/EBS disks - which are network attached. Typical numbers values for each category are:
  • Very High: Theoretical: 10Gbps = 1250MB/s; Realistic: 8.8Gbps = 1100MB/s
  • High: Theoretical: 1Gbps = 125MB/s; Realistic: 750Mbps = 95MB/s
  • Moderate: Theoretical: 250Mbps; Realistic: 80Mbps = 10MB/s
  • Low: Theoretical: 100Mbps; Realistic: 10-15Mbps = 1-2MB/s

In cases of transferring large amounts of data, it may be economically practical to use a cluster compute instance, as the effective gain in throughput (>10x) is more than the difference in cost (2-3x).

While the above ideas are fairly logical (although, the per-thread cap may not be), it is quite easy to find benchmarks backing them up. One particularly detailed one can be found here.

Using between 64 and 128 parallel (simultaneous) uploads of 1MB objects should saturate the 1Gbps uplink that an m1.xlarge has and should even saturate the 10Gbps uplink of a cluster compute (cc1.4xlarge) instance.

While it is fairly easy to change instance size, the other two factors may be harder to manage.

• File size is usually fixed - we cannot join files together on EC2 and have them split apart on S3 (so, there isn't much we can do about small files). Large files however, we can split apart on the EC2 side and reassemble on the S3 side (using S3's multi-part upload). Typically, this is advantageous for files that are larger than 100MB.
• Parallel threads is a bit harder to cater to. The simplest approach comes down to writing a wrapper for some existing upload script that will run multiple copies of it at once. Better approaches use the API directly to accomplish something similar. Keeping in mind that the key is parallel requests, it is not difficult to locate several potential scripts, for example:
  • s3cmd-modification - a fork of an early version of s3cmd that added this functionality, but hasn't been updated in several years.
  • s3-parallel-put - reasonably recent python script that works well

So, after a lot of testing s3-parallel-put did the trick awesomely. Clearly the solution if you need to upload a lot of files to S3. Thanks to cyberx86 for the comments.