Downloading a large dataset on the web directly into AWS S3

Question

Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL?

Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I just want to supply the download URL to S3 and wait for them to download it to their filesystem. It seems like an easy thing to do, but I just can't find the documentation on it.

Answer 1

Since you obviously posses an AWS account I'd recommend the following:

• Create an EC2 instance (any size)
• Use wget(or curl) to fetch the file(s) to that EC2 instance. For example: wget http://example.com/my_large_file.csv.
• Install s3cmd
• Use s3cmd to upload the file to S3. For example: s3cmd cp my_large_file.csv s3://my.bucket/my_large_file.csv

Since connections made between various AWS services leverage AWS's internal network, uploading from an EC2 instance to S3 is pretty fast. Much faster than uploading it from your own computer. This way allows you to avoid downloading the file to your computer and saving potentially significant time uploading it through the web interface.

Answer 2

Refer Aws documentation : http://aws.amazon.com/code there are libraries available for most of the programing languages. So you can create a bucket and configure in your code to fetch data from url and write to this bucket in s3

for eg in python :

from boto.s3.key import Key
k = Key(bucket)
k.key = 'foobar'
k.set_contents_from_string(url_data)

Ref : https://boto.readthedocs.org/en/latest/s3_tut.html

Answer 3

1. Launch an EC2 instance with enough storage

2. ssh to the instance

3. Obtain the curl command corresponding to the download from your local machine. You can use the developer options in Google chrome -> network tab -> copy -> copy as curl (this step is necessary for some websites requiring authentication such as kaggle)

4. From the instance terminal, run the curl command (append -o output_file to the command). This will download and save the file

5. Configure aws credentials to connect the instance to s3 (one way is to use the command aws config, provide AWS access key Id and secret),

6. Use this command to upload the file to s3:

   aws s3 cp path-to-file s3://bucket-name/
   

Answer 4

You can mount your s3 bucket to ec2 instance and then cd to the /path/to/s3_mounted_on_a_folder, there you can simply use the command:

wget https://your.download.url/

to mount s3 to your ec2, use s3fs.

Answer 5

I was facing the same task. However, all the provided answers have one flaw: They save the large file to EC2 disk. This has the drawback that you need to make sure that there is enough disk space on your instance and/or that the file system is encrypted (if you process sensitive data).

So, better avoid saving to disk. Instead, stream the curl stdout as stdin into the aws s3 cli. So,

1. On your ec2-instance, install aws-cli:

apk --no-cache add curl procps shadow python py-pip
pip install awscli

2. Down- and upload:

curl <any curl parameters as fit> | aws s3 cp - <your s3 bucket/folder> --expected-size <any max size in byte in case the file is larger than 50GB>

Note that the '-' after the cp command makes the aws client use stdin as input.