12
$\begingroup$

Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL?

Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I just want to supply the download URL to S3 and wait for them to download it to their filesystem. It seems like an easy thing to do, but I just can't find the documentation on it.

| improve this question | |
$\endgroup$
  • $\begingroup$ Great question. $\endgroup$ – Pramit Jul 24 '19 at 1:36
10
$\begingroup$

Since you obviously posses an AWS account I'd recommend the following:

  • Create an EC2 instance (any size)
  • Use wget(or curl) to fetch the file(s) to that EC2 instance. For example: wget http://example.com/my_large_file.csv.
  • Install s3cmd
  • Use s3cmd to upload the file to S3. For example: s3cmd cp my_large_file.csv s3://my.bucket/my_large_file.csv

Since connections made between various AWS services leverage AWS's internal network, uploading from an EC2 instance to S3 is pretty fast. Much faster than uploading it from your own computer. This way allows you to avoid downloading the file to your computer and saving potentially significant time uploading it through the web interface.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you. I was contemplating something like this, but I wanted to ask around to see if there was an easier way. Thanks for pointing out all the steps too. Very helpful $\endgroup$ – Will Stedden Apr 29 '15 at 16:18
  • $\begingroup$ @Daniel Zohar What if the URL is dynamic? Using wget doesn't download the file but only the page with this link: cms.unov.org/UNCorpus/en/Download?file=UNv1.0.en-zh.tar.gz.00 $\endgroup$ – echan00 Oct 27 '18 at 1:38
3
$\begingroup$

  1. Launch an EC2 instance with enough storage

  2. ssh to the instance

  3. Obtain the curl command corresponding to the download from your local machine. You can use the developer options in Google chrome -> network tab -> copy -> copy as curl (this step is necessary for some websites requiring authentication such as kaggle)

  4. From the instance terminal, run the curl command (append -o output_file to the command). This will download and save the file

  5. Configure aws credentials to connect the instance to s3 (one way is to use the command aws config, provide AWS access key Id and secret),

  6. Use this command to upload the file to s3: 

    aws s3 cp path-to-file s3://bucket-name/
| improve this answer | |
$\endgroup$
2
$\begingroup$

Refer Aws documentation : http://aws.amazon.com/code there are libraries available for most of the programing languages. So you can create a bucket and configure in your code to fetch data from url and write to this bucket in s3

for eg in python :

from boto.s3.key import Key
k = Key(bucket)
k.key = 'foobar'
k.set_contents_from_string(url_data)

Ref : https://boto.readthedocs.org/en/latest/s3_tut.html

| improve this answer | |
$\endgroup$
  • $\begingroup$ I don't think this is quite right. I'm a little unclear about where url_data would be coming from. From the documentation, k.set_contents_from_string() seems to quite literally set the contents of file 'foobar' to whatever is contained in that string. I want the content at that url to be pushed directly to s3 without needing to be downloaded locally. $\endgroup$ – Will Stedden Apr 23 '15 at 15:37
2
$\begingroup$

I was facing the same task. However, all the provided answers have one flaw: They save the large file to EC2 disk. This has the drawback that you need to make sure that there is enough disk space on your instance and/or that the file system is encrypted (if you process sensitive data).

So, better avoid saving to disk. Instead, stream the curl stdout as stdin into the aws s3 cli. So,

  1. On your ec2-instance, install aws-cli:
apk --no-cache add curl procps shadow python py-pip
pip install awscli
  1. Down- and upload:
curl <any curl parameters as fit> | aws s3 cp - <your s3 bucket/folder> --expected-size <any max size in byte in case the file is larger than 50GB>

Note that the '-' after the cp command makes the aws client use stdin as input.

| improve this answer | |
$\endgroup$
1
$\begingroup$

You can mount your s3 bucket to ec2 instance and then cd to the /path/to/s3_mounted_on_a_folder, there you can simply use the command:

wget https://your.download.url/

to mount s3 to your ec2, use s3fs.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.