Uploading to Amazon S3 using AWS CLI

Amazon S3 is an affordable resource for storing your data; you pay for what you use. There are four cost components to consider:

1. Storage pricing (how much space you use)
2. Request and data retrieval pricing (number of requests you make)
3. Data transfer and transfer acceleration pricing (how often you transfer the data)
4. Data management features pricing (Amazon storage management features)

In this post, I will show you how you can upload data to Amazon S3 on the terminal.

1. First, login to the AWS Management Console at https://aws.amazon.com/, use "Find Services" to look for S3
2. Next, click on "Create bucket"; pick your region of choice and choose a bucket name that is DNS-compliant; click on Next and accept defaults for Configure options and Set permissions.
3. In the Review step, click on Create bucket

I created a new bucket called "davetangtest" for this post. Next we will create a new user for using S3 using Identity and Access Management (IAM); use the "Find Services" search bar to look for IAM.

1. Click on Groups under Access management in the Dashboard on the left; click on Create New Group and chose a Group Name
2. Search for the AmazonS3FullAccess policy, click on the tick mark, and then click Next Step and then Create Group
3. Next, click on Users under Access management in the Dashboard on the left.
4. Click Add user, choose a User name and click on Programmatic access, then Next
5. Click on the new group, AmazonS3FullAccess, we created in step 2, then Next
6. We won't create any tags, so go to Next and Review.
7. After creating the new user, save the Access key ID and Secret access key

This is the only time that the secret access keys can be viewed or downloaded. You cannot recover them later. However, you can create new access keys at any time.

We will install AWS Command Line Interface (AWS CLI) using Conda in a new environment. If you are unfamiliar with Conda, I prepared this tutorial.

# create new Conda environment with awscli
conda create -c conda-forge -n davetangs3 awscli

# use this new environment
conda activate davetangs3

We will need to configure AWS CLI. Before configuring, find out the code for your region of choice. For example, I want the Asia Pacific (Sydney) region and the code is ap-southeast-2. You will need the Access key ID and Secret access key for the user we created earlier. For "Default output format [None]:", leave it blank.

# enter details at the prompt
aws configure

If everything was set up properly, we can use AWS CLI to upload files to our bucket. Replace "davetangtest" with your bucket name.

aws s3 cp README.md s3://davetangtest/
upload: ./README.md to s3://davetangtest/README.md

To upload a directory use --recursive and make sure the directory name is in the s3 path.

# create new directory
mkdir test

# move file into new directory
mv README.md test

# this will fail
aws s3 cp test s3://davetangtest/
upload failed: test/ to s3://davetangtest/ Parameter validation failed:
Invalid length for parameter Key, value: 0, valid range: 1-inf

# use --recursive
aws s3 cp test s3://davetangtest/test --recursive

# list objects in our bucket
aws s3 ls davetangtest
                           PRE test/
2019-12-23 11:27:52         18 README.md

# remove directory
aws s3 rm s3://davetangtest/test/ --recursive
delete: s3://davetangtest/test/README.md

# remove file
aws s3 rm s3://davetangtest/README.md
delete: s3://davetangtest/README.md

Finally, to download an s3 object use the sync subcommand.

aws s3 sync s3://davetangtest/test local_test