Amazon S3 is an affordable resource for storing your data; you pay for what you use. There are four cost components to consider:
1. Storage pricing (how much space you use)
2. Request and data retrieval pricing (number of requests you make)
3. Data transfer and transfer acceleration pricing (how often you transfer the data)
4. Data management features pricing (Amazon storage management features)
In this post, I will show you how you can upload data to Amazon S3 on the terminal.
1. First, login to the AWS Management Console at https://aws.amazon.com/, use “Find Services” to look for S3
2. Next, click on “Create bucket”; pick your region of choice and choose a bucket name that is DNS-compliant; click on Next and accept defaults for Configure options and Set permissions.
3. In the Review step, click on Create bucket
I created a new bucket called “davetangtest” for this post. Next we will create a new user for using S3 using Identity and Access Management (IAM); use the “Find Services” search bar to look for IAM.
1. Click on Groups under Access management in the Dashboard on the left; click on Create New Group and chose a Group Name
2. Search for the AmazonS3FullAccess policy, click on the tick mark, and then click Next Step and then Create Group
3. Next, click on Users under Access management in the Dashboard on the left.
4. Click Add user, choose a User name and click on Programmatic access, then Next
5. Click on the new group, AmazonS3FullAccess, we created in step 2, then Next
6. We won’t create any tags, so go to Next and Review.
7. After creating the new user, save the Access key ID and Secret access key
This is the only time that the secret access keys can be viewed or downloaded. You cannot recover them later. However, you can create new access keys at any time.
# create new Conda environment with awscli conda create -c conda-forge -n davetangs3 awscli # use this new environment conda activate davetangs3
We will need to configure AWS CLI. Before configuring, find out the code for your region of choice. For example, I want the Asia Pacific (Sydney) region and the code is ap-southeast-2. You will need the Access key ID and Secret access key for the user we created earlier. For “Default output format [None]:”, leave it blank.
# enter details at the prompt aws configure
If everything was set up properly, we can use AWS CLI to upload files to our bucket. Replace “davetangtest” with your bucket name.
aws s3 cp README.md s3://davetangtest/ upload: ./README.md to s3://davetangtest/README.md
To upload a directory use –recursive and make sure the directory name is in the s3 path.
# create new directory mkdir test # move file into new directory mv README.md test # this will fail aws s3 cp test s3://davetangtest/ upload failed: test/ to s3://davetangtest/ Parameter validation failed: Invalid length for parameter Key, value: 0, valid range: 1-inf # use --recursive aws s3 cp test s3://davetangtest/test --recursive # list objects in our bucket aws s3 ls davetangtest PRE test/ 2019-12-23 11:27:52 18 README.md # remove directory aws s3 rm s3://davetangtest/test/ --recursive delete: s3://davetangtest/test/README.md # remove file aws s3 rm s3://davetangtest/README.md delete: s3://davetangtest/README.md
Finally, to download an s3 object use the sync subcommand.
aws s3 sync s3://davetangtest/test local_test
This work is licensed under a Creative Commons
Attribution 4.0 International License.