S3 Examples using boto3

This page provides some examples of using the S3 API.

Boto3

Boto3 is the official AWS SDK for Python, used to create, configure, and manage AWS services.

We will provide examples of defining a resource/client in boto3 for the Weka S3 service, managing credentials, pre-signed URLs, generating secure temporary tokens, and using those to run S3 API calls.

Installation

pip install boto3

Credentials

There are many ways to set credentials in boto3, as described on the boto3 credentials page. It's worth emphasizing the Assume Role Provider method, which uses the access/secret keys to automatically generate and use the temporary security token.

Resource

Resources represent an object-oriented interface to Amazon Web Services (AWS). They provide a higher-level abstraction than the raw, low-level calls made by service clients. To use resources, you invoke the resource() method of a Session and pass in a service name:

s3 = boto3.resource('s3',
                    endpoint_url='https://weka:9000',
                    aws_access_key_id='s3_key',
                    aws_secret_access_key='s3_secret')

Client

Clients provide a low-level interface to AWS whose methods map close to 1:1 with service APIs. All service operations are supported by clients (in our case, s3 and sts)

s3_client = boto3.client('sts',
                         endpoint_url='https://weka:9000',
                         aws_access_key_id='s3_key',
                         aws_secret_access_key='s3_secret',
                         region_name='us-east-1'))

Assume Role Example

Example code of using an access/secret key to obtain a temporary security token for the S3 service:

#!/usr/bin/env/python
import boto3
import logging
from botocore.exceptions import ClientError
from botocore.client import Config

config = Config(
   signature_version = 's3v4'
)

s3_client = boto3.client('sts',
        endpoint_url='https://weka:9000',
        aws_access_key_id='s3_key',
        aws_secret_access_key='s3_secret',
        config=config,
        region_name='us-east-1')

try:

  response = s3_client.assume_role(
      RoleArn='arn:x:ignored:by:weka-s3:',
      RoleSessionName='ignored-by-weka-s3',
      DurationSeconds=900
  )

except ClientError as e:
    logging.error(e)

print 'AccessKeyId:' + response['Credentials']['AccessKeyId']
print 'SecretAccessKey:' + response['Credentials']['SecretAccessKey']
print 'SessionToken:' + response['Credentials']['SessionToken']

Pre-Signed URL Example

Example of signing on a GET request for myobject within mybucket for anonymous access:

#!/usr/bin/env/python
import boto3
import logging
from botocore.exceptions import ClientError
from botocore.client import Config

config = Config(
   signature_version = 's3v4'
)

s3_client = boto3.client('s3',
                         endpoint_url='https://weka:9000',
                         aws_access_key_id='s3_key',
                         aws_secret_access_key='s3_secret',
                         config=config,
                         region_name='us-east-1')

try:
    response = s3_client.generate_presigned_url('get_object',
                                                Params={'Bucket': 'mybucket',
                                                        'Key': 'myobject'},
                                                ExpiresIn=3600)
except ClientError as e:
    logging.error(e)

# The response contains the pre-signed URL
print response

Use the response to access the object without providing any credentials:

$ curl "http://weka:9000/mybucket/myobject?AWSAccessKeyId=s3_key&Expires=1624801707&Signature=4QBcfEUsUdR7Jaffg6gLRVpNTY0%3D"
myobject content

Pre-Signed URL with Assume Role Example

Now, let's combine the above two examples by providing a pre-signed URL from a temporary security token:

#!/usr/bin/env/python
import boto3
import logging
from botocore.exceptions import ClientError
from botocore.client import Config

config = Config(
   signature_version = 's3v4'
)

s3_client = boto3.client('s3',
                         endpoint_url='https://weka:9000',
                         aws_access_key_id='access_key',
                         aws_secret_access_key='secret_key',
	                       aws_session_token='session_token',
	                       config=config,
                         region_name='us-east-1')
try:
    response = s3_client.generate_presigned_url('get_object',
                                                Params={'Bucket': 'mybucket',
                                                        'Key': 'myobject'},
                                                ExpiresIn=3600)
except ClientError as e:
    logging.error(e)

# The response contains the pre-signed URL
print response

Upload/Download Example

An example of using boto3 resource to upload and download an object:

#!/usr/bin/env/python
import boto3
import logging
from botocore.exceptions import ClientError
from botocore.client import Config

config = Config(
   signature_version = 's3v4'
)

s3 = boto3.resource('s3',
                    endpoint_url='https://weka:9000',
                    aws_access_key_id='s3_key',
                    aws_secret_access_key='s3_secret',
                    config=config)

try:
  # upload a file from local file system 'myfile' to bucket 'mybucket' with 'my_uploaded_object' as the object name.
  s3.Bucket('mybucket').upload_file('myfile','my_uploaded_object')

  # download the object 'piano.mp3' from the bucket 'songs' and save it to local FS as /tmp/classical.mp3
  s3.Bucket('mybucket').download_file('my_uploaded_object', 'my_downloaded_object')

except ClientError as e:
        logging.error(e)

print "Downloaded 'my_downloaded_object' as 'my_uploaded_object'. "

Create Bucket Example

An example for creating a bucket newbucket with a boto3 client:

#!/usr/bin/env/python
import boto3
import logging
from botocore.exceptions import ClientError
from botocore.client import Config

config = Config(
   signature_version = 's3v4'
)

s3_client = boto3.client('s3',
                         endpoint_url='https://weka:9000',
                         aws_access_key_id='s3_key',
                         aws_secret_access_key='s3_secret',
                         config=config)


try:
  s3_client.create_bucket(Bucket='newbucket')
except ClientError as e:
        logging.error(e)