S3 Examples using boto3
This page provides some examples of using the S3 API.

Boto3

Boto3 is the official AWS SDK for Python, used to create, configure, and manage AWS services.
We will provide examples of defining a resource/client in boto3 for the Weka S3 service, managing credentials, pre-signed URLs, generating secure temporary tokens, and using those to run S3 API calls.

Installation

pip install boto3

Credentials

There are many ways to set credentials in boto3, as described on the boto3 credentials page. It's worth emphasizing the Assume Role Provider method, which uses the access/secret keys to automatically generate and use the temporary security token.

Resource

Resources represent an object-oriented interface to Amazon Web Services (AWS). They provide a higher-level abstraction than the raw, low-level calls made by service clients. To use resources, you invoke the resource() method of a Session and pass in a service name:
1
s3 = boto3.resource('s3',
2
endpoint_url='https://weka:9000',
3
aws_access_key_id='s3_key',
4
aws_secret_access_key='s3_secret')
5
Copied!

Client

Clients provide a low-level interface to AWS whose methods map close to 1:1 with service APIs. All service operations are supported by clients (in our case, s3 and sts)
1
s3_client = boto3.client('sts',
2
endpoint_url='https://weka:9000',
3
aws_access_key_id='s3_key',
4
aws_secret_access_key='s3_secret',
5
region_name='us-east-1'))
Copied!

Assume Role Example

Example code of using an access/secret key to obtain a temporary security token for the S3 service:
1
#!/usr/bin/env/python
2
import boto3
3
import logging
4
from botocore.exceptions import ClientError
5
from botocore.client import Config
6
7
config = Config(
8
signature_version = 's3v4'
9
)
10
11
s3_client = boto3.client('sts',
12
endpoint_url='https://weka:9000',
13
aws_access_key_id='s3_key',
14
aws_secret_access_key='s3_secret',
15
config=config,
16
region_name='us-east-1')
17
18
try:
19
20
response = s3_client.assume_role(
21
RoleArn='arn:x:ignored:by:weka-s3:',
22
RoleSessionName='ignored-by-weka-s3',
23
DurationSeconds=900
24
)
25
26
except ClientError as e:
27
logging.error(e)
28
29
print 'AccessKeyId:' + response['Credentials']['AccessKeyId']
30
print 'SecretAccessKey:' + response['Credentials']['SecretAccessKey']
31
print 'SessionToken:' + response['Credentials']['SessionToken']
Copied!

Pre-Signed URL Example

Example of signing on a GET request for myobject within mybucket for anonymous access:
1
#!/usr/bin/env/python
2
import boto3
3
import logging
4
from botocore.exceptions import ClientError
5
from botocore.client import Config
6
7
config = Config(
8
signature_version = 's3v4'
9
)
10
11
s3_client = boto3.client('s3',
12
endpoint_url='https://weka:9000',
13
aws_access_key_id='s3_key',
14
aws_secret_access_key='s3_secret',
15
config=config,
16
region_name='us-east-1')
17
18
try:
19
response = s3_client.generate_presigned_url('get_object',
20
Params={'Bucket': 'mybucket',
21
'Key': 'myobject'},
22
ExpiresIn=3600)
23
except ClientError as e:
24
logging.error(e)
25
26
# The response contains the pre-signed URL
27
print response
Copied!
Use the response to access the object without providing any credentials:
1
$ curl "http://weka:9000/mybucket/myobject?AWSAccessKeyId=s3_key&Expires=1624801707&Signature=4QBcfEUsUdR7Jaffg6gLRVpNTY0%3D"
2
myobject content
Copied!

Pre-Signed URL with Assume Role Example

Now, let's combine the above two examples by providing a pre-signed URL from a temporary security token:
1
#!/usr/bin/env/python
2
import boto3
3
import logging
4
from botocore.exceptions import ClientError
5
from botocore.client import Config
6
7
config = Config(
8
signature_version = 's3v4'
9
)
10
11
s3_client = boto3.client('s3',
12
endpoint_url='https://weka:9000',
13
aws_access_key_id='access_key',
14
aws_secret_access_key='secret_key',
15
aws_session_token='session_token',
16
config=config,
17
region_name='us-east-1')
18
try:
19
response = s3_client.generate_presigned_url('get_object',
20
Params={'Bucket': 'mybucket',
21
'Key': 'myobject'},
22
ExpiresIn=3600)
23
except ClientError as e:
24
logging.error(e)
25
26
# The response contains the pre-signed URL
27
print response
28
Copied!

Upload/Download Example

An example of using boto3 resource to upload and download an object:
1
#!/usr/bin/env/python
2
import boto3
3
import logging
4
from botocore.exceptions import ClientError
5
from botocore.client import Config
6
7
config = Config(
8
signature_version = 's3v4'
9
)
10
11
s3 = boto3.resource('s3',
12
endpoint_url='https://weka:9000',
13
aws_access_key_id='s3_key',
14
aws_secret_access_key='s3_secret',
15
config=config)
16
17
try:
18
# upload a file from local file system 'myfile' to bucket 'mybucket' with 'my_uploaded_object' as the object name.
19
s3.Bucket('mybucket').upload_file('myfile','my_uploaded_object')
20
21
# download the object 'piano.mp3' from the bucket 'songs' and save it to local FS as /tmp/classical.mp3
22
s3.Bucket('mybucket').download_file('my_uploaded_object', 'my_downloaded_object')
23
24
except ClientError as e:
25
logging.error(e)
26
27
print "Downloaded 'my_downloaded_object' as 'my_uploaded_object'. "
28
Copied!

Create Bucket Example

An example for creating a bucket newbucket with a boto3 client:
1
#!/usr/bin/env/python
2
import boto3
3
import logging
4
from botocore.exceptions import ClientError
5
from botocore.client import Config
6
7
config = Config(
8
signature_version = 's3v4'
9
)
10
11
s3_client = boto3.client('s3',
12
endpoint_url='https://weka:9000',
13
aws_access_key_id='s3_key',
14
aws_secret_access_key='s3_secret',
15
config=config)
16
17
18
try:
19
s3_client.create_bucket(Bucket='newbucket')
20
except ClientError as e:
21
logging.error(e)
22
Copied!
Last modified 1mo ago