AWS Credentials

If a workflow needs to access S3, it will be important to establish AWS credentials before first access to the object storage. If you are working in a workflow using a dask cluster, you will also need to establish AWS credentials before the cluster starts up.

The following code block will handle general configuration for establishing AWS credentials for the notebook and any spawned cluster workers.

You will need to make sure your AWS_PROFILE or AWS_S3_ENDPOINT environmental variables are established before this is run. If they are not, the defaults will be set to the access the OSN storage pod (with a profile named osn-renci and an endpoint of https://renc.osn.xsede.org). These default settings will only work if you have this AWS profile cofigured.

import os
import logging
import configparser

awsconfig = configparser.ConfigParser()
awsconfig.read(
    os.path.expanduser('~/.aws/credentials') # default location... if yours is elsewhere, change this.
)
## NOTE:  The default will be for the OSN / RENCI profile and endpoint. Override this
## by setting environment variables before executing this cell/notebook.
_profile_nm  = os.environ.get('AWS_PROFILE', 'osn-renci')
_endpoint = os.environ.get('AWS_S3_ENDPOINT', 'https://renc.osn.xsede.org')
# Set environment vars based on parsed awsconfig
try:
    os.environ['AWS_ACCESS_KEY_ID']     = awsconfig[_profile_nm]['aws_access_key_id']
    os.environ['AWS_SECRET_ACCESS_KEY'] = awsconfig[_profile_nm]['aws_secret_access_key']
    os.environ['AWS_S3_ENDPOINT']       = _endpoint
    os.environ['AWS_PROFILE'] = _profile_nm
    os.environ['AWS_DEFAULT_PROFILE'] = _profile_nm
    os.environ['AWS_S3_REGION'] = _profile_nm
except KeyError:
    logging.error("Problem parsing the AWS credentials file. ")

ERROR:root:Problem parsing the AWS credentials file.