In most cases, we recommend using Valohai inputs for any files you need during your executions. That way, the information will get tracked and anyone with access to your project can rerun the execution, as long as the files still exist in your data store.
That being said, sometimes you might need to get access to an S3 bucket from inside the execution. When using boto3, you will either need to provide the credentials (recommended to use secret environment variables) or you can allow the access based on the machine instance profile.
When using the instance profile, note that the items in the bucket are encrypted with a custom AWS KMS key that the InstanceProfile doesn't have access to. This will result in the following error:
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetObject operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.Follow these instructions from AWS knowledge center to give the IAM Role permissions to the KMS Key used.
You can verify which KMS key is being used by:
- Opn the AWS Console's S3 page
- Open your bucket and navigate to one of the files you're trying to download
- Click the checkbox next to one of the folders/files and click "Edit server-side encryption" review which key is being used there.
- It looks something like this: Encryption key ARN arn:aws:kms:eu-west-1:ACCOUNT:key/111aa2bb-333c-4d44-5555-a111bb2c33dd
Comments
0 comments
Please sign in to leave a comment.