Aurora MySQL – Export data to S3

Posted Leave a commentPosted in AWS, AWS Aurora

Using SELECT INTO OUTFILE S3 you can query data from an Aurora MySQL DB cluster and save it directly into text files stored in S3 bucket. 1. Create an IAM policy for S3. { “Version”: “2012-10-17”, “Statement”: [ { “Sid”: “VisualEditor0”, “Effect”: “Allow”, “Action”: [ “s3:DeleteObject”, “s3:GetBucketLocation”, “s3:GetObject”, “s3:ListBucket”, “s3:ListBucketMultipartUploads”, “s3:PutObject” ], “Resource”: [ “arn:aws:s3:::bucket-name”, […]

S3 – fatal error: An error occurred (404) when calling the HeadObject operation

Posted Leave a commentPosted in AWS, S3

Make sure to use –recursive parameter. [desktop: test]:${PWD}> aws s3 cp s3://demo-beta/dw/user/dt=2019-07-30/ /tmp/dw/ fatal error: An error occurred (404) when calling the HeadObject operation: Key “dw/user/dt=2019-07-30/” does not exist [desktop: test]:${PWD}> aws s3 cp s3://demo-beta/dw/user/dt=2019-07-30/ /tmp/dw/ –recursive download: s3://demo-beta/dw/user/dt=2019-07-30/part-00002-fd866c-238-489-a44-739f1d04-c000.snappy.parquet to ../../../tmp/dw/part-00002-fd866c-238-489-a44-739f1d04-c000.snappy.parquet From Documentation – –recursive (boolean) Command is performed on all files or objects under […]

S3 Bucket – IllegalLocationConstraintException

Posted Leave a commentPosted in AWS, S3

A quick one !! Creating bucket failed with “IllegalLocationConstraintException” — [[email protected] ~]$ aws s3api create-bucket –bucket labtestrand –region us-west-2 An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to. Solution — [[email protected] ~]$ [[email protected] ~]$ aws s3api create-bucket –bucket labtestrand […]