First putblob should be signed with specific region rather than with default region during createcontainer API

When container in non default region exists, only first time put operation will be applied to find out whether bucket exists and have proper access or not. Aws sigv4 signature will be created based on the default region only as create bucket method was not honoring the region specified.

So, Put request was first signed with default region and if the user does not have access to default(us-east-1) region,. Then it will be re-directed to sa-east-1 region, but the request is signed with us-east-1, the request is rejected and throws "The authorization header is malformed".

Flow in case of user has access to default region:

17:22:24.460 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking CreateBucket
17:22:24.460 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking CreateBucket
17:22:24.461 [bscThread-02] DEBUG jclouds.signature - << PUT
/

content-length:105
content-type:text/xml
host:test3.s3.amazonaws.com
x-amz-content-sha256:f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
x-amz-date:20190818T115218Z

content-length;content-type;host;x-amz-content-sha256;x-amz-date
f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
17:22:24.461 [bscThread-02] DEBUG jclouds.signature - << AWS4-HMAC-SHA256
20190818T115218Z
20190818/us-east-1/s3/aws4_request
089a5248f5eff6e8b6378154acdf07bff7d208029c98c67af44c99b4a8f2df39
17:22:24.463 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Sending request -1533211628: PUT https://test3.s3.amazonaws.com/ HTTP/1.1
17:22:24.463 [bscThread-02] DEBUG jclouds.wire - >> "<CreateBucketConfiguration><LocationConstraint>sa-east-1</LocationConstraint></CreateBucketConfiguration>"
17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> PUT https://test3.s3.amazonaws.com/ HTTP/1.1
17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml
17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105
17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Host: test3.s3.amazonaws.com
17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> x-amz-content-sha256: f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> X-Amz-Date: 20190818T115218Z
17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIGKQ7V52FQQJFYJQ/20190818/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=637d42fbf6684430ab0f08fd82cbae69f3261859e0031ad40054bccb829473da
17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml
17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105
17:22:25.671 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Receiving response -1533211628: HTTP/1.1 409 Conflict
17:22:25.671 [bscThread-02] DEBUG jclouds.headers - << HTTP/1.1 409 Conflict
17:22:25.672 [bscThread-02] DEBUG jclouds.headers - << Transfer-Encoding: chunked
17:22:25.672 [bscThread-02] DEBUG jclouds.headers - << Server: AmazonS3
17:22:25.672 [bscThread-02] DEBUG jclouds.headers - << x-amz-request-id: 09E5163C51F25F34
17:22:25.673 [bscThread-02] DEBUG jclouds.headers - << x-amz-id-2: WuN84GYMs47Nn6+48XYDpLZNvp0NPszokKyhxlzZk+ub8RhjbLpkfEI8E2tKWVCFKtJiXrhdpkc=
17:22:25.673 [bscThread-02] DEBUG jclouds.headers - << Date: Sun, 18 Aug 2019 11:52:11 GMT
17:22:25.673 [bscThread-02] DEBUG jclouds.headers - << x-amz-bucket-region: sa-east-1
17:22:25.673 [bscThread-02] DEBUG jclouds.headers - << Content-Type: application/xml
17:22:25.673 [bscThread-02] DEBUG jclouds.wire - << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
17:22:25.673 [bscThread-02] DEBUG jclouds.wire - << "<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to create the named bucket succeeded and you already own it.</Message><BucketName>test3</BucketName><RequestId>09E5163C51F25F34</RequestId><HostId>WuN84GYMs47Nn6+48XYDpLZNvp0NPszokKyhxlzZk+ub8RhjbLpkfEI8E2tKWVCFKtJiXrhdpkc=</HostId></Error>"
17:22:25.680 [bscThread-03] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking BucketExists
17:22:25.681 [bscThread-03] DEBUG jclouds.signature - << HEAD
/

Flow in case of user has access to default region

host:test3.s3-sa-east-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190818T115218Z

 

Flow in case of user does not have access to default region.

17:28:41.464 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking CreateBucket
17:28:41.464 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking CreateBucket
17:28:41.466 [bscThread-02] DEBUG jclouds.signature - << PUT
/

content-length:105
content-type:text/xml
host:test-poc-spectrum-scale.s3.amazonaws.com
x-amz-content-sha256:f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
x-amz-date:20190818T115835Z

content-length;content-type;host;x-amz-content-sha256;x-amz-date
f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
17:28:41.466 [bscThread-02] DEBUG jclouds.signature - << AWS4-HMAC-SHA256
20190818T115835Z
20190818/us-east-1/s3/aws4_request
d8c68a44175c3b0b49182ca5658bff291b3f382d7d02b3fe25d6970912df1697
17:28:41.467 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Sending request -541426552: PUT https://test-poc-spectrum-scale.s3.amazonaws.com/ HTTP/1.1
17:28:41.467 [bscThread-02] DEBUG jclouds.wire - >> "<CreateBucketConfiguration><LocationConstraint>sa-east-1</LocationConstraint></CreateBucketConfiguration>"
17:28:41.467 [bscThread-02] DEBUG jclouds.headers - >> PUT https://test-poc-spectrum-scale.s3.amazonaws.com/ HTTP/1.1
17:28:41.467 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml
17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105
17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Host: test-poc-spectrum-scale.s3.amazonaws.com
17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> x-amz-content-sha256: f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> X-Amz-Date: 20190818T115835Z
17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Authorization: AWS4-HMAC-SHA256 Credential=AKIA25YU33MFMAZO3B7F/20190818/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=8cf0d9f58bd0ba5fc607f5018e701cdf14587741169ba7deaaa7a9cad0548cb6
17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml
17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105
17:28:43.016 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Receiving response -541426552: HTTP/1.1 400 Bad Request
17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << HTTP/1.1 400 Bad Request
17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Transfer-Encoding: chunked
17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Server: AmazonS3
17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Connection: close
17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << x-amz-request-id: 43F137234826AA08
17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << x-amz-id-2: nzJhRvv6j87VgSKV/mlGNhAqdietQCH4V5ArVUdIX2j1SnTChg2t1/GH20Zs3iz6Bc0MPzzAzwc=
17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Date: Sun, 18 Aug 2019 11:58:29 GMT
17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Content-Type: application/xml
17:28:43.019 [bscThread-02] DEBUG jclouds.wire - << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
17:28:43.019 [bscThread-02] DEBUG jclouds.wire - << "<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'sa-east-1'</Message><Region>sa-east-1</Region><RequestId>43F137234826AA08</RequestId><HostId>nzJhRvv6j87VgSKV/mlGNhAqdietQCH4V5ArVUdIX2j1SnTChg2t1/GH20Zs3iz6Bc0MPzzAzwc=</HostId></Error>"
EXC:org.jclouds.aws.AWSResponseException: request PUT https://test-poc-spectrum-scale.s3.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='43F137234826AA08', requestToken='nzJhRvv6j87VgSKV/mlGNhAqdietQCH4V5ArVUdIX2j1SnTChg2t1/GH20Zs3iz6Bc0MPzzAzwc=', code='AuthorizationHeaderMalformed', message='The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'sa-east-1'', context='

{Region=sa-east-1, HostId=nzJhRvv6j87VgSKV/mlGNhAqdietQCH4V5ArVUdIX2j1SnTChg2t1/GH20Zs3iz6Bc0MPzzAzwc=}

'}

I have a fix so that the first put operation honors the custom region specified and request will be signed with specific region rather than with default region. Tested with default region as well and worked well.

After my fix, flow with stand-alone program is as below.

17:36:09.043 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking CreateBucket
17:36:09.043 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking CreateBucket
17:36:09.044 [bscThread-02] DEBUG jclouds.signature - << PUT
/

content-length:105
content-type:text/xml
host:test-poc-spectrum-scale.s3-sa-east-1.amazonaws.com
x-amz-content-sha256:f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
x-amz-date:20190818T120602Z

content-length;content-type;host;x-amz-content-sha256;x-amz-date
f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
17:36:09.044 [bscThread-02] DEBUG jclouds.signature - << AWS4-HMAC-SHA256
20190818T120602Z
20190818/sa-east-1/s3/aws4_request
2996a3b4d85ab8c68e1378ab68db2f7d80b244969d1792391a49b9b7390bc920
17:36:09.046 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Sending request -719121422: PUT https://test-poc-spectrum-scale.s3-sa-east-1.amazonaws.com/ HTTP/1.1
17:36:09.046 [bscThread-02] DEBUG jclouds.wire - >> "<CreateBucketConfiguration><LocationConstraint>sa-east-1</LocationConstraint></CreateBucketConfiguration>"
17:36:09.046 [bscThread-02] DEBUG jclouds.headers - >> PUT https://test-poc-spectrum-scale.s3-sa-east-1.amazonaws.com/ HTTP/1.1
17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml
17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105
17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Host: test-poc-spectrum-scale.s3-sa-east-1.amazonaws.com
17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> x-amz-content-sha256: f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> X-Amz-Date: 20190818T120602Z
17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Authorization: AWS4-HMAC-SHA256 Credential=AKIA25YU33MFMAZO3B7F/20190818/sa-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=982a3a62e85e03c68f4ac1e0da5cfa753cb81cf750a6fbff157b34681ed54774
17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml
17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105
17:36:11.076 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Receiving response -719121422: HTTP/1.1 409 Conflict
17:36:11.076 [bscThread-02] DEBUG jclouds.headers - << HTTP/1.1 409 Conflict
17:36:11.076 [bscThread-02] DEBUG jclouds.headers - << Transfer-Encoding: chunked
17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << Server: AmazonS3
17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << x-amz-request-id: 5FB8D9C1B41E48EB
17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << x-amz-id-2: 54oqA5Lc+yl6Y9Ppz6kMd6hZs/iGFGfNFPqrOPX90Q731UH5KkYEYc7RS/4W9btahu0dxQKd3iI=
17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << Date: Sun, 18 Aug 2019 12:05:57 GMT
17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << x-amz-bucket-region: sa-east-1
17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << Content-Type: application/xml
17:36:11.077 [bscThread-02] DEBUG jclouds.wire - << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
17:36:11.077 [bscThread-02] DEBUG jclouds.wire - << "<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to create the named bucket succeeded and you already own it.</Message><BucketName>test-poc-spectrum-scale</BucketName><RequestId>5FB8D9C1B41E48EB</RequestId><HostId>54oqA5Lc+yl6Y9Ppz6kMd6hZs/iGFGfNFPqrOPX90Q731UH5KkYEYc7RS/4W9btahu0dxQKd3iI=</HostId></Error>"
17:36:11.083 [bscThread-03] DEBUG o.j.rest.internal.InvokeHttpMethod -