Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16988. Remove source code from branch-2. (aajisaka via jhung)

This closes #1959

  1. … 10846 more files in changeset.
HADOOP-16930. Add hadoop-aws documentation for ProfileCredentialsProvider

Contributed by Nicholas Chammas.

HADOOP-16711.

This adds a new option fs.s3a.bucket.probe, range (0-2) to

control which probe for a bucket existence to perform on startup.

0: no checks

1: v1 check (as has been performend until now)

2: v2 bucket check, which also incudes a permission check. Default.

When set to 0, bucket existence checks won't be done

during initialization thus making it faster.

When the bucket is not available in S3,

or if fs.s3a.endpoint points to the wrong instance of a private S3 store

consecutive calls like listing, read, write etc. will fail with

an UnknownStoreException.

Contributed by:

* Mukund Thakur (main patch and tests)

* Rajesh Balamohan (v0 list and performance tests)

* lqjacklee (HADOOP-15990/v2 list)

* Steve Loughran (UnknownStoreException support)

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java

new file: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/UnknownStoreException.java

new file: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ErrorTranslation.java

modified: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

modified: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md

modified: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java

new file: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3ClientFactory.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AExceptionTranslation.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardToolTestBase.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java

modified: hadoop-tools/hadoop-aws/src/test/resources/core-site.xml

Change-Id: Ic174f803e655af172d81c1274ed92b51bdceb384

  1. … 15 more files in changeset.
HADOOP-16792: Make S3 client request timeout configurable.

Contributed by Mustafa Iman.

This adds a new configuration option fs.s3a.connection.request.timeout

to declare the time out on HTTP requests to the AWS service;

0 means no timeout.

Measured in seconds; the usual time suffixes are all supported

Important: this is the maximum duration of any AWS service call,

including upload and copy operations. If non-zero, it must be larger

than the time to upload multi-megabyte blocks to S3 from the client,

and to rename many-GB files. Use with care.

Change-Id: I407745341068b702bf8f401fb96450a9f987c51c

  1. … 5 more files in changeset.
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu

This closes #1733

  1. … 1 more file in changeset.
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu

This closes #1733

  1. … 1 more file in changeset.
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu

This closes #1733

  1. … 1 more file in changeset.
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu

This closes #1733

  1. … 1 more file in changeset.
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu

This closes #1733

  1. … 1 more file in changeset.
HADOOP-16599. Allow a SignerInitializer to be specified along with a Custom Signer

  1. … 9 more files in changeset.
HADOOP-16490. Avoid/handle cached 404s during S3A file creation.

Contributed by Steve Loughran.

This patch avoids issuing any HEAD path request when creating a file with overwrite=true,

so 404s will not end up in the S3 load balancers unless someone calls getFileStatus/exists/isFile

in their own code.

The Hadoop FsShell CommandWithDestination class is modified to not register uncreated files

for deleteOnExit(), because that calls exists() and so can place the 404 in the cache, even

after S3A is patched to not do it itself.

Because S3Guard knows when a file should be present, it adds a special FileNotFound retry policy

independently configurable from other retry policies; it is also exponential, but with

different parameters. This is because every HEAD request will refresh any 404 cached in

the S3 Load Balancers. It's not enough to retry: we have to have a suitable gap between

attempts to (hopefully) ensure any cached entry wil be gone.

The options and values are:

fs.s3a.s3guard.consistency.retry.interval: 2s

fs.s3a.s3guard.consistency.retry.limit: 7

The S3A copy() method used during rename() raises a RemoteFileChangedException which is not caught

so not downgraded to false. Thus: when a rename is unrecoverable, this fact is propagated.

Copy operations without S3Guard lack the confidence that the file exists, so don't retry the same way:

it will fail fast with a different error message. However, because create(path, overwrite=false) no

longer does HEAD path, we can at least be confident that S3A itself is not creating those cached

404 markers.

Change-Id: Ia7807faad8b9a8546836cb19f816cccf17cca26d

  1. … 24 more files in changeset.
HADOOP-16470. Make last AWS credential provider in default auth chain EC2ContainerCredentialsProviderWrapper.

Contributed by Steve Loughran.

Contains HADOOP-16471. Restore (documented) fs.s3a.SharedInstanceProfileCredentialsProvider.

Change-Id: I06b99b57459cac80bf743c5c54f04e59bb54c2f8

  1. … 3 more files in changeset.
HADOOP-16499. S3A retry policy to be exponential (#1246). Contributed by Steve Loughran.

  1. … 11 more files in changeset.
HADOOP-13868. [s3a] New default for S3A multi-part configuration (#1125)

  1. … 2 more files in changeset.
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries. Contributed by Steve Loughran.

(cherry picked from commit 2ac5aab8d725f761a9f9723471a4426f6b5d78c4)

  1. … 2 more files in changeset.
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.

Author: Ben Roling <ben.roling@gmail.com>

Initial patch from Brahma Reddy Battula.

  1. … 19 more files in changeset.
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.

Author: Ben Roling <ben.roling@gmail.com>

Initial patch from Brahma Reddy Battula.

  1. … 19 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran and Daryn Sharp.

  1. … 101 more files in changeset.
Revert "HADOOP-14556. S3A to support Delegation Tokens."

This reverts commit d7152332b32a575c3a92e3f4c44b95e58462528d.

  1. … 104 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran.

  1. … 104 more files in changeset.
HADOOP-14833. Remove s3a user:secret authentication. Contributed by Steve Loughran

  1. … 14 more files in changeset.
HADOOP-15583. Stabilize S3A Assumed Role support. Contributed by Steve Loughran.

  1. … 29 more files in changeset.
Revert "Revert "Merge branch 'trunk' into HDFS-7240"" After testing it was confirmed that these changes work as expected.

This reverts commit 7a542fb3270953fff039c9b1bd7ba7afa35a842c.

  1. … 1226 more files in changeset.
Revert "Merge branch 'trunk' into HDFS-7240"

This reverts commit 78d94acdced8e04252d0288b495d41371b1fb7b4, reversing

changes made to e0262147d60524d5718b2bfed48391a876dc3662.

  1. … 1226 more files in changeset.
HADOOP-15332. Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota.

  1. … 6 more files in changeset.
HADOOP-15332. Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota.

  1. … 6 more files in changeset.
HADOOP-15332. Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota.

  1. … 6 more files in changeset.
HADOOP-15297. Make S3A etag => checksum feature optional. Contributed by Steve Loughran.

  1. … 6 more files in changeset.
HADOOP-15297. Make S3A etag => checksum feature optional. Contributed by Steve Loughran.

(cherry picked from commit dd05871b8b57303fe0b0c652e03257b59c191802)

  1. … 6 more files in changeset.
HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key. Contributed by Steve Loughran.

(cherry picked from commit 7ac88244c54ce483729af3d2736d9f4731e230ca)

  1. … 8 more files in changeset.