Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16711.

This adds a new option fs.s3a.bucket.probe, range (0-2) to

control which probe for a bucket existence to perform on startup.

0: no checks

1: v1 check (as has been performend until now)

2: v2 bucket check, which also incudes a permission check. Default.

When set to 0, bucket existence checks won't be done

during initialization thus making it faster.

When the bucket is not available in S3,

or if fs.s3a.endpoint points to the wrong instance of a private S3 store

consecutive calls like listing, read, write etc. will fail with

an UnknownStoreException.

Contributed by:

* Mukund Thakur (main patch and tests)

* Rajesh Balamohan (v0 list and performance tests)

* lqjacklee (HADOOP-15990/v2 list)

* Steve Loughran (UnknownStoreException support)

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java

new file: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/UnknownStoreException.java

new file: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ErrorTranslation.java

modified: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

modified: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md

modified: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java

new file: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3ClientFactory.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AExceptionTranslation.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardToolTestBase.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java

modified: hadoop-tools/hadoop-aws/src/test/resources/core-site.xml

Change-Id: Ic174f803e655af172d81c1274ed92b51bdceb384

    • -0
    • +170
    ./fs/s3a/ITestS3ABucketExistence.java
    • -1
    • +19
    ./fs/s3a/TestS3AExceptionTranslation.java
  1. … 10 more files in changeset.
HADOOP-15961. S3A committers: make sure there's regular progress() calls.

Contributed by lqjacklee.

Change-Id: I13ca153e1e32b21dbe64d6fb25e260e0ff66154d

    • -0
    • +43
    ./fs/s3a/auth/ProgressCounter.java
    • -9
    • +22
    ./fs/s3a/commit/ITestCommitOperations.java
  1. … 2 more files in changeset.
HADOOP-16823. Large DeleteObject requests are their own Thundering Herd.

Contributed by Steve Loughran.

During S3A rename() and delete() calls, the list of objects delete is

built up into batches of a thousand and then POSTed in a single large

DeleteObjects request.

But as the IO capacity allowed on an S3 partition may only be 3500 writes

per second *and* each entry in that POST counts as a single write, then

one of those posts alone can trigger throttling on an already loaded

S3 directory tree. Which can trigger backoff and retry, with the same

thousand entry post, and so recreate the exact same problem.

Fixes

* Page size for delete object requests is set in

fs.s3a.bulk.delete.page.size; the default is 250.

* The property fs.s3a.experimental.aws.s3.throttling (default=true)

can be set to false to disable throttle retry logic in the AWS

client SDK -it is all handled in the S3A client. This

gives more visibility in to when operations are being throttled

* Bulk delete throttling events are logged to the log

org.apache.hadoop.fs.s3a.throttled log at INFO; if this appears

often then choose a smaller page size.

* The metric "store_io_throttled" adds the entire count of delete

requests when a single DeleteObjects request is throttled.

* A new quantile, "store_io_throttle_rate" can track throttling

load over time.

* DynamoDB metastore throttle resilience issues have also been

identified and fixed. Note: the fs.s3a.experimental.aws.s3.throttling

flag does not apply to DDB IO precisely because there may still be

lurking issues there and it safest to rely on the DynamoDB client

SDK.

Change-Id: I00f85cdd94fc008864d060533f6bd4870263fd84

    • -1
    • +1
    ./fs/contract/s3a/ITestS3AContractRename.java
    • -0
    • +1
    ./fs/s3a/commit/ITestCommitOperations.java
    • -2
    • +11
    ./fs/s3a/impl/ITestPartialRenamesDeletes.java
    • -7
    • +20
    ./fs/s3a/s3guard/ITestDynamoDBMetadataStore.java
    • -33
    • +108
    ./fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java
    • -15
    • +43
    ./fs/s3a/s3guard/ThrottleTracker.java
    • -0
    • +380
    ./fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java
    • -25
    • +77
    ./fs/s3a/scale/ITestS3ADeleteManyFiles.java
  1. … 19 more files in changeset.
HADOOP-16801. S3Guard listFiles will not query S3 if all listings are authoritative (#1815). Contributed by Mustafa İman.

    • -0
    • +104
    ./fs/s3a/s3guard/ITestDynamoDBMetadataStoreAuthoritativeMode.java
  1. … 5 more files in changeset.
HADOOP-16746. mkdirs and s3guard Authoritative mode.

Contributed by Steve Loughran.

This fixes two problems with S3Guard authoritative mode and

the auth directory flags which are stored in DynamoDB.

1. mkdirs was creating dir markers without the auth bit,

forcing needless scans on newly created directories and

files subsequently added; it was only with the first listStatus call

on that directory that the dir would be marked as authoritative -even

though it would be complete already.

2. listStatus(path) would reset the authoritative status bit of all

child directories even if they were already marked as authoritative.

Issue #2 is possibly the most expensive, as any treewalk using listStatus

(e.g globfiles) would clear the auth bit for all child directories before

listing them. And this would happen every single time...

essentially you weren't getting authoritative directory listings.

For the curious, that the major bug was actually found during testing

-we'd all missed it during reviews.

A lesson there: the better the tests the fewer the bugs.

Maybe also: something obvious and significant can get by code reviews.

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/BulkOperationState.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStore.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/NullMetadataStore.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardWriteBack.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestPartialDeleteFailures.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStore.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreAuthoritativeMode.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardFsck.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java

Change-Id: Ic3ffda13f2af2430afedd50fd657b595c83e90a7

    • -2
    • +11
    ./fs/s3a/ITestS3GuardWriteBack.java
    • -9
    • +8
    ./fs/s3a/auth/ITestRestrictedReadAccess.java
    • -0
    • +1
    ./fs/s3a/impl/TestPartialDeleteFailures.java
    • -58
    • +139
    ./fs/s3a/s3guard/ITestDynamoDBMetadataStoreAuthoritativeMode.java
    • -2
    • +3
    ./fs/s3a/s3guard/ITestS3GuardFsck.java
    • -24
    • +185
    ./fs/s3a/s3guard/TestS3Guard.java
  1. … 7 more files in changeset.
HADOOP-16792: Make S3 client request timeout configurable.

Contributed by Mustafa Iman.

This adds a new configuration option fs.s3a.connection.request.timeout

to declare the time out on HTTP requests to the AWS service;

0 means no timeout.

Measured in seconds; the usual time suffixes are all supported

Important: this is the maximum duration of any AWS service call,

including upload and copy operations. If non-zero, it must be larger

than the time to upload multi-megabyte blocks to S3 from the client,

and to rename many-GB files. Use with care.

Change-Id: I407745341068b702bf8f401fb96450a9f987c51c

    • -0
    • +13
    ./fs/s3a/ITestS3AConfiguration.java
  1. … 5 more files in changeset.
HADOOP-16732. S3Guard to support encrypted DynamoDB table (#1752). Contributed by Mingliang Liu.

    • -2
    • +39
    ./fs/s3a/s3guard/ITestDynamoDBMetadataStore.java
  1. … 7 more files in changeset.
HADOOP-16759. Filesystem openFile() builder to take a FileStatus param (#1761). Contributed by Steve Loughran

* Enhanced builder + FS spec

* s3a FS to use this to skip HEAD on open

* and to use version/etag when opening the file

works with S3AFileStatus FS and S3ALocatedFileStatus

    • -13
    • +128
    ./fs/s3a/ITestS3ARemoteFileChanged.java
    • -18
    • +38
    ./fs/s3a/ITestS3GuardOutOfBandOperations.java
  1. … 14 more files in changeset.
HADOOP-16346. Stabilize S3A OpenSSL support.

Introduces `openssl` as an option for `fs.s3a.ssl.channel.mode`.

The new option is documented and marked as experimental.

For details on how to use this, consult the peformance document

in the s3a documentation.

This patch is the successor to HADOOP-16050 "S3A SSL connections

should use OpenSSL" -which was reverted because of

incompatibilities between the wildfly OpenSSL client and the AWS

HTTPS servers (HADOOP-16347). With the Wildfly release moved up

to 1.0.7.Final (HADOOP-16405) everything should now work.

Related issues:

* HADOOP-15669. ABFS: Improve HTTPS Performance

* HADOOP-16050: S3A SSL connections should use OpenSSL

* HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8

* HADOOP-16405: Upgrade Wildfly Openssl version to 1.0.7.Final

Contributed by Sahil Takiar

Change-Id: I80a4bc5051519f186b7383b2c1cea140be42444e

    • -1
    • +14
    ./fs/contract/s3a/ITestS3AContractSeek.java
  1. … 8 more files in changeset.
HADOOP-16697. Tune/audit S3A authoritative mode.

Contains:

HADOOP-16474. S3Guard ProgressiveRenameTracker to mark destination

dirirectory as authoritative on success.

HADOOP-16684. S3guard bucket info to list a bit more about

authoritative paths.

HADOOP-16722. S3GuardTool to support FilterFileSystem.

This patch improves the marking of newly created/import directory

trees in S3Guard DynamoDB tables as authoritative.

Specific changes:

* Renamed directories are marked as authoritative if the entire

operation succeeded (HADOOP-16474).

* When updating parent table entries as part of any table write,

there's no overwriting of their authoritative flag.

s3guard import changes:

* new -verbose flag to print out what is going on.

* The "s3guard import" command lets you declare that a directory tree

is to be marked as authoritative

hadoop s3guard import -authoritative -verbose s3a://bucket/path

When importing a listing and a file is found, the import tool queries

the metastore and only updates the entry if the file is different from

before, where different == new timestamp, etag, or length. S3Guard can get

timestamp differences due to clock skew in PUT operations.

As the recursive list performed by the import command doesn't retrieve the

versionID, the existing entry may in fact be more complete.

When updating an existing due to clock skew the existing version ID

is propagated to the new entry (note: the etags must match; this is needed

to deal with inconsistent listings).

There is a new s3guard command to audit a s3guard bucket/path's

authoritative state:

hadoop s3guard authoritative -check-config s3a://bucket/path

This is primarily for testing/auditing.

The s3guard bucket-info command also provides some more details on the

authoritative state of a store (HADOOP-16684).

Change-Id: I58001341c04f6f3597fcb4fcb1581ccefeb77d91

    • -3
    • +33
    ./fs/contract/s3a/ITestS3AContractRename.java
    • -2
    • +3
    ./fs/s3a/impl/TestPartialDeleteFailures.java
    • -24
    • +55
    ./fs/s3a/s3guard/AbstractS3GuardToolTestBase.java
    • -0
    • +78
    ./fs/s3a/s3guard/ITestDynamoDBMetadataStore.java
    • -0
    • +732
    ./fs/s3a/s3guard/ITestDynamoDBMetadataStoreAuthoritativeMode.java
    • -50
    • +50
    ./fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java
  1. … 22 more files in changeset.
HADOOP-16642. ITestDynamoDBMetadataStoreScale fails when throttled.

Contributed by Steve Loughran.

Change-Id: If9b4ebe937200c17d7fdfb9923e6ae0ab4c541ef

HADOOP-16645. S3A Delegation Token extension point to use StoreContext.

Contributed by Steve Loughran.

This is part of the ongoing refactoring of the S3A codebase, with the

delegation token support (HADOOP-14556) no longer given a direct reference

to the owning S3AFileSystem. Instead it gets a StoreContext and a new

interface, DelegationOperations, to access those operations offered by S3AFS

which are specifically needed by the DT bindings.

The sole operation needed is listAWSPolicyRules(), which is used to allow

S3A FS and the S3Guard metastore to return the AWS policy rules needed to

access their specific services/buckets/tables, allowing the AssumedRole

delegation token to be locked down.

As further restructuring takes place, that interface's implementation

can be moved to wherever the new home for those operations ends up.

Although it changes the API of an extension point, that feature (S3

Delegation Tokens) has not shipped; backwards compatibility is not a

problem except for anyone who has implemented DT support against trunk.

To those developers: sorry.

Change-Id: I770f58b49ff7634a34875ba37b7d51c94d7c21da

  1. … 8 more files in changeset.
HADOOP-16450. ITestS3ACommitterFactory to not use useInconsistentClient. (#1145)

Contributed by Steve Loughran.

Change-Id: Ifb9771a73a07f744e4ed5f5e6be72473179db439

HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase (#1757)

Contributed by Mingliang Liu.

Signed-off-by: Steve Loughran <stevel@apache.org>

  1. … 1 more file in changeset.
HADOOP-16424. S3Guard fsck: Check internal consistency of the MetadataStore (#1691). Contributed by Gabor Bota.

    • -2
    • +118
    ./fs/s3a/s3guard/ITestS3GuardFsck.java
  1. … 4 more files in changeset.
HADOOP-16709. S3Guard: Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths (#1721). Contributed by Gabor Bota.

    • -16
    • +49
    ./fs/s3a/ITestS3GuardOutOfBandOperations.java
  1. … 5 more files in changeset.
HADOOP-16632 Speculating & Partitioned S3A magic committers can leave pending files under __magic (#1599)

Contributed by Steve Loughran.

This downgrade the checks for leftover __magic entries from fail to warn now the parallel

test runs make speculation more likely.

Change-Id: Ia4df2e90f82a06dbae69f3fdaadcbb0e0d713b38

HADOOP-16665. Filesystems to be closed if they failed during initialize().

Contributed by Steve Loughran.

This FileSystem instantiation so if an IOException or RuntimeException is

raised in the invocation of FileSystem.initialize() then a best-effort

attempt is made to close the FS instance; exceptions raised that there

are swallowed.

The S3AFileSystem is also modified to do its own cleanup if an

IOException is raised during its initialize() process, it being the

FS we know has the "potential" to leak threads, especially in

extension points (e.g AWS Authenticators) which spawn threads.

Change-Id: Ib84073a606c9d53bf53cbfca4629876a03894f04

    • -2
    • +2
    ./fs/s3a/auth/ITestRestrictedReadAccess.java
  1. … 8 more files in changeset.
HADOOP-16477. S3A delegation token tests fail if fs.s3a.encryption.key set.

Contributed by Steve Loughran.

Change-Id: I843989f32472bbdefbd4fa504b26c7a614ab1cee

    • -13
    • +67
    ./fs/s3a/AbstractTestS3AEncryption.java
    • -4
    • +14
    ./fs/s3a/ITestS3AAWSCredentialsProvider.java
    • -25
    • +21
    ./fs/s3a/ITestS3AEncryptionSSEC.java
    • -3
    • +4
    ./fs/s3a/ITestS3AEncryptionSSEKMSDefaultKey.java
    • -7
    • +13
    ./fs/s3a/ITestS3AEncryptionSSEKMSUserDefinedKey.java
    • -1
    • +14
    ./fs/s3a/ITestS3AMiscOperations.java
    • -3
    • +5
    ./fs/s3a/impl/ITestPartialRenamesDeletes.java
  1. … 3 more files in changeset.
HADOOP-16484. S3A to warn or fail if S3Guard is disabled (#1661). Contributed by Gabor Bota.

  1. … 5 more files in changeset.
HADOOP-16653. S3Guard DDB overreacts to no tag access (#1660). Contributed by Gabor Bota.

  1. … 2 more files in changeset.
HADOOP-16658. S3A connector does not support including the token renewer in the token identifier.

Contributed by Phil Zampino.

Change-Id: Iea9d5028dcf58bda4da985604f5cd3ac283619bd

  1. … 10 more files in changeset.
HADOOP-16478. S3Guard bucket-info fails if the caller lacks s3:GetBucketLocation.

Contributed by Steve Loughran.

Includes HADOOP-16651. S3 getBucketLocation() can return "US" for us-east.

Change-Id: Ifc0dca76e51495ed1a8fc0f077b86bf125deff40

    • -0
    • +61
    ./fs/s3a/impl/TestNeworkBinding.java
  1. … 5 more files in changeset.
HADOOP-16635. S3A "directories only" scan still does a HEAD.

Contributed by Steve Loughran.

Change-Id: I5e41d7f721364c392e1f4344db83dfa8c5aa06ce

    • -6
    • +134
    ./fs/s3a/ITestS3AFileOperationCost.java
  1. … 2 more files in changeset.
Revert "HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos."

This reverts commit 7a4b3d42c4e36e468c2a46fd48036a6fed547853.

The patch broke TestRouterWebHDFSContractSeek as it turns out that

WebHDFSInputStream.available() is always 0.

    • -1
    • +1
    ./fs/contract/s3a/ITestS3AContractSeek.java
  1. … 3 more files in changeset.
HADOOP-16520. Race condition in DDB table init and waiting threads. (#1576). Contributed by Gabor Bota.

Fixes HADOOP-16349. DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry

Change-Id: Ia83e92b9039ccb780090c99c41b4f71ef7539d35

    • -53
    • +120
    ./fs/s3a/s3guard/ITestDynamoDBMetadataStore.java
  1. … 6 more files in changeset.
HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos.

Contributed by lqjacklee.

Change-Id: I32bb00a683102e7ff8ff8ce0b8d9c3195ca7381c

    • -1
    • +1
    ./fs/contract/s3a/ITestS3AContractSeek.java
  1. … 3 more files in changeset.
HADOOP-16650. ITestS3AClosedFS failing.

Contributed by Steve Loughran.

Change-Id: Ia9bb84bd6455e210a54cfe9eb944feeda8b58da9

HADOOP-16626. S3A ITestRestrictedReadAccess fails without S3Guard.

Contributed by Steve Loughran.

Change-Id: Ife730b80057ddd43e919438cb5b2abbda990e636

    • -90
    • +167
    ./fs/s3a/auth/ITestRestrictedReadAccess.java
HADOOP-16570. S3A committers encounter scale issues.

Contributed by Steve Loughran.

This addresses two scale issues which has surfaced in large scale benchmarks

of the S3A Committers.

* Thread pools are not cleaned up.

This now happens, with tests.

* OOM on job commit for jobs with many thousands of tasks,

each generating tens of (very large) files.

Instead of loading all pending commits into memory as a single list, the list

of files to load is the sole list which is passed around; .pendingset files are

loaded and processed in isolation -and reloaded if necessary for any

abort/rollback operation.

The parallel commit/abort/revert operations now work at the .pendingset level,

rather than that of individual pending commit files. The existing parallelized

Tasks API is still used to commit those files, but with a null thread pool, so

as to serialize the operations.

Change-Id: I5c8240cd31800eaa83d112358770ca0eb2bca797

    • -10
    • +64
    ./fs/s3a/commit/AbstractITCommitProtocol.java
    • -15
    • +51
    ./fs/s3a/commit/staging/StagingTestBase.java
    • -0
    • +314
    ./fs/s3a/commit/staging/TestDirectoryCommitterScale.java
    • -15
    • +14
    ./fs/s3a/commit/staging/TestStagingCommitter.java
  1. … 11 more files in changeset.