Clone Tools
  • last updated 14 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-15961. S3A committers: make sure there's regular progress() calls.

Contributed by lqjacklee.

Change-Id: I13ca153e1e32b21dbe64d6fb25e260e0ff66154d

    • -0
    • +43
    ./ProgressCounter.java
  1. … 3 more files in changeset.
HADOOP-16746. mkdirs and s3guard Authoritative mode.

Contributed by Steve Loughran.

This fixes two problems with S3Guard authoritative mode and

the auth directory flags which are stored in DynamoDB.

1. mkdirs was creating dir markers without the auth bit,

forcing needless scans on newly created directories and

files subsequently added; it was only with the first listStatus call

on that directory that the dir would be marked as authoritative -even

though it would be complete already.

2. listStatus(path) would reset the authoritative status bit of all

child directories even if they were already marked as authoritative.

Issue #2 is possibly the most expensive, as any treewalk using listStatus

(e.g globfiles) would clear the auth bit for all child directories before

listing them. And this would happen every single time...

essentially you weren't getting authoritative directory listings.

For the curious, that the major bug was actually found during testing

-we'd all missed it during reviews.

A lesson there: the better the tests the fewer the bugs.

Maybe also: something obvious and significant can get by code reviews.

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/BulkOperationState.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStore.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/NullMetadataStore.java

modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardWriteBack.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestPartialDeleteFailures.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStore.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreAuthoritativeMode.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardFsck.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java

modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java

Change-Id: Ic3ffda13f2af2430afedd50fd657b595c83e90a7

  1. … 15 more files in changeset.
HADOOP-16645. S3A Delegation Token extension point to use StoreContext.

Contributed by Steve Loughran.

This is part of the ongoing refactoring of the S3A codebase, with the

delegation token support (HADOOP-14556) no longer given a direct reference

to the owning S3AFileSystem. Instead it gets a StoreContext and a new

interface, DelegationOperations, to access those operations offered by S3AFS

which are specifically needed by the DT bindings.

The sole operation needed is listAWSPolicyRules(), which is used to allow

S3A FS and the S3Guard metastore to return the AWS policy rules needed to

access their specific services/buckets/tables, allowing the AssumedRole

delegation token to be locked down.

As further restructuring takes place, that interface's implementation

can be moved to wherever the new home for those operations ends up.

Although it changes the API of an extension point, that feature (S3

Delegation Tokens) has not shipped; backwards compatibility is not a

problem except for anyone who has implemented DT support against trunk.

To those developers: sorry.

Change-Id: I770f58b49ff7634a34875ba37b7d51c94d7c21da

    • -1
    • +4
    ./delegation/AbstractDelegationIT.java
    • -1
    • +3
    ./delegation/ITestRoleDelegationTokens.java
    • -1
    • +4
    ./delegation/ITestSessionDelegationTokens.java
  1. … 8 more files in changeset.
HADOOP-16665. Filesystems to be closed if they failed during initialize().

Contributed by Steve Loughran.

This FileSystem instantiation so if an IOException or RuntimeException is

raised in the invocation of FileSystem.initialize() then a best-effort

attempt is made to close the FS instance; exceptions raised that there

are swallowed.

The S3AFileSystem is also modified to do its own cleanup if an

IOException is raised during its initialize() process, it being the

FS we know has the "potential" to leak threads, especially in

extension points (e.g AWS Authenticators) which spawn threads.

Change-Id: Ib84073a606c9d53bf53cbfca4629876a03894f04

    • -2
    • +2
    ./ITestAssumedRoleCommitOperations.java
  1. … 9 more files in changeset.
HADOOP-16477. S3A delegation token tests fail if fs.s3a.encryption.key set.

Contributed by Steve Loughran.

Change-Id: I843989f32472bbdefbd4fa504b26c7a614ab1cee

    • -0
    • +2
    ./delegation/AbstractDelegationIT.java
    • -2
    • +11
    ./delegation/ITestSessionDelegationInFileystem.java
  1. … 11 more files in changeset.
HADOOP-16653. S3Guard DDB overreacts to no tag access (#1660). Contributed by Gabor Bota.

  1. … 2 more files in changeset.
HADOOP-16658. S3A connector does not support including the token renewer in the token identifier.

Contributed by Phil Zampino.

Change-Id: Iea9d5028dcf58bda4da985604f5cd3ac283619bd

    • -1
    • +1
    ./delegation/AbstractDelegationIT.java
    • -1
    • +1
    ./delegation/ITestRoleDelegationTokens.java
    • -5
    • +30
    ./delegation/ITestSessionDelegationTokens.java
    • -0
    • +31
    ./delegation/TestS3ADelegationTokenSupport.java
  1. … 10 more files in changeset.
HADOOP-16478. S3Guard bucket-info fails if the caller lacks s3:GetBucketLocation.

Contributed by Steve Loughran.

Includes HADOOP-16651. S3 getBucketLocation() can return "US" for us-east.

Change-Id: Ifc0dca76e51495ed1a8fc0f077b86bf125deff40

  1. … 6 more files in changeset.
HADOOP-16626. S3A ITestRestrictedReadAccess fails without S3Guard.

Contributed by Steve Loughran.

Change-Id: Ife730b80057ddd43e919438cb5b2abbda990e636

    • -90
    • +167
    ./ITestRestrictedReadAccess.java
  1. … 3 more files in changeset.
HADOOP-16599. Allow a SignerInitializer to be specified along with a Custom Signer

    • -0
    • +237
    ./ITestCustomSigner.java
    • -0
    • +590
    ./TestSignerManager.java
  1. … 8 more files in changeset.
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3

Contributed by Steve Loughran.

Includes

-S3A glob scans don't bother trying to resolve symlinks

-stack traces don't get lost in getFileStatuses() when exceptions are wrapped

-debug level logging of what is up in Globber

-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.

-ITestRestrictedReadAccess tests incomplete read access to files.

This adds a builder API for constructing globbers which other stores can use

so that they too can skip symlink resolution when not needed.

Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7

    • -0
    • +707
    ./ITestRestrictedReadAccess.java
  1. … 12 more files in changeset.
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename.

Contributed by Steve Loughran.

Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb

  1. … 69 more files in changeset.
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685)

This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:

* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests

* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?

* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads

* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:

* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3

    • -1
    • +4
    ./delegation/AbstractDelegationIT.java
    • -1
    • +42
    ./delegation/ITestDelegatedMRJob.java
  1. … 5 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran and Daryn Sharp.

    • -1
    • +1
    ./ITestAssumedRoleCommitOperations.java
    • -0
    • +138
    ./TestMarshalledCredentials.java
    • -0
    • +207
    ./delegation/AbstractDelegationIT.java
    • -0
    • +52
    ./delegation/CountInvocationsProvider.java
    • -0
    • +38
    ./delegation/ILoadTestRoleCredentials.java
    • -0
    • +295
    ./delegation/ILoadTestSessionCredentials.java
    • -0
    • +272
    ./delegation/ITestDelegatedMRJob.java
    • -0
    • +68
    ./delegation/ITestRoleDelegationInFileystem.java
    • -0
    • +122
    ./delegation/ITestRoleDelegationTokens.java
    • -0
    • +727
    ./delegation/ITestSessionDelegationInFileystem.java
    • -0
    • +282
    ./delegation/ITestSessionDelegationTokens.java
    • -0
    • +378
    ./delegation/MiniKerberizedHadoopCluster.java
  1. … 87 more files in changeset.
Revert "HADOOP-14556. S3A to support Delegation Tokens."

This reverts commit d7152332b32a575c3a92e3f4c44b95e58462528d.

    • -1
    • +1
    ./ITestAssumedRoleCommitOperations.java
    • -138
    • +0
    ./TestMarshalledCredentials.java
    • -207
    • +0
    ./delegation/AbstractDelegationIT.java
    • -52
    • +0
    ./delegation/CountInvocationsProvider.java
    • -38
    • +0
    ./delegation/ILoadTestRoleCredentials.java
    • -295
    • +0
    ./delegation/ILoadTestSessionCredentials.java
    • -272
    • +0
    ./delegation/ITestDelegatedMRJob.java
    • -68
    • +0
    ./delegation/ITestRoleDelegationInFileystem.java
    • -115
    • +0
    ./delegation/ITestRoleDelegationTokens.java
    • -727
    • +0
    ./delegation/ITestSessionDelegationInFileystem.java
    • -282
    • +0
    ./delegation/ITestSessionDelegationTokens.java
    • -378
    • +0
    ./delegation/MiniKerberizedHadoopCluster.java
  1. … 90 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran.

    • -1
    • +1
    ./ITestAssumedRoleCommitOperations.java
    • -0
    • +138
    ./TestMarshalledCredentials.java
    • -0
    • +207
    ./delegation/AbstractDelegationIT.java
    • -0
    • +52
    ./delegation/CountInvocationsProvider.java
    • -0
    • +103
    ./delegation/Csvout.java
    • -0
    • +38
    ./delegation/ILoadTestRoleCredentials.java
    • -0
    • +295
    ./delegation/ILoadTestSessionCredentials.java
    • -0
    • +272
    ./delegation/ITestDelegatedMRJob.java
    • -0
    • +68
    ./delegation/ITestRoleDelegationInFileystem.java
    • -0
    • +115
    ./delegation/ITestRoleDelegationTokens.java
    • -0
    • +727
    ./delegation/ITestSessionDelegationInFileystem.java
    • -0
    • +282
    ./delegation/ITestSessionDelegationTokens.java
    • -0
    • +378
    ./delegation/MiniKerberizedHadoopCluster.java
  1. … 90 more files in changeset.
HADOOP-15781. S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran.

Missed final change to ITestAssumeRole

HADOOP-15781. S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran

This is a subset of c07715e37895a2e406 to track changes in the AWS STS error messages.

If the later AWS SDK is ever backported to older branches, the rest of that patch will need

cherry-picking

HADOOP-15781 S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran

HADOOP-15583. Stabilize S3A Assumed Role support. Contributed by Steve Loughran.

    • -3
    • +2
    ./ITestAssumedRoleCommitOperations.java
  1. … 27 more files in changeset.
HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key. Contributed by Steve Loughran.

  1. … 8 more files in changeset.
HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key. Contributed by Steve Loughran.

  1. … 8 more files in changeset.
HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key. Contributed by Steve Loughran.

(cherry picked from commit 7ac88244c54ce483729af3d2736d9f4731e230ca)

  1. … 8 more files in changeset.
HADOOP-15176. Enhance IAM Assumed Role support in S3A client. Contributed by Steve Loughran

(cherry picked from commit 96c047fbb98c2378eed9693a724d4cbbd03c00fd)

    • -0
    • +789
    ./ITestAssumeRole.java
    • -0
    • +130
    ./ITestAssumedRoleCommitOperations.java
    • -0
    • +171
    ./RoleTestUtils.java
  1. … 21 more files in changeset.
HADOOP-15176. Enhance IAM Assumed Role support in S3A client. Contributed by Steve Loughran

    • -0
    • +789
    ./ITestAssumeRole.java
    • -0
    • +130
    ./ITestAssumedRoleCommitOperations.java
    • -0
    • +171
    ./RoleTestUtils.java
  1. … 21 more files in changeset.
HADOOP-15176. Enhance IAM Assumed Role support in S3A client. Contributed by Steve Loughran

(cherry picked from commit 96c047fbb98c2378eed9693a724d4cbbd03c00fd)

    • -0
    • +130
    ./ITestAssumedRoleCommitOperations.java
  1. … 21 more files in changeset.