Clone Tools
  • last updated 13 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16478. S3Guard bucket-info fails if the caller lacks s3:GetBucketLocation.

Contributed by Steve Loughran.

Includes HADOOP-16651. S3 getBucketLocation() can return "US" for us-east.

Change-Id: Ifc0dca76e51495ed1a8fc0f077b86bf125deff40

  1. … 6 more files in changeset.
HADOOP-16626. S3A ITestRestrictedReadAccess fails without S3Guard.

Contributed by Steve Loughran.

Change-Id: Ife730b80057ddd43e919438cb5b2abbda990e636

    • -90
    • +167
    ./ITestRestrictedReadAccess.java
  1. … 3 more files in changeset.
HADOOP-16599. Allow a SignerInitializer to be specified along with a Custom Signer

    • -0
    • +237
    ./ITestCustomSigner.java
    • -0
    • +590
    ./TestSignerManager.java
  1. … 8 more files in changeset.
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3

Contributed by Steve Loughran.

Includes

-S3A glob scans don't bother trying to resolve symlinks

-stack traces don't get lost in getFileStatuses() when exceptions are wrapped

-debug level logging of what is up in Globber

-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.

-ITestRestrictedReadAccess tests incomplete read access to files.

This adds a builder API for constructing globbers which other stores can use

so that they too can skip symlink resolution when not needed.

Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7

    • -0
    • +707
    ./ITestRestrictedReadAccess.java
  1. … 12 more files in changeset.
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename.

Contributed by Steve Loughran.

Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb

  1. … 69 more files in changeset.
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685)

This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:

* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests

* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?

* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads

* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:

* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3

    • -1
    • +4
    ./delegation/AbstractDelegationIT.java
    • -1
    • +42
    ./delegation/ITestDelegatedMRJob.java
  1. … 5 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran and Daryn Sharp.

    • -1
    • +1
    ./ITestAssumedRoleCommitOperations.java
    • -0
    • +138
    ./TestMarshalledCredentials.java
    • -0
    • +207
    ./delegation/AbstractDelegationIT.java
    • -0
    • +52
    ./delegation/CountInvocationsProvider.java
    • -0
    • +38
    ./delegation/ILoadTestRoleCredentials.java
    • -0
    • +295
    ./delegation/ILoadTestSessionCredentials.java
    • -0
    • +272
    ./delegation/ITestDelegatedMRJob.java
    • -0
    • +68
    ./delegation/ITestRoleDelegationInFileystem.java
    • -0
    • +122
    ./delegation/ITestRoleDelegationTokens.java
    • -0
    • +727
    ./delegation/ITestSessionDelegationInFileystem.java
    • -0
    • +282
    ./delegation/ITestSessionDelegationTokens.java
    • -0
    • +378
    ./delegation/MiniKerberizedHadoopCluster.java
  1. … 87 more files in changeset.
Revert "HADOOP-14556. S3A to support Delegation Tokens."

This reverts commit d7152332b32a575c3a92e3f4c44b95e58462528d.

    • -1
    • +1
    ./ITestAssumedRoleCommitOperations.java
    • -138
    • +0
    ./TestMarshalledCredentials.java
    • -207
    • +0
    ./delegation/AbstractDelegationIT.java
    • -52
    • +0
    ./delegation/CountInvocationsProvider.java
    • -38
    • +0
    ./delegation/ILoadTestRoleCredentials.java
    • -295
    • +0
    ./delegation/ILoadTestSessionCredentials.java
    • -272
    • +0
    ./delegation/ITestDelegatedMRJob.java
    • -68
    • +0
    ./delegation/ITestRoleDelegationInFileystem.java
    • -115
    • +0
    ./delegation/ITestRoleDelegationTokens.java
    • -727
    • +0
    ./delegation/ITestSessionDelegationInFileystem.java
    • -282
    • +0
    ./delegation/ITestSessionDelegationTokens.java
    • -378
    • +0
    ./delegation/MiniKerberizedHadoopCluster.java
  1. … 90 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran.

    • -1
    • +1
    ./ITestAssumedRoleCommitOperations.java
    • -0
    • +138
    ./TestMarshalledCredentials.java
    • -0
    • +207
    ./delegation/AbstractDelegationIT.java
    • -0
    • +52
    ./delegation/CountInvocationsProvider.java
    • -0
    • +103
    ./delegation/Csvout.java
    • -0
    • +38
    ./delegation/ILoadTestRoleCredentials.java
    • -0
    • +295
    ./delegation/ILoadTestSessionCredentials.java
    • -0
    • +272
    ./delegation/ITestDelegatedMRJob.java
    • -0
    • +68
    ./delegation/ITestRoleDelegationInFileystem.java
    • -0
    • +115
    ./delegation/ITestRoleDelegationTokens.java
    • -0
    • +727
    ./delegation/ITestSessionDelegationInFileystem.java
    • -0
    • +282
    ./delegation/ITestSessionDelegationTokens.java
    • -0
    • +378
    ./delegation/MiniKerberizedHadoopCluster.java
  1. … 90 more files in changeset.
HADOOP-15781. S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran.

Missed final change to ITestAssumeRole

HADOOP-15781. S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran

This is a subset of c07715e37895a2e406 to track changes in the AWS STS error messages.

If the later AWS SDK is ever backported to older branches, the rest of that patch will need

cherry-picking

HADOOP-15781 S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran

HADOOP-15583. Stabilize S3A Assumed Role support. Contributed by Steve Loughran.

    • -3
    • +2
    ./ITestAssumedRoleCommitOperations.java
  1. … 27 more files in changeset.
HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key. Contributed by Steve Loughran.

  1. … 8 more files in changeset.
HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key. Contributed by Steve Loughran.

  1. … 8 more files in changeset.
HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key. Contributed by Steve Loughran.

(cherry picked from commit 7ac88244c54ce483729af3d2736d9f4731e230ca)

  1. … 8 more files in changeset.
HADOOP-15176. Enhance IAM Assumed Role support in S3A client. Contributed by Steve Loughran

(cherry picked from commit 96c047fbb98c2378eed9693a724d4cbbd03c00fd)

    • -0
    • +789
    ./ITestAssumeRole.java
    • -0
    • +130
    ./ITestAssumedRoleCommitOperations.java
    • -0
    • +171
    ./RoleTestUtils.java
  1. … 21 more files in changeset.
HADOOP-15176. Enhance IAM Assumed Role support in S3A client. Contributed by Steve Loughran

    • -0
    • +789
    ./ITestAssumeRole.java
    • -0
    • +130
    ./ITestAssumedRoleCommitOperations.java
    • -0
    • +171
    ./RoleTestUtils.java
  1. … 21 more files in changeset.
HADOOP-15176. Enhance IAM Assumed Role support in S3A client. Contributed by Steve Loughran

(cherry picked from commit 96c047fbb98c2378eed9693a724d4cbbd03c00fd)

    • -0
    • +130
    ./ITestAssumedRoleCommitOperations.java
  1. … 21 more files in changeset.