Clone Tools
  • last updated 23 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16645. S3A Delegation Token extension point to use StoreContext.

Contributed by Steve Loughran.

This is part of the ongoing refactoring of the S3A codebase, with the

delegation token support (HADOOP-14556) no longer given a direct reference

to the owning S3AFileSystem. Instead it gets a StoreContext and a new

interface, DelegationOperations, to access those operations offered by S3AFS

which are specifically needed by the DT bindings.

The sole operation needed is listAWSPolicyRules(), which is used to allow

S3A FS and the S3Guard metastore to return the AWS policy rules needed to

access their specific services/buckets/tables, allowing the AssumedRole

delegation token to be locked down.

As further restructuring takes place, that interface's implementation

can be moved to wherever the new home for those operations ends up.

Although it changes the API of an extension point, that feature (S3

Delegation Tokens) has not shipped; backwards compatibility is not a

problem except for anyone who has implemented DT support against trunk.

To those developers: sorry.

Change-Id: I770f58b49ff7634a34875ba37b7d51c94d7c21da

  1. … 8 more files in changeset.
HADOOP-16477. S3A delegation token tests fail if fs.s3a.encryption.key set.

Contributed by Steve Loughran.

Change-Id: I843989f32472bbdefbd4fa504b26c7a614ab1cee

    • -2
    • +11
    ./ITestSessionDelegationInFileystem.java
  1. … 12 more files in changeset.
HADOOP-16658. S3A connector does not support including the token renewer in the token identifier.

Contributed by Phil Zampino.

Change-Id: Iea9d5028dcf58bda4da985604f5cd3ac283619bd

    • -5
    • +30
    ./ITestSessionDelegationTokens.java
    • -0
    • +31
    ./TestS3ADelegationTokenSupport.java
  1. … 10 more files in changeset.
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685)

This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:

* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests

* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?

* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads

* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:

* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3

  1. … 5 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran and Daryn Sharp.

    • -0
    • +295
    ./ILoadTestSessionCredentials.java
    • -0
    • +68
    ./ITestRoleDelegationInFileystem.java
    • -0
    • +122
    ./ITestRoleDelegationTokens.java
    • -0
    • +727
    ./ITestSessionDelegationInFileystem.java
    • -0
    • +282
    ./ITestSessionDelegationTokens.java
    • -0
    • +378
    ./MiniKerberizedHadoopCluster.java
    • -0
    • +171
    ./TestS3ADelegationTokenSupport.java
  1. … 90 more files in changeset.
Revert "HADOOP-14556. S3A to support Delegation Tokens."

This reverts commit d7152332b32a575c3a92e3f4c44b95e58462528d.

    • -295
    • +0
    ./ILoadTestSessionCredentials.java
    • -68
    • +0
    ./ITestRoleDelegationInFileystem.java
    • -115
    • +0
    ./ITestRoleDelegationTokens.java
    • -727
    • +0
    ./ITestSessionDelegationInFileystem.java
    • -282
    • +0
    ./ITestSessionDelegationTokens.java
    • -378
    • +0
    ./MiniKerberizedHadoopCluster.java
    • -171
    • +0
    ./TestS3ADelegationTokenSupport.java
  1. … 93 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran.

    • -0
    • +207
    ./AbstractDelegationIT.java
    • -0
    • +52
    ./CountInvocationsProvider.java
    • -0
    • +38
    ./ILoadTestRoleCredentials.java
    • -0
    • +295
    ./ILoadTestSessionCredentials.java
    • -0
    • +272
    ./ITestDelegatedMRJob.java
    • -0
    • +68
    ./ITestRoleDelegationInFileystem.java
    • -0
    • +115
    ./ITestRoleDelegationTokens.java
    • -0
    • +727
    ./ITestSessionDelegationInFileystem.java
    • -0
    • +282
    ./ITestSessionDelegationTokens.java
    • -0
    • +378
    ./MiniKerberizedHadoopCluster.java
    • -0
    • +171
    ./TestS3ADelegationTokenSupport.java
  1. … 93 more files in changeset.