Clone Tools
  • last updated 26 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found.

Contributed by Adam Antal.

(Revised patch applied after stevel committed the wrong one; that has been reverted)

  1. … 2 more files in changeset.
Revert "HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found."

This reverts commit c4a00d1ad3d3cfc02a6a4e1e04353678f2d588e1.

  1. … 2 more files in changeset.
HADOOP-16098. Fix javadoc warnings in hadoop-aws. Contributed by Masatake Iwasaki.

    • -0
    • +2
    ./s3a/TemporaryAWSCredentialsProvider.java
    • -0
    • +1
    ./s3a/auth/AbstractSessionCredentialsProvider.java
    • -0
    • +3
    ./s3a/auth/MarshalledCredentialBinding.java
  1. … 3 more files in changeset.
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may

options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,

as the existing input format/record readers can't handle S3 select output where

the stream is shorter than the file length, and splitting plain text is suboptimal.

Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific

configuration parameters which can be set in jobs and used to set filesystem input stream

options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran

    • -0
    • +53
    ./s3a/InternalConstants.java
    • -0
    • +77
    ./s3a/select/InternalSelectConstants.java
    • -0
    • +431
    ./s3a/select/SelectBinding.java
    • -0
    • +296
    ./s3a/select/SelectConstants.java
    • -0
    • +457
    ./s3a/select/SelectInputStream.java
  1. … 57 more files in changeset.
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.

Contributed by Ewan Higgs and Steve Loughran.

(cherry picked from commit c1d24f848345f6d34a2ac2d570d49e9787a0df6a)

  1. … 10 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran and Daryn Sharp.

    • -6
    • +74
    ./s3a/AWSCredentialProviderList.java
    • -9
    • +24
    ./s3a/SimpleAWSCredentialsProvider.java
    • -37
    • +52
    ./s3a/TemporaryAWSCredentialsProvider.java
    • -0
    • +70
    ./s3a/auth/AbstractAWSCredentialProvider.java
    • -0
    • +170
    ./s3a/auth/AbstractSessionCredentialsProvider.java
  1. … 87 more files in changeset.
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found.

Contributed by Adam Antal.

  1. … 2 more files in changeset.
Revert "HADOOP-14556. S3A to support Delegation Tokens."

This reverts commit d7152332b32a575c3a92e3f4c44b95e58462528d.

    • -74
    • +6
    ./s3a/AWSCredentialProviderList.java
    • -24
    • +9
    ./s3a/SimpleAWSCredentialsProvider.java
    • -52
    • +37
    ./s3a/TemporaryAWSCredentialsProvider.java
    • -70
    • +0
    ./s3a/auth/AbstractAWSCredentialProvider.java
    • -170
    • +0
    ./s3a/auth/AbstractSessionCredentialsProvider.java
  1. … 90 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran.

    • -6
    • +74
    ./s3a/AWSCredentialProviderList.java
    • -9
    • +24
    ./s3a/SimpleAWSCredentialsProvider.java
    • -37
    • +52
    ./s3a/TemporaryAWSCredentialsProvider.java
    • -0
    • +70
    ./s3a/auth/AbstractAWSCredentialProvider.java
    • -0
    • +170
    ./s3a/auth/AbstractSessionCredentialsProvider.java
  1. … 90 more files in changeset.
HADOOP-15988. DynamoDBMetadataStore#innerGet should support empty directory flag when using authoritative listings. Contributed by Gabor Bota.

    • -6
    • +10
    ./s3a/s3guard/DynamoDBMetadataStore.java
  1. … 1 more file in changeset.
HADOOP-15845. Require explicit URI on CLI for s3guard init and destroy. Contributed by Gabor Bota.

  1. … 2 more files in changeset.
HADOOP-15428. s3guard bucket-info will create s3guard table if FS is set to do this automatically. (Contributed by Gabor Bota)

  1. … 2 more files in changeset.
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.

Contributed by Steve Loughran.

(cherry picked from commit 4c106fca0ca91536e288f11052568406a0b84300)

  1. … 1 more file in changeset.
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.

Contributed by Ewan Higgs and Steve Loughran.

  1. … 10 more files in changeset.
HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed by Gabor Bota.

HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.

Contributed by Steve Loughran.

(cherry picked from commit 4c106fca0ca91536e288f11052568406a0b84300)

  1. … 1 more file in changeset.
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.

Contributed by Steve Loughran.

(cherry picked from commit 4c106fca0ca91536e288f11052568406a0b84300)

  1. … 1 more file in changeset.
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.

Contributed by Steve Loughran.

  1. … 1 more file in changeset.
HADOOP-15947. Fix ITestDynamoDBMetadataStore test error issues. Contributed by Gabor Bota.

    • -1
    • +1
    ./s3a/s3guard/DynamoDBMetadataStore.java
  1. … 1 more file in changeset.
HADOOP-15798. LocalMetadataStore put() does not retain isDeleted in parent listing. Contributed by Gabor Bota.

    • -10
    • +19
    ./s3a/s3guard/LocalMetadataStore.java
  1. … 2 more files in changeset.
HADOOP-15370. S3A log message on rm s3a://bucket/ not intuitive. Contributed by Gabor Bota.

HADOOP-15826. @Retries annotation of putObject() call & uses wrong. Contributed by Steve Loughran and Ewan Higgs.

HADOOP-15837. DynamoDB table Update can fail S3A FS init. Contributed by Steve Loughran.

(cherry picked from commit f1fbc563719db88c50d57820b5052880cfd6b1aa)

    • -1
    • +4
    ./s3a/s3guard/DynamoDBMetadataStore.java
HADOOP-15837. DynamoDB table Update can fail S3A FS init. Contributed by Steve Loughran.

(cherry picked from commit f1fbc563719db88c50d57820b5052880cfd6b1aa)

    • -1
    • +4
    ./s3a/s3guard/DynamoDBMetadataStore.java
HADOOP-15837. DynamoDB table Update can fail S3A FS init. Contributed by Steve Loughran.

    • -1
    • +4
    ./s3a/s3guard/DynamoDBMetadataStore.java
HADOOP-15837. DynamoDB table Update can fail S3A FS init. Contributed by Steve Loughran.

(cherry picked from commit ee816f1fd78b029b9efa567e8b1b62391563de14)

    • -14
    • +75
    ./s3a/s3guard/DynamoDBMetadataStore.java
  1. … 1 more file in changeset.
HADOOP-15837. DynamoDB table Update can fail S3A FS init. Contributed by Steve Loughran.

    • -14
    • +75
    ./s3a/s3guard/DynamoDBMetadataStore.java
  1. … 1 more file in changeset.
HADOOP-15827. NPE in DynamoDBMetadataStore.lambda$listChildren for root + auth S3Guard. Contributed by Gabor Bota

    • -1
    • +1
    ./s3a/s3guard/DynamoDBMetadataStore.java
HADOOP-15621 2/2 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota

    • -0
    • +39
    ./s3a/s3guard/ExpirableMetadata.java
  1. … 1 more file in changeset.
HADOOP-15621 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota

    • -3
    • +12
    ./s3a/s3guard/DirListingMetadata.java
    • -3
    • +4
    ./s3a/s3guard/DynamoDBMetadataStore.java
    • -36
    • +25
    ./s3a/s3guard/PathMetadataDynamoDBTranslation.java
  1. … 6 more files in changeset.