Clone Tools
  • last updated 11 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with deletions

Contributed by Steve Loughran.

This overlaps the scanning for directory entries with batched calls to S3 DELETE and updates of the S3Guard tables.

It also uses S3Guard to list the files to delete, so find newly created files even when S3 listings are not use consistent.

For path which the client considers S3Guard to be authoritative, we also do a recursive LIST of the store and delete files; this is to find unindexed files and do guarantee that the delete(path, true) call really does delete everything underneath.

Change-Id: Ice2f6e940c506e0b3a78fa534a99721b1698708e

    • -1
    • +2
    ./fs/contract/s3a/ITestS3AContractRootDir.java
    • -18
    • +24
    ./fs/s3a/ITestS3AMetadataPersistenceException.java
    • -29
    • +97
    ./fs/s3a/ITestS3GuardListConsistency.java
    • -4
    • +59
    ./fs/s3a/ITestS3GuardOutOfBandOperations.java
    • -2
    • +2
    ./fs/s3a/commit/ITestCommitOperations.java
    • -1
    • +3
    ./fs/s3a/impl/ITestPartialRenamesDeletes.java
    • -4
    • +12
    ./fs/s3a/impl/TestPartialDeleteFailures.java
    • -52
    • +42
    ./fs/s3a/scale/ITestS3ADeleteManyFiles.java
  1. … 28 more files in changeset.
Revert "HADOOP-16193. Add extra S3A MPU test to see what happens if a file is created during the MPU. Contributed by Steve Loughran"

This reverts commit 69ddb36876c0b3819e5409d83b27d18d1da89b22.

HADOOP-16193. Add extra S3A MPU test to see what happens if a file is created during the MPU. Contributed by Steve Loughran

HADOOP-16481. ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region. (#1209). Contributed by Steve Loughran.

HADOOP-16499. S3A retry policy to be exponential (#1246). Contributed by Steve Loughran.

    • -10
    • +15
    ./fs/s3a/ITestS3AConfiguration.java
    • -10
    • +26
    ./fs/s3a/ITestS3ADelayedFNF.java
    • -8
    • +31
    ./fs/s3a/ITestS3AInconsistency.java
    • -6
    • +4
    ./fs/s3a/ITestS3ARemoteFileChanged.java
    • -1
    • +18
    ./fs/s3a/ITestS3GuardOutOfBandOperations.java
    • -0
    • +2
    ./fs/s3a/commit/staging/TestStagingCommitter.java
  1. … 5 more files in changeset.
HADOOP-16433. S3Guard: Filter expired entries and tombstones when listing with MetadataStore.listChildren().

Contributed by Gabor Bota.

This pulls the tracking of the lastUpdated timestamp of metadata entries up from the DDB metastore into all s3guard stores, and then uses this to filter out expired tombstones from listings.

Change-Id: I80f121236b49c75a024116f65a3ef29d3580b462

    • -1
    • +45
    ./fs/s3a/ITestS3GuardOutOfBandOperations.java
  1. … 6 more files in changeset.
HADOOP-16380. S3Guard to determine empty directory status for all non-root directories.

Contributed by Steve Loughran and Gabor Bota.

This

* Asks S3Guard to determine the empty directory status.

* Has S3A's root directory rm("/") command to always return false (as abfs does)

* Documents that object stores MAY do this

* Overloads ContractTestUtils.assertDeleted to let assertions declare that the source directory does not need to exist. This stops inconsistencies in directory listings failing a root test.

It avoids a recent regression (HADOOP-16279) where if there was a tombstone above the first element found in a directory listing, the directory would be considered empty, when in fact there were child entries. That could downgrade an rm(path, recursive) to a no-op, while also confusing rename(src, dest), as dest could be mistaken for an empty directory and so permit the copy above it, rather than reject it "destination path exists and is not empty".

Change-Id: I136a3d1a5a48a67e6155d790a40ff558d0d2c108

    • -3
    • +6
    ./fs/contract/s3a/ITestS3AContractRootDir.java
    • -3
    • +152
    ./fs/s3a/ITestS3GuardEmptyDirs.java
  1. … 5 more files in changeset.
HADOOP-16383. Pass ITtlTimeProvider instance in initialize method in MetadataStore interface. Contributed by Gabor Bota. (#1009)

    • -1
    • +3
    ./fs/s3a/ITestS3AMetadataPersistenceException.java
    • -8
    • +10
    ./fs/s3a/impl/TestPartialDeleteFailures.java
    • -11
    • +10
    ./fs/s3a/s3guard/ITestDynamoDBMetadataStore.java
    • -1
    • +2
    ./fs/s3a/scale/ITestLocalMetadataStoreScale.java
  1. … 10 more files in changeset.
HADOOP-15729. [s3a] Allow core threads to time out. (#1075)

    • -65
    • +110
    ./fs/s3a/scale/ITestS3AConcurrentOps.java
  1. … 1 more file in changeset.
HADOOP-16384: S3A: Avoid inconsistencies between DDB and S3.

Contributed by Steve Loughran

Contains

- HADOOP-16397. Hadoop S3Guard Prune command to support a -tombstone option.

- HADOOP-16406. ITestDynamoDBMetadataStore.testProvisionTable times out intermittently

This patch doesn't fix the underlying problem but it

* changes some tests to clean up better

* does a lot more in logging operations in against DDB, if enabled

* adds an entry point to dump the state of the metastore and s3 tables (precursor to fsck)

* adds a purge entry point to help clean up after a test run has got a store into a mess

* s3guard prune command adds -tombstone option to only clear tombstones

The outcome is that tests should pass consistently and if problems occur we have better diagnostics.

Change-Id: I3eca3f5529d7f6fec398c0ff0472919f08f054eb

    • -1
    • +17
    ./fs/contract/s3a/ITestS3AContractRootDir.java
    • -0
    • +44
    ./fs/s3a/ITestS3GuardEmptyDirs.java
    • -49
    • +70
    ./fs/s3a/ITestS3GuardListConsistency.java
    • -0
    • +9
    ./fs/s3a/commit/AbstractITCommitMRJob.java
    • -42
    • +0
    ./fs/s3a/impl/ITestPartialRenamesDeletes.java
    • -1
    • +14
    ./fs/s3a/s3guard/AbstractS3GuardToolTestBase.java
    • -80
    • +205
    ./fs/s3a/s3guard/ITestDynamoDBMetadataStore.java
    • -0
    • +269
    ./fs/s3a/s3guard/ITestS3GuardDDBRootOperations.java
  1. … 23 more files in changeset.
HADOOP-16357. TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists.

Contributed by Steve Loughran.

This patch

* changes the default for the staging committer to append, as we get for the classic FileOutputFormat committer

* adds a check for the dest path being a file not a dir

* adds tests for this

* Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present.

Change-Id: Id53742958ed1cf321ff96c9063505d64f3254f53

    • -51
    • +38
    ./fs/s3a/commit/AbstractCommitITest.java
    • -2
    • +2
    ./fs/s3a/commit/AbstractITCommitMRJob.java
    • -1
    • +33
    ./fs/s3a/commit/staging/StagingTestBase.java
    • -2
    • +2
    ./fs/s3a/commit/staging/TestStagingCommitter.java
  1. … 5 more files in changeset.
HADOOP-16393. S3Guard init command uses global settings, not those of target bucket.

Contributed by Steve Loughran.

Change-Id: I226a91ab8d7758340f8d221aa80a7abf9a0d3e8f

  1. … 1 more file in changeset.
HADOOP-15847. S3Guard testConcurrentTableCreations to set R/W capacity == 0

Contributed by lqjaclee

Change-Id: I4a4d5b29f2677c188799479e4db38f07fa0591d1

    • -0
    • +10
    ./fs/s3a/s3guard/ITestS3GuardConcurrentOps.java
  1. … 2 more files in changeset.
HADOOP-16409. Allow authoritative mode on non-qualified paths. Contributed by Sean Mackrory

  1. … 3 more files in changeset.
HADOOP-16396. Allow authoritative mode on a subdirectory. (#1043)

    • -0
    • +301
    ./fs/s3a/ITestAuthoritativePath.java
    • -43
    • +8
    ./fs/s3a/ITestS3GuardOutOfBandOperations.java
  1. … 7 more files in changeset.
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename.

Contributed by Steve Loughran.

Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb

    • -2
    • +69
    ./fs/contract/s3a/ITestS3AContractRename.java
    • -7
    • +95
    ./fs/s3a/ITestS3AFailureHandling.java
    • -1
    • +9
    ./fs/s3a/ITestS3AMetadataPersistenceException.java
    • -7
    • +6
    ./fs/s3a/ITestS3GuardListConsistency.java
    • -203
    • +16
    ./fs/s3a/auth/ITestAssumeRole.java
    • -0
    • +2
    ./fs/s3a/commit/AbstractCommitITest.java
    • -0
    • +4
    ./fs/s3a/commit/AbstractITCommitMRJob.java
    • -6
    • +121
    ./fs/s3a/commit/ITestCommitOperations.java
    • -2
    • +24
    ./fs/s3a/commit/magic/ITestMagicCommitMRJob.java
  1. … 56 more files in changeset.
HADOOP-16379: S3AInputStream.unbuffer should merge input stream stats into fs-wide stats

Contributed by Sahil Takiar

Change-Id: I2bcfaaea00d12c633757069402dcd0b91a5f5c05

  1. … 2 more files in changeset.
HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries (and tombstones).

Contributed by Gabor Bota.

Change-Id: I73a2d2861901dedfe7a0e783b310fbb95e7c1af9

    • -8
    • +341
    ./fs/s3a/ITestS3GuardOutOfBandOperations.java
    • -13
    • +114
    ./fs/s3a/s3guard/MetadataStoreTestBase.java
    • -3
    • +192
    ./fs/s3a/s3guard/TestS3Guard.java
    • -2
    • +12
    ./fs/s3a/scale/AbstractITestS3AMetadataStoreScale.java
  1. … 12 more files in changeset.
HADOOP-15563. S3Guard to support creating on-demand DDB tables.

Contributed by Steve Loughran

Change-Id: I2262b5b9f52e42ded8ed6f50fd39756f96e77087

    • -8
    • +19
    ./fs/s3a/s3guard/DDBCapacities.java
    • -59
    • +131
    ./fs/s3a/s3guard/ITestDynamoDBMetadataStore.java
    • -30
    • +11
    ./fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java
  1. … 5 more files in changeset.
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL"

This reverts commit b067f8acaa79b1230336900a5c62ba465b2adb28.

Change-Id: I584b050a56c0e6f70b11fa3f7db00d5ac46e7dd8

  1. … 14 more files in changeset.
Revert "HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors"

This reverts commit 5906268f0dd63a93eb591ddccf70d23b15e5c2ed.

  1. … 1 more file in changeset.
HADOOP-16332. Remove S3A dependency on http core.

Contributed by Steve Loughran.

Change-Id: I53209c993a405fefdb5e1b692d5a56d027d3b845

  1. … 2 more files in changeset.
HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors

  1. … 1 more file in changeset.
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite.

Contributed by Ben Roling.

S3Guard will now track the etag of uploaded files and, if an S3

bucket is versioned, the object version.

You can then control how to react to a mismatch between the data

in the DynamoDB table and that in the store: warn, fail, or, when

using versions, return the original value.

This adds two new columns to the table: etag and version.

This is transparent to older S3A clients -but when such clients

add/update data to the S3Guard table, they will not add these values.

As a result, the etag/version checks will not work with files uploaded by older clients.

For a consistent experience, upgrade all clients to use the latest hadoop version.

    • -0
    • +10
    ./fs/s3a/ITestS3AInconsistency.java
    • -38
    • +1136
    ./fs/s3a/ITestS3ARemoteFileChanged.java
    • -1
    • +40
    ./fs/s3a/ITestS3GuardListConsistency.java
    • -0
    • +9
    ./fs/s3a/ITestS3GuardOutOfBandOperations.java
    • -4
    • +193
    ./fs/s3a/TestStreamChangeTracker.java
  1. … 41 more files in changeset.
HADOOP-16221. S3Guard: add option to fail operation on metadata write failure.

    • -0
    • +136
    ./fs/s3a/ITestS3AMetadataPersistenceException.java
  1. … 10 more files in changeset.
HADOOP-16252. Add prefix to dynamo tables in tests.

Contributed by Ben Roling.

    • -8
    • +16
    ./fs/s3a/s3guard/ITestDynamoDBMetadataStore.java
  1. … 1 more file in changeset.
HADOOP-14747. S3AInputStream to implement CanUnbuffer.

Author: Sahil Takiar <stakiar@cloudera.com>

    • -0
    • +41
    ./fs/contract/s3a/ITestS3AContractUnbuffer.java
    • -0
    • +66
    ./fs/s3a/ITestS3AUnbuffer.java
    • -0
    • +76
    ./fs/s3a/TestS3AUnbuffer.java
  1. … 7 more files in changeset.
HADOOP-16118. S3Guard to support on-demand DDB tables.

This is the first step for on-demand operations: things recognize when they are using on-demand tables,

as do the tests.

Contributed by Steve Loughran.

    • -0
    • +108
    ./fs/s3a/s3guard/DDBCapacities.java
    • -85
    • +37
    ./fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java
  1. … 2 more files in changeset.
HADOOP-16050: s3a SSL connections should use OpenSSL

(cherry picked from commit aebf229c175dfa19fff3b31e9e67596f6c6124fa)

    • -0
    • +72
    ./fs/s3a/ITestS3ASSL.java
  1. … 14 more files in changeset.
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries. Contributed by Steve Loughran.

(cherry picked from commit 2ac5aab8d725f761a9f9723471a4426f6b5d78c4)

    • -0
    • +51
    ./fs/s3a/ITestS3AFailureHandling.java
  1. … 2 more files in changeset.