Clone Tools
  • last updated 16 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16811: Use JUnit TemporaryFolder Rule in TestFileUtils (#1811). Contributed by David Mollitor.

HADOOP-16827. TestHarFileSystem.testInheritedMethodsImplemented broken. (#1817)

This is a regression caused by HADOOP-16759.

The test TestHarFileSystem uses introspection to verify that HarFileSystem

Does not implement methods to which there is a suitable implementation in

the base FileSystem class. Because of the way it checks this, refactoring

(protected) FileSystem methods in an IDE do not automatically change

the probes in TestHarFileSystem.

The changes in HADOOP-16759 did exactly that, and somehow managed

to get through the build/test process without this being noticed.

This patch fixes that failure.

Caused by and fixed by Steve Loughran.

Change-Id: If60d9c97058242871c02ad1addd424478f84f446

Signed-off-by: Mingliang Liu <liuml07@apache.org>

HADOOP-16683. Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException. Contributed by Adam Antal

(cherry picked from commit 3d249301f42130de80ab3ecf8c783bb51b0fa8a3)

    • -0
    • +7
    ./io/retry/UnreliableImplementation.java
  1. … 1 more file in changeset.
HADOOP-16580. Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException. Contributed by Adam Antal

(cherry picked from commit c79a5f2d9930f58ad95864c59cd0a6164cd53280)

    • -0
    • +16
    ./io/retry/UnreliableImplementation.java
    • -0
    • +17
    ./io/retry/UnreliableInterface.java
  1. … 1 more file in changeset.
HADOOP-16759. Filesystem openFile() builder to take a FileStatus param (#1761). Contributed by Steve Loughran

* Enhanced builder + FS spec

* s3a FS to use this to skip HEAD on open

* and to use version/etag when opening the file

works with S3AFileStatus FS and S3ALocatedFileStatus

  1. … 17 more files in changeset.
HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li.

  1. … 3 more files in changeset.
HDFS-13616. Batch listing of multiple directories (#1725)

  1. … 23 more files in changeset.
HADOOP-16790. Add Write Convenience Methods.

Contributed by David Mollitor.

This adds operations in FileUtil to write text to a file via

either a FileSystem or FileContext instance.

Change-Id: I5fe8fcf1bdbdbc734e137f922a75a822f2b88410

  1. … 1 more file in changeset.
HADOOP-16790. Add Write Convenience Methods.

Contributed by David Mollitor.

This adds operations in FileUtil to write text to a file via

either a FileSystem or FileContext instance.

Change-Id: I5fe8fcf1bdbdbc734e137f922a75a822f2b88410

  1. … 1 more file in changeset.
HADOOP-16683. Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException. Contributed by Adam Antal

    • -0
    • +7
    ./io/retry/UnreliableImplementation.java
  1. … 1 more file in changeset.
HADOOP-16683. Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException. Contributed by Adam Antal

    • -0
    • +7
    ./io/retry/UnreliableImplementation.java
  1. … 1 more file in changeset.
HADOOP-16749. Configuration parsing of CDATA values are blank

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

(cherry picked from commit 23a77050e08a8d57581da43fa8d6cd11ad8eaa0e)

  1. … 1 more file in changeset.
HADOOP-16749. Configuration parsing of CDATA values are blank

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

  1. … 1 more file in changeset.
HADOOP-16749. Configuration parsing of CDATA values are blank

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

(cherry picked from commit c4fb43c94259546f2c96b50ceae9cd3adb726166)

  1. … 1 more file in changeset.
HADOOP-16749. Configuration parsing of CDATA values are blank

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

  1. … 1 more file in changeset.
HADOOP-16749. Configuration parsing of CDATA values are blank

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

(cherry picked from commit c4fb43c94259546f2c96b50ceae9cd3adb726166)

  1. … 1 more file in changeset.
HADOOP-16697. Tune/audit S3A authoritative mode.

Contains:

HADOOP-16474. S3Guard ProgressiveRenameTracker to mark destination

dirirectory as authoritative on success.

HADOOP-16684. S3guard bucket info to list a bit more about

authoritative paths.

HADOOP-16722. S3GuardTool to support FilterFileSystem.

This patch improves the marking of newly created/import directory

trees in S3Guard DynamoDB tables as authoritative.

Specific changes:

* Renamed directories are marked as authoritative if the entire

operation succeeded (HADOOP-16474).

* When updating parent table entries as part of any table write,

there's no overwriting of their authoritative flag.

s3guard import changes:

* new -verbose flag to print out what is going on.

* The "s3guard import" command lets you declare that a directory tree

is to be marked as authoritative

hadoop s3guard import -authoritative -verbose s3a://bucket/path

When importing a listing and a file is found, the import tool queries

the metastore and only updates the entry if the file is different from

before, where different == new timestamp, etag, or length. S3Guard can get

timestamp differences due to clock skew in PUT operations.

As the recursive list performed by the import command doesn't retrieve the

versionID, the existing entry may in fact be more complete.

When updating an existing due to clock skew the existing version ID

is propagated to the new entry (note: the etags must match; this is needed

to deal with inconsistent listings).

There is a new s3guard command to audit a s3guard bucket/path's

authoritative state:

hadoop s3guard authoritative -check-config s3a://bucket/path

This is primarily for testing/auditing.

The s3guard bucket-info command also provides some more details on the

authoritative state of a store (HADOOP-16684).

Change-Id: I58001341c04f6f3597fcb4fcb1581ccefeb77d91

  1. … 30 more files in changeset.
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.

This hardens the wasb and abfs output streams' resilience to being invoked

in/after close().

wasb:

Explicity raise IOEs on operations invoked after close,

rather than implicitly raise NPEs.

This ensures that invocations which catch and swallow IOEs will perform as

expected.

abfs:

When rethrowing an IOException in the close() call, explicitly wrap it

with a new instance of the same subclass.

This is needed to handle failures in try-with-resources clauses, where

any exception in closed() is added as a suppressed exception to the one

thrown in the try {} clause

*and you cannot attach the same exception to itself*

Contributed by Steve Loughran.

Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b

  1. … 5 more files in changeset.
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.

This hardens the wasb and abfs output streams' resilience to being invoked

in/after close().

wasb:

Explicity raise IOEs on operations invoked after close,

rather than implicitly raise NPEs.

This ensures that invocations which catch and swallow IOEs will perform as

expected.

abfs:

When rethrowing an IOException in the close() call, explicitly wrap it

with a new instance of the same subclass.

This is needed to handle failures in try-with-resources clauses, where

any exception in closed() is added as a suppressed exception to the one

thrown in the try {} clause

*and you cannot attach the same exception to itself*

Contributed by Steve Loughran.

Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b

  1. … 5 more files in changeset.
HADOOP-16727. KMS Jetty server does not startup if trust store password is null.

    • -0
    • +266
    ./http/TestSSLHttpServerConfigs.java
    • -39
    • +74
    ./security/ssl/KeyStoreTestUtil.java
  1. … 1 more file in changeset.
HADOOP-16789. In TestZKFailoverController, restore changes from HADOOP-11149 that were dropped by HDFS-6440. Contributed by Jim Brennan.

    • -28
    • +27
    ./ha/TestZKFailoverController.java
HADOOP-16751. DurationInfo text parsing/formatting should be moved out of hotpath.

Contributed by Rajesh Balamohan

Change-Id: Icc3dcfa81aa69164f2c088f9b533d231138cbb8b

  1. … 1 more file in changeset.
HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He.

  1. … 19 more files in changeset.
HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He.

(cherry picked from commit d79cce20abbbf321f6dcce03f4087544124a7cd2)

  1. … 19 more files in changeset.
HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He.

(cherry picked from commit d79cce20abbbf321f6dcce03f4087544124a7cd2)

  1. … 19 more files in changeset.
YARN-9923. Introduce HealthReporter interface to support multiple health checker files. Contributed by Adam Antal

    • -145
    • +0
    ./util/TestNodeHealthScriptRunner.java
  1. … 38 more files in changeset.
HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase (#1757)

Contributed by Mingliang Liu.

Signed-off-by: Steve Loughran <stevel@apache.org>

  1. … 1 more file in changeset.
HADOOP-16718. Allow disabling Server Name Indication (SNI) for Jetty. Contributed by Aravindan Vijayan.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

Reviewed-by: Siyao Meng <smeng@cloudera.com>

    • -1
    • +1
    ./conf/TestCommonConfigurationFields.java
  1. … 2 more files in changeset.
HADOOP-16718. Allow disabling Server Name Indication (SNI) for Jetty. Contributed by Aravindan Vijayan.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

Reviewed-by: Siyao Meng <smeng@cloudera.com>

(cherry picked from commit f1ab7f18c423a9cfc59292d25fa178e73715b85b)

    • -1
    • +1
    ./conf/TestCommonConfigurationFields.java
  1. … 2 more files in changeset.
HADOOP-16718. Allow disabling Server Name Indication (SNI) for Jetty. Contributed by Aravindan Vijayan.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

Reviewed-by: Siyao Meng <smeng@cloudera.com>

(cherry picked from commit f1ab7f18c423a9cfc59292d25fa178e73715b85b)

(cherry picked from commit f0c1403ec382a7a8c25b0311db0c88749576c308)

    • -1
    • +1
    ./conf/TestCommonConfigurationFields.java
  1. … 2 more files in changeset.