Clone Tools
  • last updated 15 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16811: Use JUnit TemporaryFolder Rule in TestFileUtils (#1811). Contributed by David Mollitor.

HADOOP-16827. TestHarFileSystem.testInheritedMethodsImplemented broken. (#1817)

This is a regression caused by HADOOP-16759.

The test TestHarFileSystem uses introspection to verify that HarFileSystem

Does not implement methods to which there is a suitable implementation in

the base FileSystem class. Because of the way it checks this, refactoring

(protected) FileSystem methods in an IDE do not automatically change

the probes in TestHarFileSystem.

The changes in HADOOP-16759 did exactly that, and somehow managed

to get through the build/test process without this being noticed.

This patch fixes that failure.

Caused by and fixed by Steve Loughran.

Change-Id: If60d9c97058242871c02ad1addd424478f84f446

Signed-off-by: Mingliang Liu <liuml07@apache.org>

HADOOP-16683. Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException. Contributed by Adam Antal

(cherry picked from commit 3d249301f42130de80ab3ecf8c783bb51b0fa8a3)

HADOOP-16580. Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException. Contributed by Adam Antal

(cherry picked from commit c79a5f2d9930f58ad95864c59cd0a6164cd53280)

HADOOP-16792: Make S3 client request timeout configurable.

Contributed by Mustafa Iman.

This adds a new configuration option fs.s3a.connection.request.timeout

to declare the time out on HTTP requests to the AWS service;

0 means no timeout.

Measured in seconds; the usual time suffixes are all supported

Important: this is the maximum duration of any AWS service call,

including upload and copy operations. If non-zero, it must be larger

than the time to upload multi-megabyte blocks to S3 from the client,

and to rename many-GB files. Use with care.

Change-Id: I407745341068b702bf8f401fb96450a9f987c51c

  1. … 5 more files in changeset.
HADOOP-16732. S3Guard to support encrypted DynamoDB table (#1752). Contributed by Mingliang Liu.

  1. … 8 more files in changeset.
HADOOP-16670. Stripping Submarine code from Hadoop codebase. Contributed by Zhankun Tang.

Reviewed-by: Akira Ajisaka <aajisaka@apache.org>

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

  1. … 228 more files in changeset.
HADOOP-16759. Filesystem openFile() builder to take a FileStatus param (#1761). Contributed by Steve Loughran

* Enhanced builder + FS spec

* s3a FS to use this to skip HEAD on open

* and to use version/etag when opening the file

works with S3AFileStatus FS and S3ALocatedFileStatus

    • -5
    • +7
    ./main/java/org/apache/hadoop/fs/FileContext.java
    • -20
    • +21
    ./main/java/org/apache/hadoop/fs/FileSystem.java
    • -0
    • +94
    ./main/java/org/apache/hadoop/fs/impl/OpenFileParameters.java
    • -7
    • +16
    ./site/markdown/filesystem/filesystem.md
  1. … 5 more files in changeset.
HADOOP-16346. Stabilize S3A OpenSSL support.

Introduces `openssl` as an option for `fs.s3a.ssl.channel.mode`.

The new option is documented and marked as experimental.

For details on how to use this, consult the peformance document

in the s3a documentation.

This patch is the successor to HADOOP-16050 "S3A SSL connections

should use OpenSSL" -which was reverted because of

incompatibilities between the wildfly OpenSSL client and the AWS

HTTPS servers (HADOOP-16347). With the Wildfly release moved up

to 1.0.7.Final (HADOOP-16405) everything should now work.

Related issues:

* HADOOP-15669. ABFS: Improve HTTPS Performance

* HADOOP-16050: S3A SSL connections should use OpenSSL

* HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8

* HADOOP-16405: Upgrade Wildfly Openssl version to 1.0.7.Final

Contributed by Sahil Takiar

Change-Id: I80a4bc5051519f186b7383b2c1cea140be42444e

  1. … 7 more files in changeset.
HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li.

    • -172
    • +86
    ./main/java/org/apache/hadoop/ha/HAAdmin.java
    • -5
    • +0
    ./test/java/org/apache/hadoop/ha/TestHAAdmin.java
  1. … 2 more files in changeset.
HADOOP-16621. [pb-upgrade] Remove Protobuf classes from signatures of Public APIs. Contributed by Vinayakumar B. (#1803)

  1. … 3 more files in changeset.
HDFS-13616. Batch listing of multiple directories (#1725)

    • -0
    • +27
    ./main/java/org/apache/hadoop/fs/FileSystem.java
    • -0
    • +91
    ./main/java/org/apache/hadoop/fs/PartialListing.java
  1. … 21 more files in changeset.
HADOOP-16790. Add Write Convenience Methods.

Contributed by David Mollitor.

This adds operations in FileUtil to write text to a file via

either a FileSystem or FileContext instance.

Change-Id: I5fe8fcf1bdbdbc734e137f922a75a822f2b88410

    • -0
    • +236
    ./main/java/org/apache/hadoop/fs/FileUtil.java
HADOOP-16790. Add Write Convenience Methods.

Contributed by David Mollitor.

This adds operations in FileUtil to write text to a file via

either a FileSystem or FileContext instance.

Change-Id: I5fe8fcf1bdbdbc734e137f922a75a822f2b88410

    • -0
    • +236
    ./main/java/org/apache/hadoop/fs/FileUtil.java
HADOOP-16683. Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException. Contributed by Adam Antal

HADOOP-16683. Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException. Contributed by Adam Antal

HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang.

(cherry picked from commit 26a969ec734dbdbf1d544f486dfa33f15c291789)

  1. … 2 more files in changeset.
HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang.

(cherry picked from commit 26a969ec734dbdbf1d544f486dfa33f15c291789)

  1. … 2 more files in changeset.
HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang.

  1. … 2 more files in changeset.
HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang.

(cherry picked from commit 26a969ec734dbdbf1d544f486dfa33f15c291789)

  1. … 2 more files in changeset.
HADOOP-16590. Update OS login modules for IBM JDK. Contributed by Nicholas Marion

Close #1484

HADOOP-16749. Configuration parsing of CDATA values are blank

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

(cherry picked from commit 23a77050e08a8d57581da43fa8d6cd11ad8eaa0e)

HADOOP-16749. Configuration parsing of CDATA values are blank

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

HADOOP-16749. Configuration parsing of CDATA values are blank

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

(cherry picked from commit c4fb43c94259546f2c96b50ceae9cd3adb726166)

HADOOP-16749. Configuration parsing of CDATA values are blank

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

HADOOP-16749. Configuration parsing of CDATA values are blank

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

(cherry picked from commit c4fb43c94259546f2c96b50ceae9cd3adb726166)

HADOOP-16697. Tune/audit S3A authoritative mode.

Contains:

HADOOP-16474. S3Guard ProgressiveRenameTracker to mark destination

dirirectory as authoritative on success.

HADOOP-16684. S3guard bucket info to list a bit more about

authoritative paths.

HADOOP-16722. S3GuardTool to support FilterFileSystem.

This patch improves the marking of newly created/import directory

trees in S3Guard DynamoDB tables as authoritative.

Specific changes:

* Renamed directories are marked as authoritative if the entire

operation succeeded (HADOOP-16474).

* When updating parent table entries as part of any table write,

there's no overwriting of their authoritative flag.

s3guard import changes:

* new -verbose flag to print out what is going on.

* The "s3guard import" command lets you declare that a directory tree

is to be marked as authoritative

hadoop s3guard import -authoritative -verbose s3a://bucket/path

When importing a listing and a file is found, the import tool queries

the metastore and only updates the entry if the file is different from

before, where different == new timestamp, etag, or length. S3Guard can get

timestamp differences due to clock skew in PUT operations.

As the recursive list performed by the import command doesn't retrieve the

versionID, the existing entry may in fact be more complete.

When updating an existing due to clock skew the existing version ID

is propagated to the new entry (note: the etags must match; this is needed

to deal with inconsistent listings).

There is a new s3guard command to audit a s3guard bucket/path's

authoritative state:

hadoop s3guard authoritative -check-config s3a://bucket/path

This is primarily for testing/auditing.

The s3guard bucket-info command also provides some more details on the

authoritative state of a store (HADOOP-16684).

Change-Id: I58001341c04f6f3597fcb4fcb1581ccefeb77d91

  1. … 30 more files in changeset.
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.

This hardens the wasb and abfs output streams' resilience to being invoked

in/after close().

wasb:

Explicity raise IOEs on operations invoked after close,

rather than implicitly raise NPEs.

This ensures that invocations which catch and swallow IOEs will perform as

expected.

abfs:

When rethrowing an IOException in the close() call, explicitly wrap it

with a new instance of the same subclass.

This is needed to handle failures in try-with-resources clauses, where

any exception in closed() is added as a suppressed exception to the one

thrown in the try {} clause

*and you cannot attach the same exception to itself*

Contributed by Steve Loughran.

Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b

  1. … 5 more files in changeset.
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.

This hardens the wasb and abfs output streams' resilience to being invoked

in/after close().

wasb:

Explicity raise IOEs on operations invoked after close,

rather than implicitly raise NPEs.

This ensures that invocations which catch and swallow IOEs will perform as

expected.

abfs:

When rethrowing an IOException in the close() call, explicitly wrap it

with a new instance of the same subclass.

This is needed to handle failures in try-with-resources clauses, where

any exception in closed() is added as a suppressed exception to the one

thrown in the try {} clause

*and you cannot attach the same exception to itself*

Contributed by Steve Loughran.

Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b

  1. … 5 more files in changeset.
HADOOP-16751. Followup: move java import. (#1799)

This moves the import of a java module to the preferred place in the import ordering.

Change-Id: I1a594e3d954554a72c2b71c954eda0ae940a8f70