Clone Tools
  • last updated 13 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.

This hardens the wasb and abfs output streams' resilience to being invoked

in/after close().

wasb:

Explicity raise IOEs on operations invoked after close,

rather than implicitly raise NPEs.

This ensures that invocations which catch and swallow IOEs will perform as

expected.

abfs:

When rethrowing an IOException in the close() call, explicitly wrap it

with a new instance of the same subclass.

This is needed to handle failures in try-with-resources clauses, where

any exception in closed() is added as a suppressed exception to the one

thrown in the try {} clause

*and you cannot attach the same exception to itself*

Contributed by Steve Loughran.

Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b

  1. … 5 more files in changeset.
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.

This hardens the wasb and abfs output streams' resilience to being invoked

in/after close().

wasb:

Explicity raise IOEs on operations invoked after close,

rather than implicitly raise NPEs.

This ensures that invocations which catch and swallow IOEs will perform as

expected.

abfs:

When rethrowing an IOException in the close() call, explicitly wrap it

with a new instance of the same subclass.

This is needed to handle failures in try-with-resources clauses, where

any exception in closed() is added as a suppressed exception to the one

thrown in the try {} clause

*and you cannot attach the same exception to itself*

Contributed by Steve Loughran.

Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b

  1. … 5 more files in changeset.
HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase (#1757)

Contributed by Mingliang Liu.

Signed-off-by: Steve Loughran <stevel@apache.org>

  1. … 1 more file in changeset.
HADOOP-16665. Filesystems to be closed if they failed during initialize().

Contributed by Steve Loughran.

This FileSystem instantiation so if an IOException or RuntimeException is

raised in the invocation of FileSystem.initialize() then a best-effort

attempt is made to close the FS instance; exceptions raised that there

are swallowed.

The S3AFileSystem is also modified to do its own cleanup if an

IOException is raised during its initialize() process, it being the

FS we know has the "potential" to leak threads, especially in

extension points (e.g AWS Authenticators) which spawn threads.

Change-Id: Ib84073a606c9d53bf53cbfca4629876a03894f04

    • -0
    • +110
    ./AbstractHadoopTestBase.java
  1. … 10 more files in changeset.
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager. (#1634)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

(cherry picked from commit e40e2d6ad5cbe782c3a067229270738b501ed27e)

Conflicts:

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/VolumeManagerImpl.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/lifecycle/VolumeImpl.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterServiceTestBase.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterServiceInterceptor.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java

  1. … 206 more files in changeset.
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3

Contributed by Steve Loughran.

Includes

-S3A glob scans don't bother trying to resolve symlinks

-stack traces don't get lost in getFileStatuses() when exceptions are wrapped

-debug level logging of what is up in Globber

-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.

-ITestRestrictedReadAccess tests incomplete read access to files.

This adds a builder API for constructing globbers which other stores can use

so that they too can skip symlink resolution when not needed.

Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7

  1. … 12 more files in changeset.
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager.

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

  1. … 210 more files in changeset.
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may

options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,

as the existing input format/record readers can't handle S3 select output where

the stream is shorter than the file length, and splitting plain text is suboptimal.

Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific

configuration parameters which can be set in jobs and used to set filesystem input stream

options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran

  1. … 70 more files in changeset.
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.

  1. … 334 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran and Daryn Sharp.

  1. … 101 more files in changeset.
Revert "HADOOP-14556. S3A to support Delegation Tokens."

This reverts commit d7152332b32a575c3a92e3f4c44b95e58462528d.

  1. … 104 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran.

  1. … 104 more files in changeset.
HADOOP-14624. Add GenericTestUtils.DelayAnswer that accept slf4j logger API. Contributed by Ian Pickering and Wenxin He.

  1. … 12 more files in changeset.
HADOOP-16625: Backport HADOOP-14624 to branch-3.1.

(cherry picked from commit 79c97f6a0bebc95ff81a8ef9b07d3619f05ed583)

Conflicts:

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java

  1. … 13 more files in changeset.
HADOOP-15316. GenericTestUtils can exceed maxSleepTime. Contributed by Adam Antal.

HADOOP-15495. Upgrade commons-lang version to 3.7 in hadoop-common-project and hadoop-tools. Contributed by Takanobu Asanuma.

  1. … 101 more files in changeset.
Revert "HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun."

This reverts commit c83b1fc668d30045cca23e1cbe00c9accb30d644.

  1. … 4 more files in changeset.
HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun.

(cherry picked from commit 8e7548d33be9c4874daab18b2e774bdc2ed216d3)

  1. … 4 more files in changeset.
HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun.

(cherry picked from commit 8e7548d33be9c4874daab18b2e774bdc2ed216d3)

  1. … 4 more files in changeset.
HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun.

  1. … 4 more files in changeset.
HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun.

(cherry picked from commit 8e7548d33be9c4874daab18b2e774bdc2ed216d3)

  1. … 4 more files in changeset.
YARN-8388. TestCGroupElasticMemoryController.testNormalExit() hangs on Linux. (Miklos Szegedi via Haibo Chen)

  1. … 1 more file in changeset.
YARN-8375. TestCGroupElasticMemoryController fails surefire build. (Miklos Szegedi via Haibo Chen)

  1. … 1 more file in changeset.
HADOOP-14188. Remove the usage of org.mockito.internal.util.reflection.Whitebox

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

  1. … 53 more files in changeset.
Revert "Revert "Merge branch 'trunk' into HDFS-7240"" After testing it was confirmed that these changes work as expected.

This reverts commit 7a542fb3270953fff039c9b1bd7ba7afa35a842c.

  1. … 1226 more files in changeset.
Revert "Merge branch 'trunk' into HDFS-7240"

This reverts commit 78d94acdced8e04252d0288b495d41371b1fb7b4, reversing

changes made to e0262147d60524d5718b2bfed48391a876dc3662.

  1. … 1226 more files in changeset.
HDFS-13296. GenericTestUtils generates paths with drive letter in Windows and fail webhdfs related test cases. Contributed by Xiao Liang.

HDFS-13296. GenericTestUtils generates paths with drive letter in Windows and fail webhdfs related test cases. Contributed by Xiao Liang.

(cherry picked from commit e65ff1c8be48ef4f04ed96f96ac4caef4974944d)

HDFS-13296. GenericTestUtils generates paths with drive letter in Windows and fail webhdfs related test cases. Contributed by Xiao Liang.

(cherry picked from commit e65ff1c8be48ef4f04ed96f96ac4caef4974944d)

HDFS-13296. GenericTestUtils generates paths with drive letter in Windows and fail webhdfs related test cases. Contributed by Xiao Liang.

(cherry picked from commit e65ff1c8be48ef4f04ed96f96ac4caef4974944d)