hadoop-tools

Clone Tools
  • last updated 29 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16676. Backport HADOOP-16152 to branch-3.2. Contributed by Siyao Meng.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

  1. … 8 more files in changeset.
HADOOP-16665. Filesystems to be closed if they failed during initialize().

Contributed by Steve Loughran.

This FileSystem instantiation so if an IOException or RuntimeException is

raised in the invocation of FileSystem.initialize() then a best-effort

attempt is made to close the FS instance; exceptions raised that there

are swallowed.

The S3AFileSystem is also modified to do its own cleanup if an

IOException is raised during its initialize() process, it being the

FS we know has the "potential" to leak threads, especially in

extension points (e.g AWS Authenticators) which spawn threads.

Change-Id: Ib84073a606c9d53bf53cbfca4629876a03894f04

  1. … 4 more files in changeset.
HADOOP-16477. S3A delegation token tests fail if fs.s3a.encryption.key set.

Contributed by Steve Loughran.

Change-Id: I843989f32472bbdefbd4fa504b26c7a614ab1cee

HADOOP-16681. mvn javadoc:javadoc fails in hadoop-aws. Contributed by Xieming Li

HADOOP-16484. S3A to warn or fail if S3Guard is disabled (#1661). Contributed by Gabor Bota.

  1. … 1 more file in changeset.
HDFS-14824. [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results. (#1685)

HDFS-14907. [Dynamometer] Add JUnit JAR to classpath for SimulatedDataNodes (#1680)

HADOOP-16653. S3Guard DDB overreacts to no tag access (#1660). Contributed by Gabor Bota.

HDFS-14638. [Dynamometer] Fix scripts to refer to current build structure. (#1673)

HADOOP-16658. S3A connector does not support including the token renewer in the token identifier.

Contributed by Phil Zampino.

Change-Id: Iea9d5028dcf58bda4da985604f5cd3ac283619bd

HADOOP-16152. Upgrade Eclipse Jetty version to 9.4.x. Contributed by Yuming Wang, Siyao Meng.

Co-authored-By: Siyao Meng <smeng@cloudera.com>

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

  1. … 9 more files in changeset.
HADOOP-16652. Backport of HADOOP-16587: Make ABFS AAD endpoints configurable

HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI

(cherry picked from commit 9a8edb0aeddd7787b2654f6e2a8465c325e048a2)

HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI

HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI

(cherry picked from commit 9a8edb0aeddd7787b2654f6e2a8465c325e048a2)

HADOOP-16478. S3Guard bucket-info fails if the caller lacks s3:GetBucketLocation.

Contributed by Steve Loughran.

Includes HADOOP-16651. S3 getBucketLocation() can return "US" for us-east.

Change-Id: Ifc0dca76e51495ed1a8fc0f077b86bf125deff40

Update version to 2.10.0

  1. … 69 more files in changeset.
Preparing for 2.10.1 development

  1. … 69 more files in changeset.
Preparing for 2.11.0 development

  1. … 69 more files in changeset.
HDFS-14856. Fetch file ACLs while mounting external store. (#1478)

  1. … 2 more files in changeset.
HADOOP-16635. S3A "directories only" scan still does a HEAD.

Contributed by Steve Loughran.

Change-Id: I5e41d7f721364c392e1f4344db83dfa8c5aa06ce

Revert "HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos."

This reverts commit 7a4b3d42c4e36e468c2a46fd48036a6fed547853.

The patch broke TestRouterWebHDFSContractSeek as it turns out that

WebHDFSInputStream.available() is always 0.

  1. … 2 more files in changeset.
YARN-9836. General usability improvements in showSimulationTrace.html. Contributed by Adam Antal

    • -243
    • +264
    ./hadoop-sls/src/main/html/showSimulationTrace.html
HADOOP-16520. Race condition in DDB table init and waiting threads. (#1576). Contributed by Gabor Bota.

Fixes HADOOP-16349. DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry

Change-Id: Ia83e92b9039ccb780090c99c41b4f71ef7539d35

HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos.

Contributed by lqjacklee.

Change-Id: I32bb00a683102e7ff8ff8ce0b8d9c3195ca7381c

  1. … 2 more files in changeset.
HADOOP-16650. ITestS3AClosedFS failing.

Contributed by Steve Loughran.

Change-Id: Ia9bb84bd6455e210a54cfe9eb944feeda8b58da9

HADOOP-16630 : Backport of Hadoop-16548 : Disable Flush() over config

    • -0
    • +18
    ./hadoop-azure/src/site/markdown/abfs.md
HADOOP-16587. Make ABFS AAD endpoints configurable.

Contributed by Bilahari T H.

This also addresses HADOOP-16498: AzureADAuthenticator cannot authenticate

in China.

Change-Id: I2441dd48b50b59b912b0242f7f5a4418cf94a87c

HADOOP-16626. S3A ITestRestrictedReadAccess fails without S3Guard.

Contributed by Steve Loughran.

Change-Id: Ife730b80057ddd43e919438cb5b2abbda990e636

HADOOP-16570. S3A committers encounter scale issues.

Contributed by Steve Loughran.

This addresses two scale issues which has surfaced in large scale benchmarks

of the S3A Committers.

* Thread pools are not cleaned up.

This now happens, with tests.

* OOM on job commit for jobs with many thousands of tasks,

each generating tens of (very large) files.

Instead of loading all pending commits into memory as a single list, the list

of files to load is the sole list which is passed around; .pendingset files are

loaded and processed in isolation -and reloaded if necessary for any

abort/rollback operation.

The parallel commit/abort/revert operations now work at the .pendingset level,

rather than that of individual pending commit files. The existing parallelized

Tasks API is still used to commit those files, but with a null thread pool, so

as to serialize the operations.

Change-Id: I5c8240cd31800eaa83d112358770ca0eb2bca797

  1. … 7 more files in changeset.