hadoop

Clone Tools
  • last updated 17 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16587. Make ABFS AAD endpoints configurable.

Contributed by Bilahari T H.

This also addresses HADOOP-16498: AzureADAuthenticator cannot authenticate

in China.

Change-Id: I2441dd48b50b59b912b0242f7f5a4418cf94a87c

HDDS-2264. Improve output of TestOzoneContainer

Signed-off-by: Anu Engineer <aengineer@apache.org>

address review comments

HDDS-2262. SLEEP_SECONDS: command not found

Signed-off-by: Anu Engineer <aengineer@apache.org>

HDDS-2252. Enable gdpr robot test in daily build

Closes #1602

HDDS-2169. Avoid buffer copies while submitting client requests in Ratis. Contributed by Tsz-wo Sze(#1517).

YARN-9873. Version Number for each Scheduler Config Change. Contributed by Prabhu Joseph.

(cherry picked from commit be901f49628a553cfe6a3942478cb528cce2b266)

HDDS-2259. Container Data Scrubber computes wrong checksum

Signed-off-by: Anu Engineer <aengineer@apache.org>

Revert "HADOOP-16579. Upgrade to Apache Curator 4.2.0 excluding ZK (#1531). Contributed by Norbert Kalmár."

This reverts commit 10bdc592d6157519166c262addecbff23b4232de.

HADOOP-16626. S3A ITestRestrictedReadAccess fails without S3Guard.

Contributed by Steve Loughran.

Change-Id: Ife730b80057ddd43e919438cb5b2abbda990e636

HDDS-2251. Add an option to customize unit.sh and integration.sh parameters

Closes #1598

    • -1
    • +1
    /hadoop-ozone/dev-support/checks/integration.sh
    • -1
    • +1
    /hadoop-ozone/dev-support/checks/unit.sh
Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph"

This reverts commit 3a0afcfb7f196bee8e534a3e80211bfa077b2aee.

Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph"

This reverts commit 4510970e2f7728d036c750b596985e5ffa357b60.

HDDS-2257. Fix checkstyle issues in ChecksumByteBuffer (#1603)

HDDS-2250. Generated configs missing from ozone-filesystem-lib jars

Closes #1597

    • -0
    • +3
    /hadoop-ozone/ozonefs-lib-current/pom.xml
HDDS-2158. Fixing Json Injection Issue in JsonUtils. (#1486)

  1. … 3 more files in changeset.
HDDS-2164 : om.db.checkpoints is getting filling up fast. (#1536)

HADOOP-16579. Upgrade to Apache Curator 4.2.0 excluding ZK (#1531). Contributed by Norbert Kalmár.

HADOOP-16570. S3A committers encounter scale issues.

Contributed by Steve Loughran.

This addresses two scale issues which has surfaced in large scale benchmarks

of the S3A Committers.

* Thread pools are not cleaned up.

This now happens, with tests.

* OOM on job commit for jobs with many thousands of tasks,

each generating tens of (very large) files.

Instead of loading all pending commits into memory as a single list, the list

of files to load is the sole list which is passed around; .pendingset files are

loaded and processed in isolation -and reloaded if necessary for any

abort/rollback operation.

The parallel commit/abort/revert operations now work at the .pendingset level,

rather than that of individual pending commit files. The existing parallelized

Tasks API is still used to commit those files, but with a null thread pool, so

as to serialize the operations.

Change-Id: I5c8240cd31800eaa83d112358770ca0eb2bca797

  1. … 7 more files in changeset.
HDFS-14890. Fixed namenode and journalnode startup on Windows. Contributed by Siddharth Wagle

HDFS-14890. Fixed namenode and journalnode startup on Windows. Contributed by Siddharth Wagle

(cherry picked from commit aa24add8f0e9812d1f787efb3c40155b0fdeed9c)

HDFS-14890. Fixed namenode and journalnode startup on Windows. Contributed by Siddharth Wagle

(cherry picked from commit aa24add8f0e9812d1f787efb3c40155b0fdeed9c)

(cherry picked from commit 8bb2b00d38978859b22b892034eb3f559b820942)

HDDS-2237. KeyDeletingService throws NPE if it's started too early (#1584)

YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph

YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph

(cherry picked from commit 4510970e2f7728d036c750b596985e5ffa357b60)

HDDS-2230. Invalid entries in ozonesecure-mr config. (Addendum)

HDDS-2222. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C. (#1595)

HADOOP-16207 Improved S3A MR tests.

Contributed by Steve Loughran.

Replaces the committer-specific terasort and MR test jobs with parameterization

of the (now single tests) and use of file:// over hdfs:// as the cluster FS.

The parameterization ensures that only one of the specific committer tests

run at a time -overloads of the test machines are less likely, and so the

suites can be pulled back into the parallel phase.

There's also more detailed validation of the stage outputs of the terasorting;

if one test fails the rest are all skipped. This and the fact that job

output is stored under target/yarn-${timestamp} means failures should

be more debuggable.

Change-Id: Iefa370ba73c6419496e6e69dd6673d00f37ff095

Merge remote-tracking branch 'origin/trunk' into HDDS-1880-Decom

HDDS-2199. In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host

Closes #1551