hadoop

Clone Tools
  • last updated 16 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
address review comments

HDDS-2252. Enable gdpr robot test in daily build

Closes #1602

HDDS-2169. Avoid buffer copies while submitting client requests in Ratis. Contributed by Tsz-wo Sze(#1517).

Revert "HADOOP-16579. Upgrade to Apache Curator 4.2.0 excluding ZK (#1531). Contributed by Norbert Kalmár."

This reverts commit 10bdc592d6157519166c262addecbff23b4232de.

HADOOP-16626. S3A ITestRestrictedReadAccess fails without S3Guard.

Contributed by Steve Loughran.

Change-Id: Ife730b80057ddd43e919438cb5b2abbda990e636

HDDS-2251. Add an option to customize unit.sh and integration.sh parameters

Closes #1598

    • -1
    • +1
    /hadoop-ozone/dev-support/checks/integration.sh
    • -1
    • +1
    /hadoop-ozone/dev-support/checks/unit.sh
Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph"

This reverts commit 4510970e2f7728d036c750b596985e5ffa357b60.

HDDS-2257. Fix checkstyle issues in ChecksumByteBuffer (#1603)

HDDS-2250. Generated configs missing from ozone-filesystem-lib jars

Closes #1597

    • -0
    • +3
    /hadoop-ozone/ozonefs-lib-current/pom.xml
HDDS-2158. Fixing Json Injection Issue in JsonUtils. (#1486)

  1. … 3 more files in changeset.
HDDS-2164 : om.db.checkpoints is getting filling up fast. (#1536)

HADOOP-16579. Upgrade to Apache Curator 4.2.0 excluding ZK (#1531). Contributed by Norbert Kalmár.

HADOOP-16570. S3A committers encounter scale issues.

Contributed by Steve Loughran.

This addresses two scale issues which has surfaced in large scale benchmarks

of the S3A Committers.

* Thread pools are not cleaned up.

This now happens, with tests.

* OOM on job commit for jobs with many thousands of tasks,

each generating tens of (very large) files.

Instead of loading all pending commits into memory as a single list, the list

of files to load is the sole list which is passed around; .pendingset files are

loaded and processed in isolation -and reloaded if necessary for any

abort/rollback operation.

The parallel commit/abort/revert operations now work at the .pendingset level,

rather than that of individual pending commit files. The existing parallelized

Tasks API is still used to commit those files, but with a null thread pool, so

as to serialize the operations.

Change-Id: I5c8240cd31800eaa83d112358770ca0eb2bca797

  1. … 7 more files in changeset.
HDFS-14890. Fixed namenode and journalnode startup on Windows. Contributed by Siddharth Wagle

HDDS-2237. KeyDeletingService throws NPE if it's started too early (#1584)

YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph

HDDS-2230. Invalid entries in ozonesecure-mr config. (Addendum)

HDDS-2222. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C. (#1595)

HADOOP-16207 Improved S3A MR tests.

Contributed by Steve Loughran.

Replaces the committer-specific terasort and MR test jobs with parameterization

of the (now single tests) and use of file:// over hdfs:// as the cluster FS.

The parameterization ensures that only one of the specific committer tests

run at a time -overloads of the test machines are less likely, and so the

suites can be pulled back into the parallel phase.

There's also more detailed validation of the stage outputs of the terasorting;

if one test fails the rest are all skipped. This and the fact that job

output is stored under target/yarn-${timestamp} means failures should

be more debuggable.

Change-Id: Iefa370ba73c6419496e6e69dd6673d00f37ff095

HDDS-2199. In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host

Closes #1551

HDDS-2140. Add robot test for GDPR feature

Closes #1542

    • -0
    • +89
    /hadoop-ozone/dist/src/main/smoketest/gdpr/gdpr.robot
Revert "HDDS-2222 (#1578)" (#1594)

This reverts commit 4cf0b3660f620dd8a67201b75f4c88492c9adfb3.

HDDS-2230. Invalid entries in ozonesecure-mr config

Doroszlai, Attila <adoroszlai@apache.org>

HDDS-2222 (#1578)

Thanks @jnp for reviewing this. Merging now.

YARN-9782. Avoid DNS resolution while running SLS. Contributed by Abhishek Modi.

HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina

HDDS-2225. SCM fails to start in most unsecure environments due to leftover secure config

Closes #1568

HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell.

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

HDFS-14889. Ability to check if a block has a replica on provided storage. Contributed by Ashvin Agrawal. (#1573)"

HDDS-2223. Support ReadWrite lock in LockManager. (#1564)