hadoop

Clone Tools
  • last updated 28 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming.

(cherry picked from commit d1c303a49763029fffa5164295034af8e81e74a0)

HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming.

HADOOP-16562. [pb-upgrade] Update docker image to have 3.7.1 protoc executable (#1429). Contributed by Vinayakumar B.

HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log) (#1208). Contributed by Gabor Bota.

Change-Id: I6bbb331b6c0a41c61043e482b95504fda8a50596

YARN-9816. EntityGroupFSTimelineStore#scanActiveLogs fails when undesired files are present under /ats/active. Contribued by Prabhu Joseph.

HDFS-14840. Use Java Conccurent Instead of Synchronization in BlockPoolTokenSecretManager. Contributed by David Mollitor.

YARN-9819. Make TestOpportunisticContainerAllocatorAMService more resilient. Contribued by Abhishek Modi

HDDS-2106. Avoid usage of hadoop projects as parent of hdds/ozone

Closes #1423

YARN-9815 ReservationACLsTestBase fails with NPE. Contributed by Ahmed Hussein

HDDS-2075. Tracing in OzoneManager call is propagated with wrong parent (#1415)

(cherry picked from commit 64ed6b177d6b00b22d45576a8517432dc6c03348)

HDDS-2075. Tracing in OzoneManager call is propagated with wrong parent (#1415)

HADOOP-16490. Avoid/handle cached 404s during S3A file creation.

Contributed by Steve Loughran.

This patch avoids issuing any HEAD path request when creating a file with overwrite=true,

so 404s will not end up in the S3 load balancers unless someone calls getFileStatus/exists/isFile

in their own code.

The Hadoop FsShell CommandWithDestination class is modified to not register uncreated files

for deleteOnExit(), because that calls exists() and so can place the 404 in the cache, even

after S3A is patched to not do it itself.

Because S3Guard knows when a file should be present, it adds a special FileNotFound retry policy

independently configurable from other retry policies; it is also exponential, but with

different parameters. This is because every HEAD request will refresh any 404 cached in

the S3 Load Balancers. It's not enough to retry: we have to have a suitable gap between

attempts to (hopefully) ensure any cached entry wil be gone.

The options and values are:

fs.s3a.s3guard.consistency.retry.interval: 2s

fs.s3a.s3guard.consistency.retry.limit: 7

The S3A copy() method used during rename() raises a RemoteFileChangedException which is not caught

so not downgraded to false. Thus: when a rename is unrecoverable, this fact is propagated.

Copy operations without S3Guard lack the confidence that the file exists, so don't retry the same way:

it will fail fast with a different error message. However, because create(path, overwrite=false) no

longer does HEAD path, we can at least be confident that S3A itself is not creating those cached

404 markers.

Change-Id: Ia7807faad8b9a8546836cb19f816cccf17cca26d

  1. … 10 more files in changeset.
HDDS-2103. TestContainerReplication fails due to unhealthy container (#1421)

HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li

HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li

(cherry picked from commit c255333e20c9af6166db5931d70151011d540359)

HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li

(cherry picked from commit c255333e20c9af6166db5931d70151011d540359)

HADOOP-15184. Add GitHub pull request template. (#1419)

    • -0
    • +6
    /.github/pull_request_template.md
HDFS-14835. RBF: Secured Router should not run when it can't initialize DelegationTokenSecretManager. (#1414)

YARN-9824. Fall back to configured queue ordering policy class name

YARN-9824. Fall back to configured queue ordering policy class name

(cherry picked from commit f8f8598ea5e86330fa5b553963f5632da286409d)

(cherry picked from commit 1dbf87c9ff33b5e2f6cbb0ce8efd348c4314a0e2)

YARN-9824. Fall back to configured queue ordering policy class name

(cherry picked from commit f8f8598ea5e86330fa5b553963f5632da286409d)

(cherry picked from commit 1dbf87c9ff33b5e2f6cbb0ce8efd348c4314a0e2)

(cherry picked from commit d75693bd6ed4a9547f4932d41014a21de1398115)

YARN-9824. Fall back to configured queue ordering policy class name

(cherry picked from commit f8f8598ea5e86330fa5b553963f5632da286409d)

YARN-9728. Bugfix for escaping illegal xml characters for Resource Manager REST API. Contributed by Prabhu Joseph

HADOOP-16554. mvn javadoc:javadoc fails in hadoop-aws.

Contributed by Xieming Li.

Change-Id: I78e88b5b1ae4702446d2bdd3e2faa3e10b45aef0

Refactor to have PlacementPolicy and ScmCommonPolicy to be general basis. (#1395)

  1. … 8 more files in changeset.
HDDS-1571. Create an interface for pipeline placement policy to support network topologies. (#1395)

(cherry picked from commit 753fc6703a39154ed6013e44dbae572391748906)

  1. … 8 more files in changeset.
HADOOP-16542. Update commons-beanutils version to 1.9.4. Contributed by kevin su.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from commit 38c1a10024476ae78975e4dc7d27a1524722b79d)

HADOOP-16542. Update commons-beanutils version to 1.9.4. Contributed by kevin su.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from commit 38c1a10024476ae78975e4dc7d27a1524722b79d)

(cherry picked from commit f2be3e92b1fb0c29d306ec2be78732473656b359)

HADOOP-16542. Update commons-beanutils version to 1.9.4. Contributed by kevin su.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

HADOOP-16530. Update xercesImpl in branch-2.

Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>