hadoop

Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16582. LocalFileSystem's mkdirs() does not work as expected under viewfs. Contributed by Kihwal Lee

(cherry picked from commit d4205dce176287e863f567b333e0d408bf51ae6d)

HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien.

(cherry picked from commit 1843c4688a3040c48f6834d166494cbf68b5e4fd)

Conflicts:

LICENSE-binary

(cherry picked from commit d39ebbfebc9da982cadfbe92f3fb6677e744622d)

(cherry picked from commit 7d7b415409901bbb0095af4e9d56aade158c1438)

(cherry picked from commit 699297e0e6178bb802d86fe0f4b269fa35e717c7)

Revert "HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun."

This reverts commit 12ec7c301ff19b004fd94e3f0af7238f8050bccc.

HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun.

(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)

HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun.

(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)

YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06).

(cherry picked from commit d562050cec83a2bc2ffb6d109ed3d64b394b870d)

HDFS-14725. Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks until next report). Contributed by He Xiaoqiao.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow."

This reverts commit dbc7e4d74b3494e049bef6baf16efc88dca30fc3.

HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai.

(cherry picked from commit bdab03cf711f75e88f80c21b3193fd5070cb6364)

(cherry picked from commit 0e7afe054ad67052359ab62757d6241d955b94d5)

HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from commit ab7509efe7375949db3b819b53e9461624cd2b1e)

(cherry picked from commit 8a924a81fa4140ab99468eda1452915ea60b6ce0)

Conflicts:

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java

HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow.

(cherry picked from commit 8755de108d0e711cbfa784a20bd89e28fe79266b)

(cherry picked from commit 22321f1d3dfae5bf03b8724f5d74d292d551580f)

HDFS-14723. Add helper method FSNamesystem#setBlockManagerForTesting() in branch-2. Contributed by Wei-Chiu Chuang.

Reviewed-by: Chao Sun <sunchao@apache.org>

Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>

Reviewed-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit f27d040ce329894d14ce7caf42a64816df0a9455)

Conflicts:

hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan.

YARN-8045. Reduce log output from container status calls. Contributed by Craig Condit

HADOOP-16494. Add SHA-512 checksum to release artifact to comply with the release distribution policy.

This closes #1243

(cherry picked from commit 34dd9ee36674be670013d4fc3d9b8f5b36886812)

(cherry picked from commit 786df37429b984c21d0e855f969cf3ec04b1e8cb)

YARN-9596: QueueMetrics has incorrect metrics when labelled partitions are involved. Contributed by Muhammad Samir Khan.

HADOOP-16435. RpcMetrics should not retained forever. Contributed by Zoltan Haindrich.

(cherry picked from commit cbfa3f3e988b8a3f142abadfe4f18201b6438ac9)

(cherry picked from commit 46c00aca2176400d1ba77f8af7756303b5826888)

YARN-9702: Backport YARN-5788 to 2.8: Apps not activiated and AM limit resource in UI and REST not updated after -replaceLabelsOnNode. Backport contributed by Muhammad Samir Khan

HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang.

(cherry picked from commit 6c8cfd135eea15e3dfd76dd33af24a9a8c38fd8d)

(cherry picked from commit 3f112e7347fa7b75eab3a417c9d6520ca8b3053e)

HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Contributed by Greg Senia, Ajay Kumar.

HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu.

(cherry picked from commit bf63475e7a72ff7346226a7dc8dcb4d6fc534163)

(cherry picked from commit 2ddd9878a03ded8a8c881505a2617a4e63775a5c)

HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>

(cherry picked from commit 7c00756aff6bf8d5e1d05748ba32b679e397b53f)

(cherry picked from commit 4ea3b04bf36af56612c6b47d9ff8aa49cbbbcc9e)

(cherry picked from commit 76faa41f18f1cb20a6469f53b475f52b662f8107)

(cherry picked from commit a68de439577715e960ed776c5172fd1fbb0506f7)

(cherry picked from commit 1a5e40f802a932f45b1fca1524c88cbba40f8cd6)

HDFS-13770. dfsadmin -report does not always decrease "missing blocks (with replication factor 1)" metrics when file is deleted. Contributed by Kitti Nanasi.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from commit abd11f9db1aaffeb864cca9f336f3d169d264836)

(cherry picked from commit 39e47dacaef4a8908ff43975957cb8fe3eb29db2)

HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu.

(cherry picked from commit 62ad9885ea8c75c134de43a3a925c76b253658e1)

(cherry picked from commit 45543ffd6af723bb83bd44a99f12165363c70abb)

(cherry picked from commit fc8983a5b8c94746d3e58cc203974aa09c1b640b)

(cherry picked from commit 79f8e4d5bc3afaff2f352b918e96a6ef3c2abcce)

(cherry picked from commit ce49f8a5df8b63c9bb2e50b4ad25a6bffd9e54de)

(cherry picked from commit 624fd10c38e15001ded7cef7f04536855832827a)

HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from commit a95e87d827381a7f87c01f0befc2475f9478e63b)

(cherry picked from commit f68099aa1316f34591c757bcf4198a3a3c7f691a)

(cherry picked from commit 3d1d8623a11b72c7939d6500537be5fd6565d4d6)

(cherry picked from commit 010a99104b53a23826f8f1f73d0125cb26f2adca)

(cherry picked from commit 73a59b2c42e000e2e699133b774dc17240f62886)

(cherry picked from commit c9af5fb43039997474d900f95abe252f0ab6c140)

HADOOP-16053. Backport HADOOP-14816 to branch-2. Contributed by Akira Ajisaka.

    • -82
    • +161
    /dev-support/docker/Dockerfile
YARN-8625. Aggregate Resource Allocation for each job is not present in ATS. Contributed by Prabhu Joseph.

HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read

(cherry picked from commit ea3b0a184405c3feca024a560807ea215b6858b9)

(cherry picked from commit 9b61ecfcbeeb1d3c320f775b43c68d3e39a271a3)

(cherry picked from commit b082628e5a39fc1bc9d6094beb9c253cc096c701)

(cherry picked from commit 2b9c4fccf4a8930bfff99ab4928b9b3b4980d3b2)

(cherry picked from commit 8b90c69357cf515661ec8a08e61ddd03e884b3ab)

(cherry picked from commit d14fd913f46fc8e794c39f4cc44d1674aa156ee0)

HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee

(cherry picked from commit 9122b9b6493d315c8910fbc06eaba96273ee335c)

(cherry picked from commit fd8b8c7494d19467485423ce2f3f7459a4fddc29)

HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from commit c1caab40f27e3e4f58ff1b5ef3e93efc56bbecbe)

(cherry picked from commit 2cd7577a5a58db42a14a659b85fbd475091d0cb2)

(cherry picked from commit 90cc5ddd4957fcb27ef5a1e7ee1a67c4eca9a3f5)

(cherry picked from commit 942c5b04f2f7bc33595c4df5c66104ce3c4c2c69)

Conflicts:

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java

(cherry picked from commit c63049dfd413dd790e9cc6cca13c310140edc260)

Conflicts:

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java