hadoop

Clone Tools
  • last updated 26 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HDDS-1358 : Recon Server REST API not working as expected. (#668)

HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685)

This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:

* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests

* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?

* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads

* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:

* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3

HADOOP-16011. OsSecureRandom very slow compared to other SecureRandom implementations. Contributed by Siyao Meng.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685)

This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:

* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests

* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?

* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads

* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:

* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3

HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685)

This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:

* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests

* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?

* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads

* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:

* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3

(cherry picked from commit 366186d9990ef9059b6ac9a19ad24310d6f36d04)

HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685)

This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:

* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests

* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?

* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads

* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:

* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3

HDDS-1330 : Add a docker compose for Ozone deployment with Recon. (#669)

    • -0
    • +17
    /hadoop-ozone/dist/src/main/compose/ozone-recon/.env
    • -0
    • +80
    /hadoop-ozone/dist/src/main/compose/ozone-recon/docker-config
HDDS-1377. OM failed to start with incorrect hostname set as ip address in CSR. (#683)

HDDS-1377. OM failed to start with incorrect hostname set as ip address in CSR. (#683)

(cherry picked from commit d6c233fce67104c1c4da802eb695526e60058536)

HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang.

(cherry picked from commit be488b6070a124234c77f16193ee925d32ca9a20)

(cherry picked from commit c8703dda0727e17d759d7ad27f0caee88103a530)

(cherry picked from commit 2a94603ae66d9000c0bb07df0d592279339af103)

HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang.

(cherry picked from commit be488b6070a124234c77f16193ee925d32ca9a20)

HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang.

HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang.

(cherry picked from commit be488b6070a124234c77f16193ee925d32ca9a20)

(cherry picked from commit c8703dda0727e17d759d7ad27f0caee88103a530)

HDFS-14394: Add -std=c99 / -std=gnu99 to libhdfs compile flags

Signed-off-by: Todd Lipcon <todd@apache.org>

HDDS-1164. Add New blockade Tests to test Replica Manager. Contributed by Nilotpal Nandi.

    • -87
    • +168
    /hadoop-ozone/dist/src/main/blockade/test_blockade_mixed_failure_three_nodes_isolate.py
YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko.

HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji.

HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji.

(cherry picked from commit 8b6deebb1dda49e5e35180ed5c5fb5b5221c1516)

HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji.

(cherry picked from commit 8b6deebb1dda49e5e35180ed5c5fb5b5221c1516)

HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji.

(cherry picked from commit 8b6deebb1dda49e5e35180ed5c5fb5b5221c1516)

(cherry picked from commit 875435dc7d6a7cb7925a25ca3347a7ebaabec595)

HDFS-12245. Fix INodeId javadoc

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

HDDS-1365. Fix error handling in KeyValueContainerCheck. Contributed by Supratim Deka.

HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

HDDS-1153. Make tracing instrumentation configurable

Closes #673

upgrade

    • -0
    • +5
    /hadoop-ozone/common/src/main/bin/ozone
    • -0
    • +54
    /hadoop-ozone/upgrade/pom.xml
HDDS-1337. Handle GroupMismatchException in OzoneClient. Contributed by Shashikant Banerjee.

HDDS-1337. Handle GroupMismatchException in OzoneClient. Contributed by Shashikant Banerjee.

(cherry picked from commit d31c86892e0ceec5d642f76fc9123fac4fd80db8)

HADOOP-16218. Findbugs warning of null param to non-nullable method in Configuration with Guava update. (#655)

Change-Id: I461e518ce9a4730b91a8138ad55b39e9a4b0a4b8

HADOOP-16227. Upgrade checkstyle to 8.19

(cherry picked from commit 4b4fef2f0e0ed1e185ea1058db7a65d68d4970b9)

(cherry picked from commit b45813b25915bec19ff3e31add56c6fef7cf6e15)

HADOOP-16227. Upgrade checkstyle to 8.19

(cherry picked from commit 4b4fef2f0e0ed1e185ea1058db7a65d68d4970b9)