Clone Tools
  • last updated 27 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena

  1. … 9 more files in changeset.
YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko

  1. … 2 more files in changeset.
YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko

  1. … 2 more files in changeset.
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3

Contributed by Steve Loughran.

Includes

-S3A glob scans don't bother trying to resolve symlinks

-stack traces don't get lost in getFileStatuses() when exceptions are wrapped

-debug level logging of what is up in Globber

-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.

-ITestRestrictedReadAccess tests incomplete read access to files.

This adds a builder API for constructing globbers which other stores can use

so that they too can skip symlink resolution when not needed.

Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7

  1. … 11 more files in changeset.
HDFS-14564: Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable (#963) Contributed by Sahil Takiar.

Reviewed-by: Siyao Meng <smeng@cloudera.com>

    • -0
    • +24
    ./fs/ByteBufferPositionedReadable.java
  1. … 13 more files in changeset.
HDFS-14874. Fix TestHDFSCLI and TestDFSShell test break because of logging change in mkdir (#1522). Contributed by Gabor Bota.

HADOOP-15691 Add PathCapabilities to FileSystem and FileContext.

Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client

to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.

* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,

file systems are free to add their own capabilities, prefixed with

fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;

It is more a check of client API and the client's configuration/knowledge

of the state of the remote system.

* Permissions are not checked.

Change-Id: I80bfebe94f4a8bdad8f3ac055495735b824968f5

    • -0
    • +126
    ./fs/CommonPathCapabilities.java
    • -0
    • +61
    ./fs/PathCapabilities.java
    • -0
    • +18
    ./fs/http/AbstractHttpFileSystem.java
    • -0
    • +98
    ./fs/impl/FsLinkResolution.java
    • -0
    • +50
    ./fs/impl/PathCapabilitiesSupport.java
  1. … 25 more files in changeset.
YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko

  1. … 2 more files in changeset.
HADOOP-16138. hadoop fs mkdir / of nonexistent abfs container raises NPE (#1302). Contributed by Gabor Bota.

Change-Id: I2f637865c871e400b95fe7ddaa24bf99fa192023

  1. … 1 more file in changeset.
HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He.

(cherry picked from commit 659c88801d008bb352d10a1cb3bd0e401486cc9b)

  1. … 5 more files in changeset.
HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He.

  1. … 5 more files in changeset.
HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He.

(cherry picked from commit 659c88801d008bb352d10a1cb3bd0e401486cc9b)

  1. … 5 more files in changeset.
HADOOP-16581. Revise ValueQueue to correctly replenish queues that go below the watermark (#1463)

(cherry picked from dd0834696a694564af65a1355c9d13275f44df51)

  1. … 1 more file in changeset.
HADOOP-16581. Revise ValueQueue to correctly replenish queues that go below the watermark (#1463)

In the existing implementation, the ValueQueue::getAtMost() method will only trigger a refill on a key queue if it has gone empty, instead of triggering a refill when it has gone below the watermark. Revise the test suite to correctly verify this behavior.

  1. … 1 more file in changeset.
HADOOP-16581. Revise ValueQueue to correctly replenish queues that go below the watermark (#1463)

(cherry picked from dd0834696a694564af65a1355c9d13275f44df51)

(cherry picked from e31594f20ed43670a9ff1463a752d8b8e4b3b6f0)

  1. … 1 more file in changeset.
HADOOP-16581. Revise ValueQueue to correctly replenish queues that go below the watermark (#1463)

(cherry picked from dd0834696a694564af65a1355c9d13275f44df51)

(cherry picked from e31594f20ed43670a9ff1463a752d8b8e4b3b6f0)

(cherry picked from b091abc471e4dbeda697732809f0ee3db2622917)

  1. … 1 more file in changeset.
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 (#1432)

HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.

  1. … 17 more files in changeset.
HADOOP-16069. Support configure ZK_DTSM_ZK_KERBEROS_PRINCIPAL in ZKDelegationTokenSecretManager using principal with Schema /_HOST. Contributed by luhuachao.

(cherry picked from commit 298f07abceb4bb854630c9058e1697d4ef3d2ae2)

HADOOP-16069. Support configure ZK_DTSM_ZK_KERBEROS_PRINCIPAL in ZKDelegationTokenSecretManager using principal with Schema /_HOST. Contributed by luhuachao.

(cherry picked from commit 298f07abceb4bb854630c9058e1697d4ef3d2ae2)

HADOOP-16069. Support configure ZK_DTSM_ZK_KERBEROS_PRINCIPAL in ZKDelegationTokenSecretManager using principal with Schema /_HOST. Contributed by luhuachao.

HADOOP-16582. LocalFileSystem's mkdirs() does not work as expected under viewfs. Contributed by Kihwal Lee

(cherry picked from commit d4205dce176287e863f567b333e0d408bf51ae6d)

  1. … 1 more file in changeset.
HADOOP-16582. LocalFileSystem's mkdirs() does not work as expected under viewfs. Contributed by Kihwal Lee

(cherry picked from commit d4205dce176287e863f567b333e0d408bf51ae6d)

  1. … 1 more file in changeset.
HADOOP-16582. LocalFileSystem's mkdirs() does not work as expected under viewfs. Contributed by Kihwal Lee

(cherry picked from commit d4205dce176287e863f567b333e0d408bf51ae6d)

  1. … 1 more file in changeset.
HADOOP-16582. LocalFileSystem's mkdirs() does not work as expected under viewfs. Contributed by Kihwal Lee

(cherry picked from commit d4205dce176287e863f567b333e0d408bf51ae6d)

  1. … 1 more file in changeset.
HADOOP-16582. LocalFileSystem's mkdirs() does not work as expected under viewfs. Contributed by Kihwal Lee

(cherry picked from commit d4205dce176287e863f567b333e0d408bf51ae6d)

  1. … 1 more file in changeset.
HADOOP-16582. LocalFileSystem's mkdirs() does not work as expected under viewfs. Contributed by Kihwal Lee

  1. … 1 more file in changeset.
HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8.

Contributed by Sahil Takiar.

This moves the SSLSocketFactoryEx class from hadoop-azure into hadoop-common

as the DelegatingSSLSocketFactory and binds the S3A connector to it so that

it can avoid using those HTTPS algorithms which are underperformant on Java 8.

Change-Id: Ie9e6ac24deac1aa05e136e08899620efa7d22abd

    • -0
    • +287
    ./security/ssl/DelegatingSSLSocketFactory.java
  1. … 15 more files in changeset.
HADOOP-16490. Avoid/handle cached 404s during S3A file creation.

Contributed by Steve Loughran.

This patch avoids issuing any HEAD path request when creating a file with overwrite=true,

so 404s will not end up in the S3 load balancers unless someone calls getFileStatus/exists/isFile

in their own code.

The Hadoop FsShell CommandWithDestination class is modified to not register uncreated files

for deleteOnExit(), because that calls exists() and so can place the 404 in the cache, even

after S3A is patched to not do it itself.

Because S3Guard knows when a file should be present, it adds a special FileNotFound retry policy

independently configurable from other retry policies; it is also exponential, but with

different parameters. This is because every HEAD request will refresh any 404 cached in

the S3 Load Balancers. It's not enough to retry: we have to have a suitable gap between

attempts to (hopefully) ensure any cached entry wil be gone.

The options and values are:

fs.s3a.s3guard.consistency.retry.interval: 2s

fs.s3a.s3guard.consistency.retry.limit: 7

The S3A copy() method used during rename() raises a RemoteFileChangedException which is not caught

so not downgraded to false. Thus: when a rename is unrecoverable, this fact is propagated.

Copy operations without S3Guard lack the confidence that the file exists, so don't retry the same way:

it will fail fast with a different error message. However, because create(path, overwrite=false) no

longer does HEAD path, we can at least be confident that S3A itself is not creating those cached

404 markers.

Change-Id: Ia7807faad8b9a8546836cb19f816cccf17cca26d

    • -31
    • +43
    ./fs/shell/CommandWithDestination.java
  1. … 23 more files in changeset.
HADOOP-16549. Remove Unsupported SSL/TLS Versions from Docs/Properties. Contributed by Daisuke Kobayashi.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

Reviewed-by: Akira Ajisaka <aajisaka@apache.org>

  1. … 2 more files in changeset.
HADOOP-16255. Add ChecksumFs.rename(path, path, boolean)

Contributed by Jungtaek Lim

Change-Id: If00a4d7d30456c08eb2b0f7e2b242197bc4ee05d

  1. … 1 more file in changeset.