Clone Tools
  • last updated 18 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16005. NativeAzureFileSystem does not support setXAttr.

Contributed by Clemens Wolff.

  1. … 5 more files in changeset.
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.

This hardens the wasb and abfs output streams' resilience to being invoked

in/after close().

wasb:

Explicity raise IOEs on operations invoked after close,

rather than implicitly raise NPEs.

This ensures that invocations which catch and swallow IOEs will perform as

expected.

abfs:

When rethrowing an IOException in the close() call, explicitly wrap it

with a new instance of the same subclass.

This is needed to handle failures in try-with-resources clauses, where

any exception in closed() is added as a suppressed exception to the one

thrown in the try {} clause

*and you cannot attach the same exception to itself*

Contributed by Steve Loughran.

Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b

  1. … 5 more files in changeset.
HADOOP-16699. Add verbose TRACE logging to ABFS.

Contributed by Sneha Vijayarajan,

Change-Id: Ic616a10406e6e9f11616c9cc05d8630ebbedaf65

HADOOP-16660. ABFS: Make RetryCount in ExponentialRetryPolicy Configurable.

Contributed by Sneha Vijayarajan.

    • -7
    • +4
    ./services/ExponentialRetryPolicy.java
  1. … 3 more files in changeset.
HADOOP-16455. ABFS: Implement FileSystem.access() method.

Contributed by Bilahari T H.

    • -1
    • +2
    ./constants/FileSystemConfigurations.java
  1. … 3 more files in changeset.
HADOOP-16612. Track Azure Blob File System client-perceived latency

Contributed by Jeetesh Mangwani.

This add the ability to track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency in the Hadoop ABFS driver.

The latency information is sent back to the ADLS Gen 2 REST API endpoints in the subsequent requests.

    • -330
    • +514
    ./AzureBlobFileSystemStore.java
    • -0
    • +1
    ./constants/FileSystemConfigurations.java
    • -0
    • +1
    ./constants/HttpHeaderConfigurations.java
    • -0
    • +34
    ./contracts/services/AbfsPerfLoggable.java
    • -0
    • +133
    ./services/AbfsPerfInfo.java
    • -0
    • +319
    ./services/AbfsPerfTracker.java
  1. … 5 more files in changeset.
HADOOP-16587. Make ABFS AAD endpoints configurable.

Contributed by Bilahari T H.

This also addresses HADOOP-16498: AzureADAuthenticator cannot authenticate

in China.

Change-Id: I2441dd48b50b59b912b0242f7f5a4418cf94a87c

    • -0
    • +45
    ./constants/AuthConfigurations.java
    • -3
    • +9
    ./oauth2/RefreshTokenBasedTokenProvider.java
HADOOP-16578 : Avoid FileSystem API calls when FileSystem already exists

  1. … 1 more file in changeset.
HADOOP-16548 : Disable Flush() over config

    • -0
    • +1
    ./constants/FileSystemConfigurations.java
  1. … 2 more files in changeset.
HADOOP-15691 Add PathCapabilities to FileSystem and FileContext.

Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client

to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.

* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,

file systems are free to add their own capabilities, prefixed with

fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;

It is more a check of client API and the client's configuration/knowledge

of the state of the remote system.

* Permissions are not checked.

Change-Id: I80bfebe94f4a8bdad8f3ac055495735b824968f5

  1. … 39 more files in changeset.
HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8.

Contributed by Sahil Takiar.

This moves the SSLSocketFactoryEx class from hadoop-azure into hadoop-common

as the DelegatingSSLSocketFactory and binds the S3A connector to it so that

it can avoid using those HTTPS algorithms which are underperformant on Java 8.

Change-Id: Ie9e6ac24deac1aa05e136e08899620efa7d22abd

    • -3
    • +3
    ./constants/FileSystemConfigurations.java
  1. … 11 more files in changeset.
HADOOP-16315. ABFS: transform full UPN for named user in AclStatus

Contributed by Da Zhou

Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290

  1. … 1 more file in changeset.
HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC.

Contributed by Bilahari T H

Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0

  1. … 1 more file in changeset.
HADOOP-16404. ABFS default blocksize change(256MB from 512MB)

Contributed by: Arun Singh

    • -1
    • +1
    ./constants/FileSystemConfigurations.java
  1. … 1 more file in changeset.
HADOOP-16340. ABFS driver continues to retry on IOException responses from REST operations.

Contributed by Robert Levas.

This makes the HttpException constructor protected rather than public, so it is possible

to implement custom subclasses of this exception -exceptions which will not be retried.

Change-Id: Ie8aaa23a707233c2db35948784908b6778ff3a8f

HADOOP-16376. ABFS: Override access() to no-op.

Contributed by Da Zhou.

Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0

Revert "HADOOP-16050: s3a SSL connections should use OpenSSL"

This reverts commit b067f8acaa79b1230336900a5c62ba465b2adb28.

Change-Id: I584b050a56c0e6f70b11fa3f7db00d5ac46e7dd8

    • -3
    • +3
    ./constants/FileSystemConfigurations.java
  1. … 10 more files in changeset.
HADOOP-16269. ABFS: add listFileStatus with StartFrom.

Author: Da Zhou

  1. … 3 more files in changeset.
HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream.

Contributed by Da Zhou.

HADOOP-16050: s3a SSL connections should use OpenSSL

(cherry picked from commit aebf229c175dfa19fff3b31e9e67596f6c6124fa)

    • -3
    • +3
    ./constants/FileSystemConfigurations.java
  1. … 10 more files in changeset.
HADOOP-16182. Update abfs storage back-end with "close" flag when application is done writing to a file.

Contributed by Vishwajeet Dusane.

    • -10
    • +11
    ./services/AbfsOutputStream.java
HADOOP-16174. Disable wildfly logs to the console.

Follow-on to HADOOP-15851.

Author: Denes Gerencser <dgerencser@cloudera.com>

HADOOP-16169. ABFS: Bug fix for getPathProperties.

Author: Da Zhou <da.zhou@microsoft.com>

  1. … 1 more file in changeset.
HADOOP-16068. ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store.

Contributed by Steve Loughran.

    • -0
    • +97
    ./extensions/BoundDTExtension.java
    • -0
    • +7
    ./extensions/CustomDelegationTokenManager.java
    • -0
    • +114
    ./extensions/ExtensionHelper.java
    • -13
    • +109
    ./oauth2/AzureADAuthenticator.java
    • -2
    • +43
    ./oauth2/CustomTokenProviderAdapter.java
    • -0
    • +16
    ./security/AbfsDelegationTokenIdentifier.java
    • -8
    • +91
    ./security/AbfsDelegationTokenManager.java
    • -0
    • +87
    ./security/AbfsDtFetcher.java
    • -0
    • +36
    ./security/AbfssDtFetcher.java
  1. … 22 more files in changeset.
HADOOP-16136. ABFS: Should only transform username to short name

Contributed by Da Zhou.

    • -19
    • +20
    ./oauth2/IdentityTransformer.java
  1. … 1 more file in changeset.
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.

Contributed by Da Zhou and Junhua Gu.

    • -0
    • +278
    ./oauth2/IdentityTransformer.java
  1. … 2 more files in changeset.
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth."

(accidentally mixed in two patches)

This reverts commit fa8cd1bf28f5b81849ba351a2d7225fbc580350d.

    • -275
    • +0
    ./oauth2/IdentityTransformer.java
  1. … 8 more files in changeset.
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.

Contributed by Da Zhou and Junhua Gu.

    • -0
    • +275
    ./oauth2/IdentityTransformer.java
  1. … 8 more files in changeset.
HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. Contributed by Shweta Yakkali.

Signed-off-by: Sean Mackrory <mackrorysd@apache.org>

  1. … 1 more file in changeset.
HADOOP-16048. ABFS: Fix Date format parser.

Contributed by Da Zhou.