Clone Tools
  • last updated 21 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HDFS-15168: ABFS enhancement to translate AAD to Linux identities. (#1978)

    • -0
    • +42
    ./IdentityHandler.java
    • -0
    • +195
    ./TextFileBasedIdentityHandler.java
  1. … 7 more files in changeset.
Hadoop-17015. ABFS: Handling Rename and Delete idempotency

Contributed by Sneha Vijayarajan.

    • -0
    • +71
    ./DateTimeUtils.java
  1. … 11 more files in changeset.
HADOOP-16916: ABFS: Delegation SAS generator for integration with Ranger

Contributed by Thomas Marquardt.

DETAILS:

Previously we had a SASGenerator class which generated Service SAS, but we need to add DelegationSASGenerator.

I separated SASGenerator into a base class and two subclasses ServiceSASGenerator and DelegationSASGenreator. The

code in ServiceSASGenerator is copied from SASGenerator but the DelegationSASGenrator code is new. The

DelegationSASGenerator code demonstrates how to use Delegation SAS with minimal permissions, as would be used

by an authorization service such as Apache Ranger. Adding this to the tests helps us lock in this behavior.

Added a MockDelegationSASTokenProvider for testing User Delegation SAS.

Fixed the ITestAzureBlobFileSystemCheckAccess tests to assume oauth client ID so that they are ignored when that

is not configured.

To improve performance, AbfsInputStream/AbfsOutputStream re-use SAS tokens until the expiry is within 120 seconds.

After this a new SAS will be requested. The default period of 120 seconds can be changed using the configuration

setting "fs.azure.sas.token.renew.period.for.streams".

The SASTokenProvider operation names were updated to correspond better with the ADLS Gen2 REST API, since these

operations must be provided tokens with appropriate SAS parameters to succeed.

Support for the version 2.0 AAD authentication endpoint was added to AzureADAuthenticator.

The getFileStatus method was mistakenly calling the ADLS Gen2 Get Properties API which requires read permission

while the getFileStatus call only requires execute permission. ADLS Gen2 Get Status API is supposed to be used

for this purpose, so the underlying AbfsClient.getPathStatus API was updated with a includeProperties

parameter which is set to false for getFileStatus and true for getXAttr.

Added SASTokenProvider support for delete recursive.

Fixed bugs in AzureBlobFileSystem where public methods were not validating the Path by calling makeQualified. This is

necessary to avoid passing null paths and to convert relative paths into absolute paths.

Canonicalized the path used for root path internally so that root path can be used with SAS tokens, which requires

that the path in the URL and the path in the SAS token match. Internally the code was using

"//" instead of "/" for the root path, sometimes. Also related to this, the AzureBlobFileSystemStore.getRelativePath

API was updated so that we no longer remove and then add back a preceding forward / to paths.

To run ITestAzureBlobFileSystemDelegationSAS tests follow the instructions in testing_azure.md under the heading

"To run Delegation SAS test cases". You also need to set "fs.azure.enable.check.access" to true.

TEST RESULTS:

namespace.enabled=true

auth.type=SharedKey

-------------------

$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 63, Failures: 0, Errors: 0, Skipped: 0

Tests run: 432, Failures: 0, Errors: 0, Skipped: 41

Tests run: 206, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=false

auth.type=SharedKey

-------------------

$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 63, Failures: 0, Errors: 0, Skipped: 0

Tests run: 432, Failures: 0, Errors: 0, Skipped: 244

Tests run: 206, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true

auth.type=SharedKey

sas.token.provider.type=MockDelegationSASTokenProvider

enable.check.access=true

-------------------

$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 63, Failures: 0, Errors: 0, Skipped: 0

Tests run: 432, Failures: 0, Errors: 0, Skipped: 33

Tests run: 206, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true

auth.type=OAuth

-------------------

$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 63, Failures: 0, Errors: 0, Skipped: 0

Tests run: 432, Failures: 0, Errors: 1, Skipped: 74

Tests run: 206, Failures: 0, Errors: 0, Skipped: 140

    • -0
    • +207
    ./CachedSASToken.java
  1. … 28 more files in changeset.
HADOOP-16988. Remove source code from branch-2. (aajisaka via jhung)

This closes #1959

  1. … 10842 more files in changeset.
HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8.

Contributed by Sahil Takiar.

This moves the SSLSocketFactoryEx class from hadoop-azure into hadoop-common

as the DelegatingSSLSocketFactory and binds the S3A connector to it so that

it can avoid using those HTTPS algorithms which are underperformant on Java 8.

Change-Id: Ie9e6ac24deac1aa05e136e08899620efa7d22abd

  1. … 15 more files in changeset.
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL"

This reverts commit b067f8acaa79b1230336900a5c62ba465b2adb28.

Change-Id: I584b050a56c0e6f70b11fa3f7db00d5ac46e7dd8

  1. … 14 more files in changeset.
HADOOP-16269. ABFS: add listFileStatus with StartFrom.

Author: Da Zhou

  1. … 5 more files in changeset.
HADOOP-16269. ABFS: add listFileStatus with StartFrom.

Author: Da Zhou

  1. … 5 more files in changeset.
HADOOP-16269. ABFS: add listFileStatus with StartFrom.

Author: Da Zhou

  1. … 5 more files in changeset.
HADOOP-16205 Backport ABFS driver from trunk to branch 2.0: Fix build and test failures.

Contributed by Yuan Gao.

  1. … 14 more files in changeset.
HADOOP-16050: s3a SSL connections should use OpenSSL

(cherry picked from commit aebf229c175dfa19fff3b31e9e67596f6c6124fa)

  1. … 14 more files in changeset.
HADOOP-16174. Disable wildfly logs to the console.

Follow-on to HADOOP-15851.

Author: Denes Gerencser <dgerencser@cloudera.com>

(cherry picked from commit ddede7ae6fbbadbe08861bc85a664b73d66f77c7)

HADOOP-16174. Disable wildfly logs to the console.

Follow-on to HADOOP-15851.

Author: Denes Gerencser <dgerencser@cloudera.com>

HADOOP-15851. Disable wildfly logs to the console. Contributed by Vishwajeet Dusane.

HADOOP-15851. Disable wildfly logs to the console. Contributed by Vishwajeet Dusane.

(cherry picked from commit ef9dc6c44c686e836bb25e31ff355cff80572d23)

HADOOP-15757. ABFS: remove dependency on common-codec Base64. Contributed by Da Zhou.

  1. … 6 more files in changeset.
HADOOP-15757. ABFS: remove dependency on common-codec Base64. Contributed by Da Zhou.

  1. … 6 more files in changeset.
HADOOP-15663. ABFS: Simplify configuration. Contributed by Da Zhou.

  1. … 24 more files in changeset.
HADOOP-15663. ABFS: Simplify configuration. Contributed by Da Zhou.

  1. … 24 more files in changeset.
HADOOP-15682. ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle. Contributed by Thomas Marquardt.

  1. … 5 more files in changeset.
HADOOP-15682. ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle. Contributed by Thomas Marquardt.

  1. … 5 more files in changeset.
HADOOP-15669. ABFS: Improve HTTPS Performance. Contributed by Vishwajeet Dusane.

    • -0
    • +240
    ./SSLSocketFactoryEx.java
  1. … 9 more files in changeset.
HADOOP-15669. ABFS: Improve HTTPS Performance. Contributed by Vishwajeet Dusane.

    • -0
    • +240
    ./SSLSocketFactoryEx.java
  1. … 9 more files in changeset.
HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit.

Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran

  1. … 127 more files in changeset.
HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit.

Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran

  1. … 127 more files in changeset.