Clone Tools
  • last updated 21 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16910 : ABFS Streams to update FileSystem.Statistics counters on IO. (#1918). Contributed by Mehakmeet Singh.

  1. … 4 more files in changeset.
Revert "HADOOP-16818. ABFS: Combine append+flush calls for blockblob & appendblob"

This reverts commit 3612317038196ee0cb6d7204056d54b7a7ed8bf7.

Change-Id: Ie0d36f25de0b55a937894f4d9963c495bae0576a

  1. … 9 more files in changeset.
Revert ""HADOOP-16910. ABFS Streams to update FileSystem.Statistics counters on IO."

This reverts commit e2c7ac71b5ee47bb40294acd10c0c21dd6ee430f.

Change-Id: I5b5a93f5a36cdb0c3d56d1b3f747c318f089de20

  1. … 4 more files in changeset.
ABFS Streams to update FileSystem.Statistics counters on IO.

Contributed by Mehakmeet Singh

    • -0
    • +157
    ./org/apache/hadoop/fs/azurebfs/ITestAbfsStreamStatistics.java
  1. … 4 more files in changeset.
HADOOP-16818. ABFS: Combine append+flush calls for blockblob & appendblob

Contributed by Ishani Ahuja.

    • -0
    • +407
    ./org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
  1. … 9 more files in changeset.
HADOOP-16920 ABFS: Make list page size configurable.

Contributed by Bilahari T H.

The page limit is set in "fs.azure.list.max.results"; default value is 500.

There's currently a limit of 5000 in the store -there are no range checks

in the client code so that limit can be changed on the server without

any need to update the abfs connector.

  1. … 4 more files in changeset.
Hadoop 16890. Change in expiry calculation for MSI token provider.

Contributed by Bilahari T H

  1. … 4 more files in changeset.
HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden

Contributed by Steve Loughran.

Not all stores do complete validation here; in particular the S3A

Connector does not: checking up the entire directory tree to see if a path matches

is a file significantly slows things down.

This check does take place in S3A mkdirs(), which walks backwards up the list of

parent paths until it finds a directory (success) or a file (failure).

In practice production applications invariably create destination directories

before writing 1+ file into them -restricting check purely to the mkdirs()

call deliver significant speed up while implicitly including the checks.

Change-Id: I2c9df748e92b5655232e7d888d896f1868806eb0

  1. … 10 more files in changeset.
HADOOP-16730: ABFS: Support for Shared Access Signatures (SAS). Contributed by Sneha Vijayarajan.

    • -0
    • +129
    ./org/apache/hadoop/fs/azurebfs/utils/SASGenerator.java
  1. … 14 more files in changeset.
HADOOP-16859: ABFS: Add unbuffer support to ABFS connector.

Contributed by Sahil Takiar

  1. … 3 more files in changeset.
HADOOP-16706. ITestClientUrlScheme fails for accounts which don't support HTTP

Adds a new service code to recognise accounts without HTTP support; catches

that and considers such a responset a successful validation of the ability of the

client to switch to http when the test parameters expect that.

Contributed by Steve Loughran

  1. … 1 more file in changeset.
HADOOP-16825: ITestAzureBlobFileSystemCheckAccess failing. Contributed by Bilahari T H.

HADOOP-16845: Disable ITestAbfsClient.testContinuationTokenHavingEqualSign due to ADLS Gen2 service bug. Contributed by Sneha Vijayarajan.

HADOOP-16785. followup to abfs close() fix.

Adds one extra test to the ABFS close logic, to explicitly

verify that the close sequence of FilterOutputStream is

not going to fail.

This is just a due-diligence patch, but it helps ensure

that no regressions creep in in future.

Contributed by Steve Loughran.

Change-Id: Ifd33a8c322d32513411405b15f50a1aebcfa6e48

HADOOP-16005. NativeAzureFileSystem does not support setXAttr.

Contributed by Clemens Wolff.

  1. … 5 more files in changeset.
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.

This hardens the wasb and abfs output streams' resilience to being invoked

in/after close().

wasb:

Explicity raise IOEs on operations invoked after close,

rather than implicitly raise NPEs.

This ensures that invocations which catch and swallow IOEs will perform as

expected.

abfs:

When rethrowing an IOException in the close() call, explicitly wrap it

with a new instance of the same subclass.

This is needed to handle failures in try-with-resources clauses, where

any exception in closed() is added as a suppressed exception to the one

thrown in the try {} clause

*and you cannot attach the same exception to itself*

Contributed by Steve Loughran.

Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b

  1. … 3 more files in changeset.
HADOOP-16660. ABFS: Make RetryCount in ExponentialRetryPolicy Configurable.

Contributed by Sneha Vijayarajan.

    • -0
    • +67
    ./org/apache/hadoop/fs/azurebfs/oauth2/RetryTestTokenProvider.java
  1. … 3 more files in changeset.
HADOOP-16455. ABFS: Implement FileSystem.access() method.

Contributed by Bilahari T H.

  1. … 10 more files in changeset.
HADOOP-16687. ABFS: Fix testcase added for HADOOP-16138 for namespace enabled account. (#1701)

HADOOP-16612. Track Azure Blob File System client-perceived latency

Contributed by Jeetesh Mangwani.

This add the ability to track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency in the Hadoop ABFS driver.

The latency information is sent back to the ADLS Gen 2 REST API endpoints in the subsequent requests.

    • -0
    • +408
    ./org/apache/hadoop/fs/azurebfs/services/TestAbfsPerfTracker.java
  1. … 16 more files in changeset.
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI

  1. … 1 more file in changeset.
HADOOP-16578 : Avoid FileSystem API calls when FileSystem already exists

  1. … 1 more file in changeset.
HADOOP-16548 : Disable Flush() over config

  1. … 6 more files in changeset.
HADOOP-16138. hadoop fs mkdir / of nonexistent abfs container raises NPE (#1302). Contributed by Gabor Bota.

Change-Id: I2f637865c871e400b95fe7ddaa24bf99fa192023

  1. … 1 more file in changeset.
HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8.

Contributed by Sahil Takiar.

This moves the SSLSocketFactoryEx class from hadoop-azure into hadoop-common

as the DelegatingSSLSocketFactory and binds the S3A connector to it so that

it can avoid using those HTTPS algorithms which are underperformant on Java 8.

Change-Id: Ie9e6ac24deac1aa05e136e08899620efa7d22abd

  1. … 14 more files in changeset.
HADOOP-16315. ABFS: transform full UPN for named user in AclStatus

Contributed by Da Zhou

Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290

  1. … 2 more files in changeset.
HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC.

Contributed by Bilahari T H

Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0

  1. … 1 more file in changeset.
HADOOP-16404. ABFS default blocksize change(256MB from 512MB)

Contributed by: Arun Singh

  1. … 1 more file in changeset.
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL"

This reverts commit b067f8acaa79b1230336900a5c62ba465b2adb28.

Change-Id: I584b050a56c0e6f70b11fa3f7db00d5ac46e7dd8

  1. … 13 more files in changeset.
Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest"

This reverts commit 7c2d7c05a9a9cc981674f97cc3825e917a17b1f7.

Git Commit name and email were incorrect. Will re-commit.