Clone Tools
  • last updated 18 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Revert "HADOOP-16818. ABFS: Combine append+flush calls for blockblob & appendblob"

This reverts commit 3612317038196ee0cb6d7204056d54b7a7ed8bf7.

Change-Id: Ie0d36f25de0b55a937894f4d9963c495bae0576a

  1. … 9 more files in changeset.
Revert ""HADOOP-16910. ABFS Streams to update FileSystem.Statistics counters on IO."

This reverts commit e2c7ac71b5ee47bb40294acd10c0c21dd6ee430f.

Change-Id: I5b5a93f5a36cdb0c3d56d1b3f747c318f089de20

  1. … 4 more files in changeset.
ABFS Streams to update FileSystem.Statistics counters on IO.

Contributed by Mehakmeet Singh

    • -0
    • +157
    ./org/apache/hadoop/fs/azurebfs/ITestAbfsStreamStatistics.java
  1. … 4 more files in changeset.
HADOOP-16818. ABFS: Combine append+flush calls for blockblob & appendblob

Contributed by Ishani Ahuja.

    • -0
    • +407
    ./org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
  1. … 9 more files in changeset.
HADOOP-16920 ABFS: Make list page size configurable.

Contributed by Bilahari T H.

The page limit is set in "fs.azure.list.max.results"; default value is 500.

There's currently a limit of 5000 in the store -there are no range checks

in the client code so that limit can be changed on the server without

any need to update the abfs connector.

  1. … 4 more files in changeset.
Hadoop 16890. Change in expiry calculation for MSI token provider.

Contributed by Bilahari T H

  1. … 4 more files in changeset.
HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden

Contributed by Steve Loughran.

Not all stores do complete validation here; in particular the S3A

Connector does not: checking up the entire directory tree to see if a path matches

is a file significantly slows things down.

This check does take place in S3A mkdirs(), which walks backwards up the list of

parent paths until it finds a directory (success) or a file (failure).

In practice production applications invariably create destination directories

before writing 1+ file into them -restricting check purely to the mkdirs()

call deliver significant speed up while implicitly including the checks.

Change-Id: I2c9df748e92b5655232e7d888d896f1868806eb0

  1. … 10 more files in changeset.
HADOOP-16730: ABFS: Support for Shared Access Signatures (SAS). Contributed by Sneha Vijayarajan.

    • -0
    • +129
    ./org/apache/hadoop/fs/azurebfs/utils/SASGenerator.java
  1. … 14 more files in changeset.
HADOOP-16859: ABFS: Add unbuffer support to ABFS connector.

Contributed by Sahil Takiar

  1. … 3 more files in changeset.
HADOOP-16706. ITestClientUrlScheme fails for accounts which don't support HTTP

Adds a new service code to recognise accounts without HTTP support; catches

that and considers such a responset a successful validation of the ability of the

client to switch to http when the test parameters expect that.

Contributed by Steve Loughran

  1. … 1 more file in changeset.
HADOOP-16825: ITestAzureBlobFileSystemCheckAccess failing. Contributed by Bilahari T H.

HADOOP-16845: Disable ITestAbfsClient.testContinuationTokenHavingEqualSign due to ADLS Gen2 service bug. Contributed by Sneha Vijayarajan.

HADOOP-16785. followup to abfs close() fix.

Adds one extra test to the ABFS close logic, to explicitly

verify that the close sequence of FilterOutputStream is

not going to fail.

This is just a due-diligence patch, but it helps ensure

that no regressions creep in in future.

Contributed by Steve Loughran.

Change-Id: Ifd33a8c322d32513411405b15f50a1aebcfa6e48

HADOOP-16785. followup to abfs close() fix.

Adds one extra test to the ABFS close logic, to explicitly

verify that the close sequence of FilterOutputStream is

not going to fail.

This is just a due-diligence patch, but it helps ensure

that no regressions creep in in future.

Contributed by Steve Loughran.

Change-Id: Ifd33a8c322d32513411405b15f50a1aebcfa6e48

HADOOP-16005. NativeAzureFileSystem does not support setXAttr.

Contributed by Clemens Wolff.

  1. … 5 more files in changeset.
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.

This hardens the wasb and abfs output streams' resilience to being invoked

in/after close().

wasb:

Explicity raise IOEs on operations invoked after close,

rather than implicitly raise NPEs.

This ensures that invocations which catch and swallow IOEs will perform as

expected.

abfs:

When rethrowing an IOException in the close() call, explicitly wrap it

with a new instance of the same subclass.

This is needed to handle failures in try-with-resources clauses, where

any exception in closed() is added as a suppressed exception to the one

thrown in the try {} clause

*and you cannot attach the same exception to itself*

Contributed by Steve Loughran.

Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b

  1. … 3 more files in changeset.
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.

This hardens the wasb and abfs output streams' resilience to being invoked

in/after close().

wasb:

Explicity raise IOEs on operations invoked after close,

rather than implicitly raise NPEs.

This ensures that invocations which catch and swallow IOEs will perform as

expected.

abfs:

When rethrowing an IOException in the close() call, explicitly wrap it

with a new instance of the same subclass.

This is needed to handle failures in try-with-resources clauses, where

any exception in closed() is added as a suppressed exception to the one

thrown in the try {} clause

*and you cannot attach the same exception to itself*

Contributed by Steve Loughran.

Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b

  1. … 3 more files in changeset.
HADOOP-16778. ABFS: Backport HADOOP-16660 ABFS: Make RetryCount in ExponentialRetryPolicy Configurable to Branch-2. Contributed by Sneha Vijayarajan.

    • -0
    • +67
    ./org/apache/hadoop/fs/azurebfs/oauth2/RetryTestTokenProvider.java
  1. … 3 more files in changeset.
HADOOP-16734. Backport HADOOP-16455- "ABFS: Implement FileSystem.access() method" to branch-2. Contributed by Bilahari T H.

  1. … 10 more files in changeset.
HADOOP-16660. ABFS: Make RetryCount in ExponentialRetryPolicy Configurable.

Contributed by Sneha Vijayarajan.

    • -0
    • +67
    ./org/apache/hadoop/fs/azurebfs/oauth2/RetryTestTokenProvider.java
  1. … 3 more files in changeset.
HADOOP-16660. ABFS: Make RetryCount in ExponentialRetryPolicy Configurable.

Contributed by Sneha Vijayarajan.

    • -0
    • +67
    ./org/apache/hadoop/fs/azurebfs/oauth2/RetryTestTokenProvider.java
  1. … 3 more files in changeset.
HADOOP-16455. ABFS: Implement FileSystem.access() method.

Contributed by Bilahari T H.

  1. … 10 more files in changeset.
HADOOP-16455. ABFS: Implement FileSystem.access() method.

Contributed by Bilahari T H.

  1. … 10 more files in changeset.
HADOOP-16687. ABFS: Fix testcase added for HADOOP-16138 for namespace enabled account. (#1701)

HADOOP-16612. Track Azure Blob File System client-perceived latency

Contributed by Jeetesh Mangwani.

This add the ability to track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency in the Hadoop ABFS driver.

The latency information is sent back to the ADLS Gen 2 REST API endpoints in the subsequent requests.

    • -0
    • +408
    ./org/apache/hadoop/fs/azurebfs/services/TestAbfsPerfTracker.java
  1. … 16 more files in changeset.
HADOOP-16612. Track Azure Blob File System client-perceived latency

Contributed by Jeetesh Mangwani.

This add the ability to track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency in the Hadoop ABFS driver.

The latency information is sent back to the ADLS Gen 2 REST API endpoints in the subsequent requests.

    • -0
    • +408
    ./org/apache/hadoop/fs/azurebfs/services/TestAbfsPerfTracker.java
  1. … 16 more files in changeset.
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI

(cherry picked from commit 9a8edb0aeddd7787b2654f6e2a8465c325e048a2)

  1. … 1 more file in changeset.
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI

  1. … 1 more file in changeset.
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI

(cherry picked from commit 9a8edb0aeddd7787b2654f6e2a8465c325e048a2)

  1. … 1 more file in changeset.
HADOOP-16630 : Backport of Hadoop-16548 : Disable Flush() over config

  1. … 6 more files in changeset.