Clone Tools
  • last updated 19 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3

Contributed by Steve Loughran.

Includes

-S3A glob scans don't bother trying to resolve symlinks

-stack traces don't get lost in getFileStatuses() when exceptions are wrapped

-debug level logging of what is up in Globber

-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.

-ITestRestrictedReadAccess tests incomplete read access to files.

This adds a builder API for constructing globbers which other stores can use

so that they too can skip symlink resolution when not needed.

Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7

  1. … 11 more files in changeset.
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may

options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,

as the existing input format/record readers can't handle S3 select output where

the stream is shorter than the file length, and splitting plain text is suboptimal.

Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific

configuration parameters which can be set in jobs and used to set filesystem input stream

options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran

  1. … 69 more files in changeset.
MAPREDUCE-7149. Javadocs for FileInputFormat and OutputFormat to mention DT collection. Contributed by Steve Loughran.

  1. … 3 more files in changeset.
MAPREDUCE-7063. Fix log level inconsistency in CombineFileInputFormat.java

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

MAPREDUCE-7086. Add config to allow FileInputFormat to ignore directories when recursive=false. Contributed by Sergey Shelukhin

(cherry picked from commit 68c6ec719da8e79ada31c8f3a82124f90b9a71fd)

  1. … 3 more files in changeset.
MAPREDUCE-7086. Add config to allow FileInputFormat to ignore directories when recursive=false. Contributed by Sergey Shelukhin

  1. … 3 more files in changeset.
MAPREDUCE-6983. Moving logging APIs over to slf4j in hadoop-mapreduce-client-core. Contributed by Jinjiang Ling.

  1. … 89 more files in changeset.
MAPREDUCE-6983. Moving logging APIs over to slf4j in hadoop-mapreduce-client-core. Contributed by Jinjiang Ling.

  1. … 89 more files in changeset.
MAPREDUCE-6983. Moving logging APIs over to slf4j in hadoop-mapreduce-client-core. Contributed by Jinjiang Ling.

(cherry picked from commit 178751ed8c9d47038acf8616c226f1f52e884feb)

  1. … 89 more files in changeset.
MAPREDUCE-6887. Modifier 'static' is redundant for inner enums. Contributed by ZhangBing Lin.

  1. … 37 more files in changeset.
MAPREDUCE-6887. Modifier 'static' is redundant for inner enums. Contributed by ZhangBing Lin.

  1. … 37 more files in changeset.
HADOOP-14250. Correct spelling of 'separate' and variants. Contributed by Doris Gu

  1. … 19 more files in changeset.
MAPREDUCE-6751. Add debug log message when splitting is not possible due to unsplittable compression. (Peter Vary via rchiang)

(cherry picked from commit 6c154abd33279475315b5f7f78dc47f1b0aa7028)

  1. … 1 more file in changeset.
MAPREDUCE-6751. Add debug log message when splitting is not possible due to unsplittable compression. (Peter Vary via rchiang)

  1. … 1 more file in changeset.
MAPREDUCE-6741. Refactor UncompressedSplitLineReader.fillBuffer(). Contributed by Daniel Templeton.

(cherry picked from commit 0b7b8a377611b2a3041a2995504a437c36dfa6e6)

MAPREDUCE-6741. Refactor UncompressedSplitLineReader.fillBuffer(). Contributed by Daniel Templeton.

(cherry picked from commit 0b7b8a377611b2a3041a2995504a437c36dfa6e6)

(cherry picked from commit db2d96b378d6a4ba9a504b4daaca5d8ff6f5f227)

MAPREDUCE-6741. Refactor UncompressedSplitLineReader.fillBuffer(). Contributed by Daniel Templeton.

MAPREDUCE-6558. multibyte delimiters with compressed input files generate duplicate records. Contributed by Wilfred Spiegelenburg (cherry picked from commit 9227dfc25f373a99cb66ad7d6bacef8dcf336f77)

  1. … 4 more files in changeset.
MAPREDUCE-6558. multibyte delimiters with compressed input files generate duplicate records. Contributed by Wilfred Spiegelenburg (cherry picked from commit 9227dfc25f373a99cb66ad7d6bacef8dcf336f77)

  1. … 4 more files in changeset.
MAPREDUCE-6558. multibyte delimiters with compressed input files generate duplicate records. Contributed by Wilfred Spiegelenburg (cherry picked from commit 9227dfc25f373a99cb66ad7d6bacef8dcf336f77)

  1. … 3 more files in changeset.
MAPREDUCE-6558. multibyte delimiters with compressed input files generate duplicate records. Contributed by Wilfred Spiegelenburg

  1. … 3 more files in changeset.
MAPREDUCE-6558. multibyte delimiters with compressed input files generate duplicate records. Contributed by Wilfred Spiegelenburg (cherry picked from commit 9227dfc25f373a99cb66ad7d6bacef8dcf336f77)

  1. … 3 more files in changeset.
MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException. Contributed by Junping Du.

(cherry picked from commit c6f2d761d5430eac6b9f07f137a7028de4e0660c)

  1. … 2 more files in changeset.
MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException. Contributed by Junping Du.

(cherry picked from commit c6f2d761d5430eac6b9f07f137a7028de4e0660c)

(cherry picked from commit f1999fe2754cbf11b138fb048c7486cab9b02c97)

Conflicts:

hadoop-mapreduce-project/CHANGES.txt

  1. … 2 more files in changeset.
MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException. Contributed by Junping Du.

(cherry picked from commit c6f2d761d5430eac6b9f07f137a7028de4e0660c)

(cherry picked from commit f1999fe2754cbf11b138fb048c7486cab9b02c97)

  1. … 2 more files in changeset.
MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException. Contributed by Junping Du.

  1. … 2 more files in changeset.
MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException. Contributed by Junping Du.

(cherry picked from commit c6f2d761d5430eac6b9f07f137a7028de4e0660c)

(cherry picked from commit f1999fe2754cbf11b138fb048c7486cab9b02c97)

Conflicts:

hadoop-mapreduce-project/CHANGES.txt

  1. … 2 more files in changeset.
MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause duplicate records (wilfreds via rkanter) (cherry picked from commit 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)

Conflicts:

hadoop-mapreduce-project/CHANGES.txt

  1. … 4 more files in changeset.
MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause duplicate records (wilfreds via rkanter) (cherry picked from commit 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)

Conflicts:

hadoop-mapreduce-project/CHANGES.txt

  1. … 4 more files in changeset.
MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause duplicate records (wilfreds via rkanter) (cherry picked from commit 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)

Conflicts:

hadoop-mapreduce-project/CHANGES.txt

(cherry picked from commit e1b6a5413f018c939e0100bdfbdc018d49c7c6a4)

  1. … 4 more files in changeset.