Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may

options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,

as the existing input format/record readers can't handle S3 select output where

the stream is shorter than the file length, and splitting plain text is suboptimal.

Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific

configuration parameters which can be set in jobs and used to set filesystem input stream

options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran

    • -0
    • +14
    ./hadoop/mapreduce/MRJobConfig.java
  1. … 67 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran and Daryn Sharp.

  1. … 101 more files in changeset.
Revert "HADOOP-14556. S3A to support Delegation Tokens."

This reverts commit d7152332b32a575c3a92e3f4c44b95e58462528d.

  1. … 104 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran.

  1. … 104 more files in changeset.
HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota.

  1. … 12 more files in changeset.
MAPREDUCE-7164. FileOutputCommitter does not report progress while merging paths. Contributed by Kuhu Shukla

(cherry picked from commit 4d8de7ab690ef919b392b12d856482a6a1f2bb3d)

  1. … 1 more file in changeset.
MAPREDUCE-7164. FileOutputCommitter does not report progress while merging paths. Contributed by Kuhu Shukla

(cherry picked from commit 4d8de7ab690ef919b392b12d856482a6a1f2bb3d)

  1. … 1 more file in changeset.
MAPREDUCE-7164. FileOutputCommitter does not report progress while merging paths. Contributed by Kuhu Shukla

  1. … 1 more file in changeset.
MAPREDUCE-7164. FileOutputCommitter does not report progress while merging paths. Contributed by Kuhu Shukla

(cherry picked from commit 4d8de7ab690ef919b392b12d856482a6a1f2bb3d)

  1. … 1 more file in changeset.
MAPREDUCE-6190. If a task stucks before its first heartbeat, it never timeouts and the MR job becomes stuck. Contributed by Zhaohui Xin.

  1. … 3 more files in changeset.
MAPREDUCE-7162. TestEvents#testEvents fails. Contributed by Zhaohui Xin.

(cherry picked from commit 1aad99a71813660b83628cacfed393d0b3a123cc)

MAPREDUCE-7162. TestEvents#testEvents fails. Contributed by Zhaohui Xin.

(cherry picked from commit 1aad99a71813660b83628cacfed393d0b3a123cc)

MAPREDUCE-7162. TestEvents#testEvents fails. Contributed by Zhaohui Xin.

MAPREDUCE-7158. Inefficient Flush Logic in JobHistory EventWriter. (Zichen Sun via wangda)

Change-Id: I99ace87980da03bb35a8012cea7218d602a8817a

(cherry picked from commit 762a56cc64bc07d57f94e253920534b8e049f238)

MAPREDUCE-7158. Inefficient Flush Logic in JobHistory EventWriter. (Zichen Sun via wangda)

Change-Id: I99ace87980da03bb35a8012cea7218d602a8817a

(cherry picked from commit 762a56cc64bc07d57f94e253920534b8e049f238)

MAPREDUCE-7158. Inefficient Flush Logic in JobHistory EventWriter. (Zichen Sun via wangda)

Change-Id: I99ace87980da03bb35a8012cea7218d602a8817a

MAPREDUCE-7148. Fast fail jobs when exceeds dfs quota limitation. Contributed by Wang Yan

  1. … 5 more files in changeset.
MAPREDUCE-4669. MRAM web UI does not work with HTTPS. (Contributed by Robert Kanter)

    • -0
    • +22
    ./hadoop/mapreduce/MRJobConfig.java
  1. … 8 more files in changeset.
MAPREDUCE-7150. Optimize collections used by MR JHS to reduce its memory. (Contributed by Misha Dmitriev)

  1. … 2 more files in changeset.
MAPREDUCE-7132. JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used. Contributed by Peter Bacsko.

(cherry picked from commit 25f8fcb06476938826cdc92858a61124b18cd98d)

  1. … 2 more files in changeset.
MAPREDUCE-7132. JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used. Contributed by Peter Bacsko.

(cherry picked from commit 25f8fcb06476938826cdc92858a61124b18cd98d)

  1. … 2 more files in changeset.
MAPREDUCE-7132. JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used. Contributed by Peter Bacsko.

(cherry picked from commit 25f8fcb06476938826cdc92858a61124b18cd98d)

  1. … 2 more files in changeset.
MAPREDUCE-7132. JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used. Contributed by Peter Bacsko.

  1. … 2 more files in changeset.
MAPREDUCE-7149. Javadocs for FileInputFormat and OutputFormat to mention DT collection. Contributed by Steve Loughran.

MAPREDUCE-7125. JobResourceUploader creates LocalFileSystem when it's not necessary. (Peter Cseh via wangda)

Change-Id: I1aa720ed03739f6f4abeec46f6068e2ab332987a

(cherry picked from commit 230d90c5b9cc0d6e74bf16b1cf6685a163a24d48)

MAPREDUCE-7125. JobResourceUploader creates LocalFileSystem when it's not necessary. (Peter Cseh via wangda)

Change-Id: I1aa720ed03739f6f4abeec46f6068e2ab332987a

HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs. Contributed by Steve Loughran.

(cherry picked from commit 5a0babf76550f63dad4c17173c4da2bf335c6532)

  1. … 18 more files in changeset.
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs. Contributed by Steve Loughran.

  1. … 18 more files in changeset.
MAPREDUCE-6861. Add metrics tags for ShuffleClientMetrics. (Contributed by Zoltan Siegl)

    • -11
    • +13
    ./hadoop/mapreduce/task/reduce/Shuffle.java
  1. … 1 more file in changeset.
HADOOP-15550. Avoid static initialization of ObjectMappers

(cherry picked from commit 7a3c6e9c3cd9ffdc71946fd12f5c3d59718c4939)

  1. … 9 more files in changeset.