Clone Tools
  • last updated 28 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
MAPREDUCE-7241. FileInputFormat listStatus with less memory footprint. Contributed by Zhihua Deng

  1. … 1 more file in changeset.
MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

    • -0
    • +31
    ./hadoop/mapreduce/MRJobConfig.java
  1. … 10 more files in changeset.
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3

Contributed by Steve Loughran.

Includes

-S3A glob scans don't bother trying to resolve symlinks

-stack traces don't get lost in getFileStatuses() when exceptions are wrapped

-debug level logging of what is up in Globber

-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.

-ITestRestrictedReadAccess tests incomplete read access to files.

This adds a builder API for constructing globbers which other stores can use

so that they too can skip symlink resolution when not needed.

Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7

    • -0
    • +4
    ./hadoop/mapred/InvalidInputException.java
    • -18
    • +48
    ./hadoop/mapred/LocatedFileStatusFetcher.java
  1. … 8 more files in changeset.
MAPREDUCE-7225: Fix broken current folder expansion during MR job start. Contributed by Peter Bacsko.

  1. … 1 more file in changeset.
MAPREDUCE-7214. Remove unused pieces related to `mapreduce.job.userlog.retain.hours`

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

    • -2
    • +0
    ./hadoop/mapreduce/util/ConfigUtil.java
  1. … 3 more files in changeset.
MAPREDUCE-6794. Remove unused properties from TTConfig.java

    • -41
    • +0
    ./hadoop/mapreduce/util/ConfigUtil.java
  1. … 3 more files in changeset.
HADOOP-16196. Path Parameterize Comparable.

Author: David Mollitor <david.mollitor@cloudera.com>

  1. … 1 more file in changeset.
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may

options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,

as the existing input format/record readers can't handle S3 select output where

the stream is shorter than the file length, and splitting plain text is suboptimal.

Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific

configuration parameters which can be set in jobs and used to set filesystem input stream

options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran

    • -0
    • +14
    ./hadoop/mapreduce/MRJobConfig.java
  1. … 67 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran and Daryn Sharp.

  1. … 101 more files in changeset.
Revert "HADOOP-14556. S3A to support Delegation Tokens."

This reverts commit d7152332b32a575c3a92e3f4c44b95e58462528d.

  1. … 104 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran.

  1. … 104 more files in changeset.
HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota.

  1. … 12 more files in changeset.
MAPREDUCE-7164. FileOutputCommitter does not report progress while merging paths. Contributed by Kuhu Shukla

  1. … 1 more file in changeset.
MAPREDUCE-6190. If a task stucks before its first heartbeat, it never timeouts and the MR job becomes stuck. Contributed by Zhaohui Xin.

  1. … 3 more files in changeset.
MAPREDUCE-7162. TestEvents#testEvents fails. Contributed by Zhaohui Xin.

MAPREDUCE-7158. Inefficient Flush Logic in JobHistory EventWriter. (Zichen Sun via wangda)

Change-Id: I99ace87980da03bb35a8012cea7218d602a8817a

MAPREDUCE-7148. Fast fail jobs when exceeds dfs quota limitation. Contributed by Wang Yan

  1. … 5 more files in changeset.
MAPREDUCE-4669. MRAM web UI does not work with HTTPS. (Contributed by Robert Kanter)

    • -0
    • +22
    ./hadoop/mapreduce/MRJobConfig.java
  1. … 8 more files in changeset.
MAPREDUCE-7150. Optimize collections used by MR JHS to reduce its memory. (Contributed by Misha Dmitriev)

  1. … 2 more files in changeset.
MAPREDUCE-7132. JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used. Contributed by Peter Bacsko.

  1. … 2 more files in changeset.
MAPREDUCE-7149. Javadocs for FileInputFormat and OutputFormat to mention DT collection. Contributed by Steve Loughran.

MAPREDUCE-7125. JobResourceUploader creates LocalFileSystem when it's not necessary. (Peter Cseh via wangda)

Change-Id: I1aa720ed03739f6f4abeec46f6068e2ab332987a

HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs. Contributed by Steve Loughran.

  1. … 18 more files in changeset.
MAPREDUCE-6861. Add metrics tags for ShuffleClientMetrics. (Contributed by Zoltan Siegl)

    • -11
    • +13
    ./hadoop/mapreduce/task/reduce/Shuffle.java
  1. … 1 more file in changeset.
HADOOP-15550. Avoid static initialization of ObjectMappers

  1. … 9 more files in changeset.
MAPREDUCE-7063. Fix log level inconsistency in CombineFileInputFormat.java

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

HADOOP-15507. Add MapReduce counters about EC bytes read.

    • -0
    • +1
    ./hadoop/mapreduce/FileSystemCounter.java
  1. … 7 more files in changeset.
MAPREDUCE-7098. Upgrade common-langs version to 3.7 in hadoop-mapreduce-project

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

  1. … 17 more files in changeset.
MAPREDUCE-7073. Optimize TokenCache#obtainTokensForNamenodesInternal

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

  1. … 1 more file in changeset.
MAPREDUCE-7086. Add config to allow FileInputFormat to ignore directories when recursive=false. Contributed by Sergey Shelukhin

    • -7
    • +18
    ./hadoop/mapred/FileInputFormat.java
  1. … 2 more files in changeset.