Clone Tools
  • last updated 19 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

  1. … 11 more files in changeset.
MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

(cherry picked from commit c59b1b66a5264c3164ac6c9356b51de9b5349c66)

  1. … 11 more files in changeset.
MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

  1. … 10 more files in changeset.
MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

(cherry picked from commit ed302f1fed6d124d682486d24dae958946dba9be)

  1. … 10 more files in changeset.
MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)

Signed-off-by: Jonathan Eagles <jeagles@gmail.com>

(cherry picked from commit ed302f1fed6d124d682486d24dae958946dba9be)

  1. … 10 more files in changeset.
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3

Contributed by Steve Loughran.

Includes

-S3A glob scans don't bother trying to resolve symlinks

-stack traces don't get lost in getFileStatuses() when exceptions are wrapped

-debug level logging of what is up in Globber

-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.

-ITestRestrictedReadAccess tests incomplete read access to files.

This adds a builder API for constructing globbers which other stores can use

so that they too can skip symlink resolution when not needed.

Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7

  1. … 8 more files in changeset.
MAPREDUCE-7225: Fix broken current folder expansion during MR job start. Contributed by Peter Bacsko.

  1. … 1 more file in changeset.
MAPREDUCE-7225: Fix broken current folder expansion during MR job start. Contributed by Peter Bacsko.

  1. … 1 more file in changeset.
MAPREDUCE-7225: Fix broken current folder expansion during MR job start. Contributed by Peter Bacsko.

  1. … 1 more file in changeset.
HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota.

  1. … 21 more files in changeset.
HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota.

  1. … 21 more files in changeset.
MAPREDUCE-7214. Remove unused pieces related to `mapreduce.job.userlog.retain.hours`

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

  1. … 3 more files in changeset.
MAPREDUCE-6794. Remove unused properties from TTConfig.java

  1. … 3 more files in changeset.
HADOOP-16212. Update guava to 27.0-jre in hadoop-project branch-3.0. Contributed by Gabor Bota.

  1. … 19 more files in changeset.
MAPREDUCE-7210. Replace `mapreduce.job.counters.limit` with `mapreduce.job.counters.max` in mapred-default.xml

This closes #878

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

MAPREDUCE-7198. mapreduce.task.timeout=0 configuration used to disable timeout doesn't work.

  1. … 2 more files in changeset.
HADOOP-14524. Make CryptoCodec Closeable so it can be cleaned up proactively.

(cherry picked from commit 2afe9722af2fa4c8e870b0dbf1328e31d1f0fec7)

Conflicts:

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java

  1. … 8 more files in changeset.
HADOOP-16196. Path Parameterize Comparable.

Author: David Mollitor <david.mollitor@cloudera.com>

(cherry picked from commit 246ab77f281614b2206bfb1dae70a8dec62147a7)

  1. … 1 more file in changeset.
HADOOP-16196. Path Parameterize Comparable.

Author: David Mollitor <david.mollitor@cloudera.com>

  1. … 1 more file in changeset.
YARN-9271. Backport YARN-6927 for resource type support in MapReduce

  1. … 6 more files in changeset.
YARN-9271. Backport YARN-6927 for resource type support in MapReduce

  1. … 6 more files in changeset.
YARN-9271. Backport YARN-6927 for resource type support in MapReduce

  1. … 6 more files in changeset.
YARN-9271. Backport YARN-6927 for resource type support in MapReduce

  1. … 6 more files in changeset.
YARN-9271. Backport YARN-6927 for resource type support in MapReduce

  1. … 6 more files in changeset.
YARN-9271. Backport YARN-6927 for resource type support in MapReduce

  1. … 6 more files in changeset.
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may

options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,

as the existing input format/record readers can't handle S3 select output where

the stream is shorter than the file length, and splitting plain text is suboptimal.

Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific

configuration parameters which can be set in jobs and used to set filesystem input stream

options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran

  1. … 67 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran and Daryn Sharp.

  1. … 101 more files in changeset.
Revert "HADOOP-14556. S3A to support Delegation Tokens."

This reverts commit d7152332b32a575c3a92e3f4c44b95e58462528d.

  1. … 104 more files in changeset.
HADOOP-14556. S3A to support Delegation Tokens.

Contributed by Steve Loughran.

  1. … 104 more files in changeset.
HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota.

  1. … 12 more files in changeset.