Clone Tools
  • last updated 15 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
HDFS-14869 Copy renamed files which are not excluded anymore by filter (#1530)

    • -1
    • +1
    ./main/java/org/apache/hadoop/tools/DistCp.java
HDFS-13660. DistCp job fails when new data is appended in the file while the DistCp copy job is running

This uses the length of the file known at the start of the copy to determine the amount of data to copy.

* If a file is appended to during the copy, the original bytes are copied.

* If a file is truncated during a copy, or the attempt to read the data fails with a truncated stream,

distcp will now fail. Until now these failures were not detected.

Contributed by Mukund Thakur.

Change-Id: I576a49d951fa48d37a45a7e4c82c47488aa8e884

HADOOP-16512. [hadoop-tools] Fix order of actual and expected expression in assert statements

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

  1. … 5 more files in changeset.
HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel (#919)

* DistCp to support checksum validation when copy blocks in parallel

* address review comments

* add checksums comparison test for combine mode

HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel (#919)

* DistCp to support checksum validation when copy blocks in parallel

* address review comments

* add checksums comparison test for combine mode

(cherry picked from commit c765584eb231f8482f5b90b7e8f61f9f7a931d09)

(cherry picked from commit b3c14d4132ed6aa871bb88c4f84f3e3d90da6f93)

Conflicts:

hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java

HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel (#919)

* DistCp to support checksum validation when copy blocks in parallel

* address review comments

* add checksums comparison test for combine mode

(cherry picked from commit c765584eb231f8482f5b90b7e8f61f9f7a931d09)

HADOOP-16440. Distcp can not preserve timestamp with -delete option. Contributed by ludun.

HADOOP-16440. Distcp can not preserve timestamp with -delete option. Contributed by ludun.

HADOOP-16440. Distcp can not preserve timestamp with -delete option. Contributed by ludun.

Revert "HDFS-9913. DistCp to add -useTrash to move deleted files to Trash."

Reverting due to test failures if ~/.Trash not present during test setup.

This reverts commit ee3115f488ce8e44bffac15af9c646190bf67b88.

Change-Id: Icbeeb261570b9131ff99d765ac0945c335b26658

HDFS-9913. DistCp to add -useTrash to move deleted files to Trash.

Contributed by Shen Yinjie.

Change-Id: I03ac7d22ab1054f8e5de4aa7552909c734438f4a

HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

  1. … 2 more files in changeset.
HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from commit 98d20656433cdec76c2108d24ff3b935657c1e80)

  1. … 2 more files in changeset.
HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from commit 98d20656433cdec76c2108d24ff3b935657c1e80)

  1. … 2 more files in changeset.
HADOOP-16294: Enable access to input options by DistCp subclasses.

Adding a protected-scope getter for the DistCpOptions, so that a subclass does

not need to save its own copy of the inputOptions supplied to its constructor,

if it wishes to override the createInputFileListing method with logic similar

to the original implementation, i.e. calling CopyListing#buildListing with a path and input options.

Author: Andrew Olson

    • -0
    • +9
    ./main/java/org/apache/hadoop/tools/DistCp.java
HADOOP-16294: Enable access to input options by DistCp subclasses.

Adding a protected-scope getter for the DistCpOptions, so that a subclass does

not need to save its own copy of the inputOptions supplied to its constructor,

if it wishes to override the createInputFileListing method with logic similar

to the original implementation, i.e. calling CopyListing#buildListing with a path and input options.

Author: Andrew Olson

(cherry picked from commit c15b3bca86a0f973ccdddd020f3ff2d5767ff1bd)

    • -0
    • +9
    ./main/java/org/apache/hadoop/tools/DistCp.java
HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena.

  1. … 41 more files in changeset.
HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki.

(cherry picked from commit bbdbc7a9a158f36955c2253acb0edb14219ccb04)

HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki.

HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki.

(cherry picked from commit bbdbc7a9a158f36955c2253acb0edb14219ccb04)

Conflicts:

hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm

HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki.

(cherry picked from commit bbdbc7a9a158f36955c2253acb0edb14219ccb04)

Conflicts:

hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm

(cherry picked from commit 7985d9b1ced4371f4cdc48ea74fbf120eab50309)

Conflicts:

hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm

HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki.

(cherry picked from commit bbdbc7a9a158f36955c2253acb0edb14219ccb04)

HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail.

Contributed by Siyao Meng

(cherry picked from commit ce4bafdf442c004b6deb25eaa2fa7e947b8ad269)

HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail.

Contributed by Siyao Meng

HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized.

Author: Andrew Olson

(cherry picked from commit faba3591d32f2e4808c2faeb9472348d52619c8a)

HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized.

Author: Andrew Olson

HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0.

Contributed by Kai Xie.

(cherry picked from commit a49cb4465e6849a4346dcfa6f4a235d6fde917d3)

HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0.

Contributed by Kai Xie.

HADOOP-15281. Distcp to add no-rename copy option. Contributed by Andrew Olson.

  1. … 1 more file in changeset.
HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on.

Contributed by Ranith Sardar.

(cherry picked from commit 546c5d70efebb828389f609a89b123c4ee51f867)