Clone Tools
  • last updated 10 mins ago
Constraints: committers
Constraints: files
Constraints: dates
ASTERIXDB-1791, ASTERIXDB-1796: fix failure handling in runtime operators.

This change includes the following parts:

- Fix the implementation of fail() and close() in several runtime operators

to avoid file handle leak and job hang;

- Add an erase method to RunFileWriter which closes files before deleting

them in order release the holding disk space;

- Call RunFileWriter.close() and RunFileReader.close() in "finally" blocks.

- Fix RunFileReader to not truncate files to be deleted - it is not the root

cause of un-released disk space - open deleted files are the root cuase;

- Check file handle leaks in LangExecutionUtil.tearDown().

Change-Id: I203168171e6dac16b57d2eda960823e3810e22a3


Tested-by: Jenkins <>

Integration-Tests: Jenkins <>

Reviewed-by: Till Westmann <>

  1. … 51 more files in changeset.
Truncate a temporary file before deleting it.

- Before RunFileReader tries to delete a temporary file,

make sure to truncate the file since OS might keep the

deleted file for a while, thus taking disk space.

Change-Id: Ie906c9f950e2f31af6f1b5ecc9cb35829d3edf8a


Sonar-Qube: Jenkins <>

Reviewed-by: Yingyi Bu <>

Tested-by: Jenkins <>

Integration-Tests: Jenkins <>

  1. … 3 more files in changeset.
Introduce IStorageComponentProvider

Change-Id: If86750cdb2436c713f6598e54d4aaaf23d9f7bbf


Tested-by: Jenkins <>

Reviewed-by: Yingyi Bu <>

Integration-Tests: Jenkins <>

  1. … 419 more files in changeset.
ASTERIXDB-1556, ASTERIXDB-1733: Hash Group By and Hash Join conform to the memory budget

- External Hash Group By and Hash Join now conform to the memory budget (compiler.groupmemory and compiler.joinmemory)

- For Optimzed Hybrid Hash Join, we calculate the expected hash table size when the build phase is done and

try to spill one or more partitions if the freespace can't afford the hash table size.

- For External Hash Group By, the number of hash entries (hash table size) is calculated based on

an estimation of the aggregated tuple size and possible hash values for the given field size in that tuple.

- Garbage Collection feature has been added to SerializableHashTable. For external hash group-by,

whenever we spill a data partition to the disk, we also check the ratio of garbage in the hash table.

If it's greater than the given threshold, we conduct a GC on Hash Table.

Change-Id: I2b323e9a2141b4c1dd1652a360d2d9354d3bc3f5


Tested-by: Jenkins <>

BAD: Jenkins <>

Integration-Tests: Jenkins <>

Reviewed-by: Yingyi Bu <>

  1. … 40 more files in changeset.
Optimize PartitionWriter

Change-Id: Ic36c58b143f3fc2d37b180559c11c1566bcc1a86


Sonar-Qube: Jenkins <>

Tested-by: Jenkins <>

Integration-Tests: Jenkins <>

Reviewed-by: Yingyi Bu <>

  1. … 2 more files in changeset.
Fix Upsert Pipeline

Change-Id: I5c19d448f9664ecaeac600668a6dbdcf40673c56


Reviewed-by: Ian Maxon <>

Tested-by: Jenkins <>

Reviewed-by: Till Westmann <>

Integration-Tests: Jenkins <>

  1. … 8 more files in changeset.
working partition join.

  1. … 6 more files in changeset.
updated partition algorithm with build only side memory

  1. … 3 more files in changeset.
Deletable frame tuple appender with reuse of tuple index slots

  1. … 1 more file in changeset.
snapshot for range state transition

  1. … 67 more files in changeset.
Add Test NodeController, Test Data Generator, and Marker Logs

This test enable creating a node controller for unit test purposes.

The Node controller is identical to the regular node controller

except that it doesn't communicate with a cluster controller at all.

In this change, Test Data Generator is introduced which should

facilitate writing unit test cases which requires data generation.

The change also includes enabling feeds to send progress data. progress

information can then be sent through the pipeline and persisted in the

transaction logs and primary index component. A Unit test case has

been created to test adding progress markers to logs and index

components and then reading them.

The last part of this change is the addition of marker logs and their

callbacks. They enable components to create arbitrary logs and get a

callback when they are written to the transaction logs. Initial set of

unit tests were added for marker logs.

Change-Id: I3b9aa8de758b7d26ca34868b16e5ce693e0c0243


Tested-by: Jenkins <>

Integration-Tests: Jenkins <>

Reviewed-by: Michael Blow <>

Reviewed-by: abdullah alamoudi <>

  1. … 79 more files in changeset.
Eliminate Hard Tabs From Java Files


Change-Id: I0dfbeaa9b9b24d3da106ead85afd9df02b14ddaa


Tested-by: Jenkins <>

Reviewed-by: Till Westmann <>

  1. … 66 more files in changeset.
sonar update

  1. … 45 more files in changeset.
Improve Messaging Connector and Add Unit Tests

Before this change, messaging connector always reserves 100 bytes

for messages which are mostly un-used. With this change, it only

reserves two bytes and sends null messages by default. In case a

new message doesn't fit in the leftover space of a frame, it sends

the frame with a null message, followed by a dedicated frame for

the message.

Change-Id: If4336e9c234e8d282798cfba9f48432b46cccfca


Reviewed-by: Murtadha Hubail <>

Reviewed-by: Jenkins <>

Tested-by: Jenkins <>

  1. … 21 more files in changeset.
Cleanup Feed CodeBase

In order to expedite cleaning feeds up and reaching a maintainable

state, we disabled the following:

1. Policies (At least once, throttling, discarding, elasticity).

2. Statistics Reporting.

3. Load management.

4. Feed re-activation upon System reboot.

Right now on master, none of these features work reliably.

We will re-introduce them one feature at a time.

The rules followed in this change:

1. Keep X if X is tested in a test case.

2. Remove X if X is not used in test cases.

After a few meetings with Mike and Till, the policies

1. Buffering

2. Spill

3. Discard

4. Throttle

have been fixed and unit tests have been added.

Change-Id: I545bc4f8560564e4c868a80d27c77a4edd97a8b8


Tested-by: Jenkins <>

Reviewed-by: abdullah alamoudi <>

  1. … 141 more files in changeset.
ASTERIXDB-1436: Big Object Support For Storage

Preliminary submit for big object storage support (values only).

Enable hyracks frames up to Integer.MAX_VALUE

Change-Id: Ie0a3cb855768dfd0cd1da4a5fc9f62aedebcbc5f


Tested-by: Jenkins <>

Reviewed-by: abdullah alamoudi <>

  1. … 91 more files in changeset.
Squashed changes for interval index join.

  1. … 53 more files in changeset.
moved files for the merge

  1. … 176 more files in changeset.
great merge - hyracks

  1. … 63 more files in changeset.
Move Hyracks to subfolder

  1. … 4413 more files in changeset.