Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][RT] Fail Fast When Interrupted on Frame Write Attempt

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Before writing a frame to the next operator, check the thread

interrupted status and fail fast if the thread is interrupted.

This ensures that queries that were aborted will fail as soon

as they attempt to write a frame.

Change-Id: If4f154f11305e2f8286e0707b5b3adec905633a4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4203

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 1 more file in changeset.
[NO ISSUE][RT] Improve PreclusteredGroupWriter

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Modified PreclusteredGroupWriter to only save group fields

from a last tuple in a frame instead of the whole frame

- move PermutingFrameTupleReference and PermutingTupleReference

from 'hyracks-storage-am-common' to 'hyracks-dataflow-common'

Change-Id: Ic75de2e6b64d0aacaf48096ecc9d47fc8e95c9cf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3351

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 37 more files in changeset.
[NO ISSUE][OTH] Support log redaction

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Support log redaction.

Change-Id: I602c833ba2a055da8fbe8782ec62be683ff4581b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3105

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 28 more files in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 983 more files in changeset.
[ASTERIXDB-1564][CONF] Consistently Use Log4j2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace java.util.logging by Log4j2.

- Excluded classes due to their tests:

- IoUtil

- Tracer

Change-Id: Ic137571292f45de1f1994c61d328b97185012197

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2226

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 388 more files in changeset.
[ASTERIXDB-2132] Replace ITracer.check

- user model changes: no

- storage format changes: no

- interface changes: yes

Trace categories in the ITracer interface are represented by long

values where each bit represents a category.

Introduce ITraceCategoryRegistry, an interface for a registry/mapping

from Strings to bit positions representing categories.

- add simple tests for Tracer and TraceCategoryRegistry

Change-Id: I3f26ae988d3786ef4ddb17e8fe78f911e7e628f1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2087

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 16 more files in changeset.
[NO ISSUE][OTH] Appender flush call with tracing call normal flush

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- The flush with tracing now calls the normal flush. This enables

sub appenders overriding the flush call to maintain correctness.

Change-Id: I3f649798fa4cac049f66cc3621acdb28b1c94694

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2080

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2 more files in changeset.
Introduce ITracer

Change-Id: I1d41d9cf74f481ba26882cf2ca318d0d2b9607f7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2050

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 23 more files in changeset.
[NO ISSUE] Add performance traces for frame writes

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Add traces for writes to end of local pipeline.

Change-Id: Ib32f4122fdddff1d0dce282a99829e0e0ad820e9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2047

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2 more files in changeset.
[ASTERIXDB-2008][CLUS] Only add pending removal if node known

[ASTERIXDB-2023][ING] Introduce Enums instead of using bytes

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Only nodes which are known to cluster manager are added

to the list of nodes pending removal. Other nodes are ignored

- Enums introduced:

- ActiveEvent.Kind

- ActivePartitionMessage.Event

- Remove AdapterRuntimeManager

- Remove AdapterExecutor

Change-Id: I7044896559798426c04a3f46861bc5335b25d140

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1921

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 102 more files in changeset.
Cleanup and bug fixes in Feeds pipeline

The bug fix is for the MessagingFrameTupleAppender. It used

to consume one extra byte per frame.

Cleanups are for feeds. These include:

1. Remove the use of feed dataflow marker. Feeds which

need to send progress can and should do that without

a marker thread.

2. Lock the memory component for feed commit to be able

to add information to the memory component's metadata

page safely.

In addition, this change introduces a frame level callback

for index operations.

Change-Id: Ie97b2133ebecb7380cf0ba336e60ed714d06f8ee

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1523

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 72 more files in changeset.
ASTERIXDB-1791, ASTERIXDB-1796: fix failure handling in runtime operators.

This change includes the following parts:

- Fix the implementation of fail() and close() in several runtime operators

to avoid file handle leak and job hang;

- Add an erase method to RunFileWriter which closes files before deleting

them in order release the holding disk space;

- Call RunFileWriter.close() and RunFileReader.close() in "finally" blocks.

- Fix RunFileReader to not truncate files to be deleted - it is not the root

cause of un-released disk space - open deleted files are the root cuase;

- Check file handle leaks in LangExecutionUtil.tearDown().

Change-Id: I203168171e6dac16b57d2eda960823e3810e22a3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1513

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 53 more files in changeset.
ASTERIXDB-1556, ASTERIXDB-1733: Hash Group By and Hash Join conform to the memory budget

- External Hash Group By and Hash Join now conform to the memory budget (compiler.groupmemory and compiler.joinmemory)

- For Optimzed Hybrid Hash Join, we calculate the expected hash table size when the build phase is done and

try to spill one or more partitions if the freespace can't afford the hash table size.

- For External Hash Group By, the number of hash entries (hash table size) is calculated based on

an estimation of the aggregated tuple size and possible hash values for the given field size in that tuple.

- Garbage Collection feature has been added to SerializableHashTable. For external hash group-by,

whenever we spill a data partition to the disk, we also check the ratio of garbage in the hash table.

If it's greater than the given threshold, we conduct a GC on Hash Table.

Change-Id: I2b323e9a2141b4c1dd1652a360d2d9354d3bc3f5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1056

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -2
    • +28
    ./io/FixedSizeFrameTupleAppender.java
  1. … 41 more files in changeset.
Fix Upsert Pipeline

Change-Id: I5c19d448f9664ecaeac600668a6dbdcf40673c56

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1308

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 9 more files in changeset.
Deletable frame tuple appender with reuse of tuple index slots

  1. … 1 more file in changeset.
Add Test NodeController, Test Data Generator, and Marker Logs

This test enable creating a node controller for unit test purposes.

The Node controller is identical to the regular node controller

except that it doesn't communicate with a cluster controller at all.

In this change, Test Data Generator is introduced which should

facilitate writing unit test cases which requires data generation.

The change also includes enabling feeds to send progress data. progress

information can then be sent through the pipeline and persisted in the

transaction logs and primary index component. A Unit test case has

been created to test adding progress markers to logs and index

components and then reading them.

The last part of this change is the addition of marker logs and their

callbacks. They enable components to create arbitrary logs and get a

callback when they are written to the transaction logs. Initial set of

unit tests were added for marker logs.

Change-Id: I3b9aa8de758b7d26ca34868b16e5ce693e0c0243

Reviewed-on: https://asterix-gerrit.ics.uci.edu/962

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 81 more files in changeset.
Eliminate Hard Tabs From Java Files

tabs->spaces

Change-Id: I0dfbeaa9b9b24d3da106ead85afd9df02b14ddaa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/921

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 66 more files in changeset.
Improve Messaging Connector and Add Unit Tests

Before this change, messaging connector always reserves 100 bytes

for messages which are mostly un-used. With this change, it only

reserves two bytes and sends null messages by default. In case a

new message doesn't fit in the leftover space of a frame, it sends

the frame with a null message, followed by a dedicated frame for

the message.

Change-Id: If4336e9c234e8d282798cfba9f48432b46cccfca

Reviewed-on: https://asterix-gerrit.ics.uci.edu/880

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -13
    • +0
    ./io/FrameTupleAppenderWrapper.java
  1. … 22 more files in changeset.
ASTERIXDB-1436: Big Object Support For Storage

Preliminary submit for big object storage support (values only).

Enable hyracks frames up to Integer.MAX_VALUE

Change-Id: Ie0a3cb855768dfd0cd1da4a5fc9f62aedebcbc5f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/840

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 94 more files in changeset.
Move Hyracks to subfolder

    • -0
    • +113
    ./io/AbstractFrameAppender.java
    • -0
    • +178
    ./io/ArrayTupleBuilder.java
    • -0
    • +59
    ./io/ArrayTupleReference.java
    • -0
    • +32
    ./io/FixedSizeFrameTupleAppender.java
    • -0
    • +101
    ./io/FrameDeserializer.java
    • -0
    • +79
    ./io/FrameDeserializingDataReader.java
    • -0
    • +66
    ./io/FrameDeserializingDataWriter.java
    • -0
    • +127
    ./io/FrameFixedFieldAppender.java
    • -0
    • +148
    ./io/FrameFixedFieldTupleAppender.java
    • -0
    • +63
    ./io/FrameOutputStream.java
    • -0
    • +249
    ./io/FrameTupleAccessor.java
    • -0
    • +273
    ./io/FrameTupleAppender.java
    • -0
    • +135
    ./io/FrameTupleAppenderAccessor.java
    • -0
    • +102
    ./io/FrameTupleAppenderWrapper.java
    • -0
    • +45
    ./io/ResultFrameTupleAccessor.java
  1. … 4413 more files in changeset.