Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Merge branch 'gerrit/mad-hatter'

Change-Id: If1c03edce783ccd249d90383da938132ae654886

    • -22
    • +21
    ./std/join/OptimizedHybridHashJoin.java
  1. … 11 more files in changeset.
[ASTERIXDB-2688][HYR] Fix use of a Hyracks task across join stages

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

In hash join, a task from the build stage is being used in the probe stage.

This is a problem since such tasks have already finished and notified the CC they are done.

One observed issue is related to issuing a warning in the probe phase where some warnings are

not reported because they are issued to tasks that have finished (the way this happened is that

a comparator was created in the build phase using the build-phase task. Then, this comparator

was used in the probe phase and issued a warning).

- make IHyracksJobletContext extend IHyracksCommonContext so that it is also a frame manager context

- make activites of join operators use the joblet context instead of the task context for acquiring buffers

- create the probe-to-build comparator in the probe phase so that the right task is used in the comparator

Change-Id: I38a4a779b9620494f15606162f0f1e9487fd0984

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4563

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Michael Blow <mblow@apache.org>

    • -19
    • +24
    ./std/join/InMemoryHashJoin.java
    • -30
    • +19
    ./std/join/InMemoryHashJoinOperatorDescriptor.java
    • -8
    • +7
    ./std/join/NestedLoopJoinOperatorDescriptor.java
    • -22
    • +21
    ./std/join/OptimizedHybridHashJoin.java
    • -22
    • +22
    ./std/join/OptimizedHybridHashJoinOperatorDescriptor.java
  1. … 6 more files in changeset.
Merge branch 'gerrit/stabilization-f69489' into 'gerrit/mad-hatter'

Change-Id: I17edb6b03563ac527bcad39f93193067ee42a33d

    • -0
    • +18
    ./std/join/OptimizedHybridHashJoinOperatorDescriptor.java
    • -1
    • +1
    ./std/sort/AbstractExternalSortRunMerger.java
  1. … 1 more file in changeset.
[ASTERIXDB-2686][RT] Delete run files eagerly

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Delete run files eagerly.

Change-Id: Idc780b44b0554bf36e2378700aae1b38b8e57353

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4403

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +18
    ./std/join/OptimizedHybridHashJoinOperatorDescriptor.java
    • -1
    • +1
    ./std/sort/AbstractExternalSortRunMerger.java
  1. … 7 more files in changeset.
[ASTERIXDB-2656] Remove frame constraint in HHJ probe

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

During the probe phase, we should not have the 1-frame

limit for spilled partitions. It wastes the memory and

leads to flushing small objects directly to the disk as

a large object.

Change-Id: I7580ecbe7db36d018348cd82d3864ca6c2ff439d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3703

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +1
    ./std/join/OptimizedHybridHashJoin.java
[ASTERIXDB-2532][RT] per-operator profiling

Enables profiling in queries at the operator-level when the analyze

variable is set in a query.

Change-Id: Ie16f3901ae5b32920d8552d5fd1ec8bb6e2ec8ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3226

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +10
    ./std/join/OptimizedHybridHashJoinOperatorDescriptor.java
    • -0
    • +5
    ./std/result/ResultWriterOperatorDescriptor.java
    • -2
    • +13
    ./std/sort/AbstractSorterOperatorDescriptor.java
    • -3
    • +6
    ./std/sort/ExternalSortOperatorDescriptor.java
    • -0
    • +51
    ./std/sort/TimedRunGenerator.java
    • -3
    • +12
    ./std/sort/TopKSorterOperatorDescriptor.java
  1. … 53 more files in changeset.
[NO ISSUE][COMP] Introduce connector for partial broadcasts

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduce MToNPartialBroadcastConnectorDescriptor

- Refactor FieldRangePartitionComputerFactory to

use RangeMapSupplier

Change-Id: I4a6f8f17d1709862300db7ab386161b4dfbfee5a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3524

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +187
    ./std/connectors/AbstractPartitionDataWriter.java
    • -0
    • +50
    ./std/connectors/MToNPartialBroadcastConnectorDescriptor.java
    • -0
    • +54
    ./std/connectors/MultiPartitionDataWriter.java
    • -143
    • +7
    ./std/connectors/PartitionDataWriter.java
  1. … 10 more files in changeset.
[ASTERIXDB-2577][RT] One frame per spilled partition in optimized hybrid hash join

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Makes sure there is one frame for each spilled partition during the build close.

Checks the length of large record before flushing a record to the disk blindly

as being large.

Change-Id: I82d4da57e9a9835cab61dc5cc43c45c728e0c2b6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3485

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -220
    • +182
    ./std/join/OptimizedHybridHashJoin.java
  1. … 1 more file in changeset.
[NO ISSUE][API] Return Async Query Results As JSON

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Add ResultMetadata that can hold metadata about a job's

result.

- Add required CC functions to get ResultMetadata.

- Add IJobResultCallback that is called when a job's result

is written successfully.

- Calculate a job's processed objects on the job completion.

- Calculate a job's duration on the job completion.

- Use the async query request format when serving the result

from QueryResultApiServlet.

- Return a proper JSON envelop in QueryResultApiServlet

which includes the results as well as metrics fields.

- Add test case for returning JSON response for async

query.

Change-Id: Ic4812a14925099a677f9e77a0040f881d2600724

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3401

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -5
    • +6
    ./std/result/ResultWriterOperatorDescriptor.java
  1. … 59 more files in changeset.
[ASTERIXDB-2555][RT][COMP] Make hash join use logical comparison

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This patch changes the hash join operator to use the join condition

to evaluate if tuples are equal when joining. Binary physical comparators

have been removed. The join condition evaluator is in TuplePairEvaluator.

- extraced TuplePairEvaluatorFactory out of nested loop join class

into a separate class so that it is shared among nested loop and

hash join.

- switched from FrameTuplePairComparator to ITuplePairComparator in

in OptimizedHybridHashJoin and InMemoryHashJoin.

- moved debugging code from OptimizedHybridHashJoin into a separate

class, JoinUtil.

- temporarily made the logical comparison of multisets use raw binary

comparison instead of returning null until the logic is implemented.

- made IBinaryBooleanInspector a functional interface and updated

the implementations.

- updated record and array test cases to reflect the new

behaviour of hash join where logical comparison could produce null.

Also, updated sorting, group by and distinct test cases since

the input data has been modified.

- added two new input files arrays1nulls.adm & arrays2nulls.adm

to be used by the open dataset. previous arrays1.adm & arrays2.adm

are used by the closed dataset since it cannot accept arrays with

null values.

Change-Id: If1834967fdd913fdc76003f09636b2450d07cd5e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3387

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -0
    • +99
    ./std/join/HybridHashJoinUtil.java
    • -17
    • +13
    ./std/join/InMemoryHashJoinOperatorDescriptor.java
    • -86
    • +10
    ./std/join/OptimizedHybridHashJoin.java
    • -59
    • +30
    ./std/join/OptimizedHybridHashJoinOperatorDescriptor.java
  1. … 44 more files in changeset.
[NO ISSUE][HYR][RT] Handle calls to close() properly in hash join

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Hash join build activity assumes that upon calling close()

the join state must have been allocated in open() and

therefore, it will directly call "close build" using the join

state. However, if the build activity throws an exception

in open(), then it could happen that the join state has

not been initialized. Calling close() on the build activity

will lead to a NPE since it will use the join state directly.

This patch checks if the join state has been initialized. If

so, it will close the build as usual.

Change-Id: I41fff7deed5e56818bfefb4d36b8018a9f512cbb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3375

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -7
    • +9
    ./std/join/OptimizedHybridHashJoinOperatorDescriptor.java
[ASTERIXDB-2552][RT] Reuse objects on reseting buffer manager

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

The "VariableFrameMemoryManager" releases the physical frames

objects, the logical frames objects and the free slots objects

on reset(). This patch is to retain those objects and only

release them on close() (as also stated in the interface).

- added close() to IFrameFreeSlotPolicy to make the interface similar

to the other interfaces (IFrameBufferManager, IFramePool, ... etc)

where close() is supposed to release while reset() is supposed to only

reset but retain resources.

- renamed "InMemorySortRuntimeFactory" to "MicroSortRuntimeFactory".

Change-Id: I88bea27e5024c621412ef609475e2a7ba1913afa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3372

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 4 more files in changeset.
[ASTERIXDB-2552][RT] Implement micro external sort for subplans

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This patch is to change the in memory sort used in subplans to

be a micro external sort to avoid out of memory error.

- added reset() to the runs merger to allow reusing the runs merger.

- renamed "InMemoryStableSortPOperator" to "MicroStableSortPOperator".

- changed the tag from "IN_MEMORY_STABLE_SORT" to "MICRO_STABLE_SORT".

- added test cases.

Change-Id: I930849d644c60d461d2869c9773b85e49b46fbdb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3353

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -6
    • +23
    ./std/sort/AbstractExternalSortRunMerger.java
  1. … 33 more files in changeset.
[ASTERIXDB-2552][RT] Refactor runs generator and merger

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Moved the writer and sorter out of the merger to allow

micro external sort to use the run generator and merger

plus minor clean-ups.

Change-Id: Idda31c92cbcddba5ebef8bbbf7855b9c8293dd51

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3363

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -118
    • +73
    ./std/sort/AbstractExternalSortRunMerger.java
    • -22
    • +1
    ./std/sort/AbstractFrameSorter.java
    • -8
    • +16
    ./std/sort/AbstractSortRunGenerator.java
    • -19
    • +40
    ./std/sort/AbstractSorterOperatorDescriptor.java
    • -8
    • +6
    ./std/sort/ExternalSortOperatorDescriptor.java
    • -11
    • +8
    ./std/sort/FrameSorterQuickSort.java
    • -1
    • +1
    ./std/sort/HybridTopKSortRunGenerator.java
    • -8
    • +6
    ./std/sort/TopKSorterOperatorDescriptor.java
  1. … 1 more file in changeset.
[NO ISSUE][COMP] Refactor physical operators for GroupBy and DistinctBy

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduce AbstractGroupByPOperator -

base class for physical GroupBy operators

- Introduce AbstractDistinctByPOperator -

base class for physical DistinctBy operators

- Set memory limit for MicroPreclusteredGroupByPOperator

- Remove unused PhysicalOperatorTag.HASH_GROUP_BY

- Rename GroupByOperator.getGbyVarList() to getGroupByVarList()

- Remove constructors that do not specify memory limit from

PreclusteredGroupOperatorDescriptor and PreclusteredGroupWriter

Change-Id: I93f17a6a3e0df6587e518e18eea01ca54a289b4c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3364

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 27 more files in changeset.
[NO ISSUE] Create an abstraction for the ForwardOperatorDescriptor

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Abstract the activities out of the ForwardOperatorDescriptor so we can reuse

the same basic framework for multiple forward Operators.

- Abstract the ForwardOperatorDescriptor out of the ForwardPOperator so we can

reuse the same basic framework for multiple forward Operators.

Change-Id: Icc3db4b386e69a98c2a1c40dadc96eb3e1a5d4fa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3338

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +73
    ./std/base/AbstractForwardOperatorDescriptor.java
    • -250
    • +0
    ./std/misc/ForwardOperatorDescriptor.java
    • -0
    • +232
    ./std/misc/SortForwardOperatorDescriptor.java
  1. … 15 more files in changeset.
[NO ISSUE][RT] Improve PreclusteredGroupWriter

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Modified PreclusteredGroupWriter to only save group fields

from a last tuple in a frame instead of the whole frame

- move PermutingFrameTupleReference and PermutingTupleReference

from 'hyracks-storage-am-common' to 'hyracks-dataflow-common'

Change-Id: Ic75de2e6b64d0aacaf48096ecc9d47fc8e95c9cf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3351

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 36 more files in changeset.
[NO ISSUE][RT] Window operator performance improvement

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Improve performance of window operator: save only

last values of partitioning columns from the

previous frame instead of the whole frame

Change-Id: Ib8d6b3b99ee24c73b76fd118040ed6972e6798d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3346

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 7 more files in changeset.
[ASTERIXDB-2516][RT] Modifty range map function of parallel sort

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Currently, the range map function of parallel sort passes "null"

to get a comparator to sort the samples and produce the range map.

The comparator provider will enforce providing a type and prevent

passing "null". The range map function needs to be updated to get

the types of the sort fields and use them to get comparators.

- changed the output type of the local sampling function from

list of list of ANY to binary. The old type computer was removed.

- added null writer aggregate function that just produces null as

an aggregate value. this is needed in order to propagate the type

of the sort fields from the local step to the global step so that

the range map function can know the types of the sort fields.

Change-Id: I7edbb10906cc4464210af87a5b1630ba3aecbde0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3333

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -2
    • +6
    ./std/misc/ForwardOperatorDescriptor.java
  1. … 13 more files in changeset.
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -3
    • +4
    ./std/base/AbstractConnectorDescriptor.java
    • -1
    • +1
    ./std/sort/AbstractExternalSortRunGenerator.java
    • -1
    • +1
    ./std/sort/util/GroupFrameAccessor.java
  1. … 622 more files in changeset.
[NO ISSUE][RT] Eliminate Excessive Object Creation in Sort Merge

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently, every binary search for frame lookup during

the sort merge phase results in auto-boxing to an Integer.

In addition, every call to InnerFrameInfo#compareTo during

the binary search results another auto-boxing. This change

eliminates these objects creation by using a MutableInt and

and using a primitive int comparison while comparing.

Change-Id: I72b498d476d242f85966caf67d2543978e5c88bb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3276

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -6
    • +9
    ./std/sort/util/GroupFrameAccessor.java
[ASTERIXDB-2523][RT][COMP] add support for hashing array fields

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Add support for hashing array fields.

- Modified AMurmurHash3BinaryHashFunctionFamily and extracted the hashing function

into a private named hashing function "GenericHashFunction". Added hashing arrays.

- Modified hash join to include generating hash functions for the right branch

since now hash functions are type-dependent and cannot use the same hash functions

generated for the left branch.

- Added test cases.

Change-Id: Ibd0dc7f270730140226f54445705822049f5c863

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3241

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -11
    • +15
    ./std/join/InMemoryHashJoinOperatorDescriptor.java
    • -12
    • +15
    ./std/join/OptimizedHybridHashJoinOperatorDescriptor.java
  1. … 53 more files in changeset.
[ASTERIXDB-2516][RT] prepare physical comparators for deep comparison

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

This change is to make physical comparators type-aware in order to do

deep comparison of complex types like arrays and records. The IAType

is propagated to the comparators.

- added new methods in IBinaryComparatorFactoryProvider to accept the

type of left and right inputs for operations like hash join where

the join key types come from different dataset sources.

- defaulted some arrays functions to use the old comparator behaviour temporarily

until complex comparison is implemented

- modified AObjectAscBinaryComparatorFactory & AObjectDescBinaryComparatorFactory to

create a comparator with IAType information. Changed the serialization/deserialization

of their instances to take care of the newly added fields since they are not

present in old instances.

Change-Id: I02011e7151398d5f5f9ba9c1e1db6518484b9fe5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3229

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -25
    • +32
    ./std/join/OptimizedHybridHashJoinOperatorDescriptor.java
  1. … 24 more files in changeset.
[NO ISSUE][RT] Replace HybridHashJoin with OptimizedHybridHashJoin

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

The HybridHashJoinOperator is an old implenetation which haven't been

used in the runtime, and it lacks necessary documentation and memory

management. The OptimizedHybridHashJoinOperatorDescriptor serves the

same purpose. We should use this instead and avoid maintaining the old

one.

Change-Id: I6ed612cc233af1b78d453c7b711077b82e721e82

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3023

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Xikui Wang <xkkwww@gmail.com>

    • -577
    • +0
    ./std/join/HybridHashJoinOperatorDescriptor.java
  1. … 3 more files in changeset.
[ASTERIXDB-2286][COMP] Parallel sort changes p.2

- user model changes: no

- storage format changes: no

- interface changes: no

details:

This patch is change the way the SequentialMergeExchangePOperator

connector computes its local property instead of blindly

propagating the child's local property.

The patch also includes minor code clean-ups (moved some methods down)

Change-Id: Ie37e03b6fc6e55fc21f8324c0f09a7fa05b51769

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3005

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 7 more files in changeset.
[ASTERIXDB-2466][FUN] Implement window functions

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Implement window functions with SQL syntax:

function() OVER ((PARTITION BY expr1, expr2, ...)? ORDER BY exprA, exprB, ...)

- Where supported functions are:

ROW_NUMBER(), RANK(), DENSE_RANK(), PERCENT_RANK(), NTILE()

Change-Id: Ia28af8773cb11049c38d440c51b9c3cd1ed2bab4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3002

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 167 more files in changeset.
[ASTERIXDB-2286][COMP][FUN][HYR] Parallel Sort Optimization

- user model changes: yes

- storage format changes: no

- interface changes: yes

details:

- new plan for sort operation which includes sampling and

replicating the stream of data to be sorted. Sort-merge connector

is removed from the plan. The sorted result now is in multiple partitions.

- new optimization rule to check whether full parallel sort is applicable.

- new Forward operator to read the replicated sort input stream and

to receive the ouput of the sampling.

- new sequential merge connector to merge a globally ordered result residing

in multiple partitions (in addition to the connector's partition computer).

- "asterix-lang-aql/pom.xml" is changed as a result of refactoring

code related to the range map handling.

- new private sampling function to generate the range map object

(local & global functions) & their type computers.

user model changes:

- new compiler property is added to enable and disable parallel sort.

interface changes:

- "ILogicalOperatorVisitor.java" includes Forward Operator.

- "ITuplePartitionComputer.java" includes initialize() to enable partitioner

to do some initialization. FieldRangePartitionComputerFactory uses it to

pick a range map.

- "ITuplePartitionComputerFactory.java". createPartitioner() is changed to

createPartitioner(IHyracksTaskContext hyracksTaskContext). Context is needed

for transferring the range map throught the context.

Change-Id: I73e128029a46f45e6b68c23dfb9310d5de10582f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2393

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +18
    ./std/base/AbstractMToNConnectorDescriptor.java
    • -0
    • +1
    ./std/base/AbstractReplicateOperatorDescriptor.java
    • -0
    • +80
    ./std/collectors/DeterministicPartitionBatchManager.java
    • -0
    • +77
    ./std/collectors/SequentialMergeFrameReader.java
    • -1
    • +1
    ./std/connectors/LocalityAwareMToNPartitioningConnectorDescriptor.java
    • -18
    • +1
    ./std/connectors/MToNPartitioningConnectorDescriptor.java
    • -1
    • +1
    ./std/connectors/MToNPartitioningMergingConnectorDescriptor.java
    • -1
    • +1
    ./std/connectors/MToNPartitioningWithMessageConnectorDescriptor.java
    • -0
    • +65
    ./std/connectors/MToOneSequentialMergingConnectorDescriptor.java
    • -0
    • +1
    ./std/connectors/PartitionDataWriter.java
    • -6
    • +6
    ./std/join/HybridHashJoinOperatorDescriptor.java
    • -2
    • +2
    ./std/join/InMemoryHashJoinOperatorDescriptor.java
    • -0
    • +246
    ./std/misc/ForwardOperatorDescriptor.java
    • -3
    • +6
    ./std/sort/AbstractExternalSortRunMerger.java
  1. … 343 more files in changeset.
[NO ISSUE][OTH] Remove Unused Imports

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: Iafff39073d0fedaff74a26ef7e3260008a79ff0c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2893

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 62 more files in changeset.
[NO ISSUE][NET] Skip Channel Write on Connection Failure

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not attempt to adjust channel writability if

the failure was due to connection failure since

no more messages will be sent on that connection.

This is done to prevent a possible deadlock between

network IOThread that detected connection failure

and another thread that might be accessing the channel.

- Make sending error code conditions more explicit since

we currently have a single error code that is sent.

Change-Id: Ic25f05ac2c0d02699324f2d1b80c51f392654106

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2892

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 4 more files in changeset.
[NO ISSUE][OTH] Cleanup Redundant Type Cast

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: I9c93ce90ad4aa789ae46284431e172ae4777bf55

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2887

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 30 more files in changeset.