Clone Tools
  • last updated 25 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2532][RT] per-operator profiling

Enables profiling in queries at the operator-level when the analyze

variable is set in a query.

Change-Id: Ie16f3901ae5b32920d8552d5fd1ec8bb6e2ec8ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3226

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -5
    • +10
    ./operators/aggreg/NestedPlansRunningAggregatorFactory.java
    • -0
    • +5
    ./operators/meta/AlgebricksMetaOperatorDescriptor.java
    • -3
    • +4
    ./operators/meta/PipelineAssembler.java
    • -1
    • +1
    ./operators/std/AssignRuntimeFactory.java
  1. … 57 more files in changeset.
[NO ISSUE][COMP] Support runtime warnings during constant folding

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add IWarningCollector accessor to IEvaluatorContext

- Provide compiler's warning collector to function evaluators

during constant folding, so they can emit warnings

Change-Id: I61dd53adfbcad2f8479333579185258876f074e9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3484

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 8 more files in changeset.
[NO ISSUE][FUN] Introduce IEvaluatorContext

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduce IEvaluatorContext - a context for evaluators

that is available both at runtime and during constant

folding at compile time

- IEvaluatorContext provides access to IHyracksTaskContext

which is only available at runtime. At compile time this

accessor returns null

- Instantiate evaluators with IEvaluatorContext

instead of IHyracksTaskContext

Change-Id: Iff90c7739e46d6602fcd0609e67f6a8cd9714ca5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3481

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -2
    • +2
    ./aggregators/TupleCountAggregateFunctionFactory.java
    • -2
    • +2
    ./aggregators/TupleCountRunningAggregateFunctionFactory.java
    • -2
    • +1
    ./base/IAggregateEvaluatorFactory.java
    • -0
    • +33
    ./base/IEvaluatorContext.java
    • -2
    • +1
    ./base/IRunningAggregateEvaluatorFactory.java
    • -2
    • +1
    ./base/ISerializedAggregateEvaluatorFactory.java
    • -2
    • +1
    ./base/IUnnestingEvaluatorFactory.java
    • -2
    • +2
    ./evaluators/ColumnAccessEvalFactory.java
    • -0
    • +37
    ./evaluators/EvaluatorContext.java
    • -2
    • +2
    ./evaluators/TupleFieldEvaluatorFactory.java
    • -3
    • +5
    ./evaluators/TuplePairEvaluatorFactory.java
    • -1
    • +4
    ./operators/aggreg/SerializableAggregatorDescriptorFactory.java
  1. … 675 more files in changeset.
[NO ISSUE][FUN] Support IGNORE NULLS in window functions

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Support IGNORE NULLS / RESPECT NULLS modifiers in window

  functions LEAD(), LAG(), FIRST_VALUE(), LAST_VALUE(), NTH_VALUE()

  (RESPECT NULLS is the default)

- Add testcases for each function

- Fix CollectionMemberResultType and PushAggregateIntoNestedSubplanRule

  to handle remaining non-list arguments

Change-Id: I400c9c95bac9159efa17fca4e97047fd089eb931

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3407

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -26
    • +47
    ./operators/win/WindowNestedPlansPushRuntime.java
    • -4
    • +9
    ./operators/win/WindowNestedPlansRuntimeFactory.java
  1. … 49 more files in changeset.
[ASTERIXDB-2555][RT][COMP] Make hash join use logical comparison

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This patch changes the hash join operator to use the join condition

to evaluate if tuples are equal when joining. Binary physical comparators

have been removed. The join condition evaluator is in TuplePairEvaluator.

- extraced TuplePairEvaluatorFactory out of nested loop join class

into a separate class so that it is shared among nested loop and

hash join.

- switched from FrameTuplePairComparator to ITuplePairComparator in

in OptimizedHybridHashJoin and InMemoryHashJoin.

- moved debugging code from OptimizedHybridHashJoin into a separate

class, JoinUtil.

- temporarily made the logical comparison of multisets use raw binary

comparison instead of returning null until the logic is implemented.

- made IBinaryBooleanInspector a functional interface and updated

the implementations.

- updated record and array test cases to reflect the new

behaviour of hash join where logical comparison could produce null.

Also, updated sorting, group by and distinct test cases since

the input data has been modified.

- added two new input files arrays1nulls.adm & arrays2nulls.adm

to be used by the open dataset. previous arrays1.adm & arrays2.adm

are used by the closed dataset since it cannot accept arrays with

null values.

Change-Id: If1834967fdd913fdc76003f09636b2450d07cd5e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3387

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -0
    • +150
    ./evaluators/TuplePairEvaluatorFactory.java
  1. … 48 more files in changeset.
[ASTERIXDB-2552][RT] Reuse objects on reseting buffer manager

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

The "VariableFrameMemoryManager" releases the physical frames

objects, the logical frames objects and the free slots objects

on reset(). This patch is to retain those objects and only

release them on close() (as also stated in the interface).

- added close() to IFrameFreeSlotPolicy to make the interface similar

to the other interfaces (IFrameBufferManager, IFramePool, ... etc)

where close() is supposed to release while reset() is supposed to only

reset but retain resources.

- renamed "InMemorySortRuntimeFactory" to "MicroSortRuntimeFactory".

Change-Id: I88bea27e5024c621412ef609475e2a7ba1913afa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3372

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -185
    • +0
    ./operators/sort/InMemorySortRuntimeFactory.java
    • -0
    • +185
    ./operators/sort/MicroSortRuntimeFactory.java
  1. … 9 more files in changeset.
[NO ISSUE][COMP] Refactor physical window operator

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Create a new physical operator (WindowStreamPOperator)

for window operators that do not require partition materialization

- Create AbstractWindowPOperator which is now a base

class for both physical window operators

- Rename WindowSimpleRuntime* to WindowStreamRuntime*

Change-Id: I3863fa3d298aef53d4098be9fc17b0451eb2c23e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3369

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -59
    • +0
    ./operators/win/WindowSimplePushRuntime.java
    • -54
    • +0
    ./operators/win/WindowSimpleRuntimeFactory.java
    • -0
    • +59
    ./operators/win/WindowStreamPushRuntime.java
    • -0
    • +55
    ./operators/win/WindowStreamRuntimeFactory.java
  1. … 11 more files in changeset.
[ASTERIXDB-2552][RT] Implement micro external sort for subplans

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This patch is to change the in memory sort used in subplans to

be a micro external sort to avoid out of memory error.

- added reset() to the runs merger to allow reusing the runs merger.

- renamed "InMemoryStableSortPOperator" to "MicroStableSortPOperator".

- changed the tag from "IN_MEMORY_STABLE_SORT" to "MICRO_STABLE_SORT".

- added test cases.

Change-Id: I930849d644c60d461d2869c9773b85e49b46fbdb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3353

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -40
    • +114
    ./operators/sort/InMemorySortRuntimeFactory.java
  1. … 33 more files in changeset.
[NO ISSUE][COMP] Refactor physical operators for GroupBy and DistinctBy

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduce AbstractGroupByPOperator -

base class for physical GroupBy operators

- Introduce AbstractDistinctByPOperator -

base class for physical DistinctBy operators

- Set memory limit for MicroPreclusteredGroupByPOperator

- Remove unused PhysicalOperatorTag.HASH_GROUP_BY

- Rename GroupByOperator.getGbyVarList() to getGroupByVarList()

- Remove constructors that do not specify memory limit from

PreclusteredGroupOperatorDescriptor and PreclusteredGroupWriter

Change-Id: I93f17a6a3e0df6587e518e18eea01ca54a289b4c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3364

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -2
    • +4
    ./operators/group/MicroPreClusteredGroupRuntimeFactory.java
  1. … 28 more files in changeset.
[NO ISSUE][RT] Improve PreclusteredGroupWriter

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Modified PreclusteredGroupWriter to only save group fields

from a last tuple in a frame instead of the whole frame

- move PermutingFrameTupleReference and PermutingTupleReference

from 'hyracks-storage-am-common' to 'hyracks-dataflow-common'

Change-Id: Ic75de2e6b64d0aacaf48096ecc9d47fc8e95c9cf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3351

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -6
    • +6
    ./operators/win/AbstractWindowPushRuntime.java
  1. … 37 more files in changeset.
[NO ISSUE][RT] Window operator performance improvement

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Improve performance of window operator: save only

last values of partitioning columns from the

previous frame instead of the whole frame

Change-Id: Ib8d6b3b99ee24c73b76fd118040ed6972e6798d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3346

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -16
    • +6
    ./operators/win/AbstractWindowNestedPlansPushRuntime.java
    • -17
    • +19
    ./operators/win/AbstractWindowPushRuntime.java
    • -6
    • +11
    ./operators/win/WindowNestedPlansPushRuntime.java
    • -3
    • +5
    ./operators/win/WindowNestedPlansRunningPushRuntime.java
  1. … 4 more files in changeset.
[NO ISSUE][COMP][RT] Handle type mismatch in window frame boundaries

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- If window frame boundary is defined as N preceding/following

and the incoming value is not of a numeric or temporal type

then assume that the frame is empty for this value because

its boundaries cannot be computed

- Add tests for NULL/MISSING/complex types for window operator

- Fix typo in the name resolution documentation

Change-Id: I4dc1b010674eb9a8b679039dc68c81163d156956

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3342

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -4
    • +47
    ./operators/win/AbstractWindowNestedPlansPushRuntime.java
    • -0
    • +36
    ./operators/win/IWindowAggregatorDescriptor.java
    • -6
    • +14
    ./operators/win/WindowAggregatorDescriptorFactory.java
    • -95
    • +155
    ./operators/win/WindowNestedPlansPushRuntime.java
    • -50
    • +95
    ./operators/win/WindowNestedPlansRunningPushRuntime.java
    • -5
    • +14
    ./operators/win/WindowNestedPlansRunningRuntimeFactory.java
    • -11
    • +23
    ./operators/win/WindowNestedPlansRuntimeFactory.java
  1. … 25 more files in changeset.
[NO ISSUE][RT] Make StreamSelectRuntimeFactory Extensible

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Make StreamSelectRuntimeFactory and its runtime

extensible by providing accessors and replacing

the runtime anonymous class by a named class.

Change-Id: I9c575e6c037e5c8c1818cfa3c6b0bf65697bfb9e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3308

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -65
    • +93
    ./operators/std/StreamSelectRuntimeFactory.java
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +0
    ./operators/aggreg/NestedPlansRunningAggregatorFactory.java
    • -1
    • +1
    ./operators/aggrun/AbstractRunningAggregatePushRuntime.java
  1. … 624 more files in changeset.
[NO ISSUE][RT] Window operator runtime optimization

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Improve memory management for window operators

- Add "compiler.windowmemory" property that specifies memory

budget for each window operator (default is 4MB, min is 160KB)

- Consolidated negative window operator testcases into a single one

Change-Id: I6756e92046883f79db339ef490cca8bc8b7b1fb8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3227

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -3
    • +4
    ./operators/aggrun/AbstractRunningAggregatePushRuntime.java
    • -1
    • +1
    ./operators/aggrun/RunningAggregatePushRuntime.java
    • -8
    • +10
    ./operators/win/AbstractWindowNestedPlansPushRuntime.java
    • -3
    • +3
    ./operators/win/AbstractWindowNestedPlansRuntimeFactory.java
    • -2
    • +13
    ./operators/win/AbstractWindowPushRuntime.java
    • -73
    • +32
    ./operators/win/WindowMaterializingPushRuntime.java
    • -2
    • +6
    ./operators/win/WindowMaterializingRuntimeFactory.java
    • -72
    • +46
    ./operators/win/WindowNestedPlansPushRuntime.java
    • -82
    • +55
    ./operators/win/WindowNestedPlansRunningPushRuntime.java
    • -3
    • +3
    ./operators/win/WindowNestedPlansRunningRuntimeFactory.java
    • -3
    • +3
    ./operators/win/WindowNestedPlansRuntimeFactory.java
    • -2
    • +5
    ./operators/win/WindowNestedPlansUnboundedPushRuntime.java
    • -3
    • +3
    ./operators/win/WindowNestedPlansUnboundedRuntimeFactory.java
    • -0
    • +42
    ./operators/win/WindowPartitionReader.java
    • -0
    • +365
    ./operators/win/WindowPartitionWriter.java
  1. … 44 more files in changeset.
[NO ISSUE][RT] Window operator runtime optimization

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Runtime optimization for window operators with accumulating

frames (unbounded preceding to current row or n following)

- Refactor window function properties into generic

builtin function properties

Change-Id: I8d1574defc73076ad960c4067432da29ead160a5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3151

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +110
    ./operators/aggreg/AggregatePushRuntime.java
    • -69
    • +2
    ./operators/aggreg/AggregateRuntimeFactory.java
    • -1
    • +1
    ./operators/aggreg/NestedPlansAccumulatingAggregatorFactory.java
    • -2
    • +2
    ./operators/aggreg/NestedPlansRunningAggregatorFactory.java
    • -3
    • +7
    ./operators/meta/PipelineAssembler.java
    • -1
    • +1
    ./operators/std/NestedTupleSourceRuntimeFactory.java
    • -0
    • +121
    ./operators/win/AbstractWindowNestedPlansPushRuntime.java
    • -0
    • +46
    ./operators/win/AbstractWindowNestedPlansRuntimeFactory.java
    • -10
    • +56
    ./operators/win/WindowAggregatorDescriptorFactory.java
    • -48
    • +6
    ./operators/win/WindowNestedPlansPushRuntime.java
    • -0
    • +240
    ./operators/win/WindowNestedPlansRunningPushRuntime.java
    • -0
    • +76
    ./operators/win/WindowNestedPlansRunningRuntimeFactory.java
    • -11
    • +6
    ./operators/win/WindowNestedPlansRuntimeFactory.java
    • -21
    • +5
    ./operators/win/WindowNestedPlansUnboundedPushRuntime.java
    • -8
    • +3
    ./operators/win/WindowNestedPlansUnboundedRuntimeFactory.java
  1. … 16 more files in changeset.
[NO ISSUE][RT] Window operator runtime optimization

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Runtime optimization for window operator with monotonic

frame start expression. In this case continue scanning

from the beginning of the frame that was found in the

previous iteration

- Allow inlining variables into window operator expressions

except PARTITION BY, ORDER BY and frame value expressions

Change-Id: I65bed4092f4fd3622f1525b26ce25e2ac07d7538

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3135

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -2
    • +1
    ./operators/win/WindowMaterializingPushRuntime.java
    • -57
    • +112
    ./operators/win/WindowNestedPlansPushRuntime.java
    • -7
    • +10
    ./operators/win/WindowNestedPlansRuntimeFactory.java
  1. … 21 more files in changeset.
[NO ISSUE][COMP] Window operator runtime optimization

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add optimized runtime for window operator that computes

nested aggregates over the whole partition (unbounded frame)

- Do not generate unnecessary expressions when compiling

window operator with unbounded frame

Change-Id: If34d8eb05c069257c974f61810bee399136825fa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3124

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -1
    • +1
    ./operators/win/WindowMaterializingPushRuntime.java
    • -0
    • +119
    ./operators/win/WindowNestedPlansUnboundedPushRuntime.java
    • -0
    • +68
    ./operators/win/WindowNestedPlansUnboundedRuntimeFactory.java
  1. … 6 more files in changeset.
[ASTERIXDB-2488][COMP] Support aggregate window functions

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Implement aggregate window functions:

agg_func() OVER (frame_var AS)? (PARTITION BY ... ORDER BY ... frame_spec)

- Where agg_func is a SQL/SQL++ aggregate function

- Fix percent_rank() to always return 0 for the first tuple

- Fix ntile() to handle NULL argument

- Log query after each rewrite rule in SqlppQueryRewriter

- Implement toString() for ADayTimeDuration, fix it for AYearMonthDuration

- Add seek() method to RunFileReader

Change-Id: If0f71118a04c2dbd3462070673d52e67f076b7e1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3049

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -48
    • +10
    ./operators/aggreg/NestedPlansAccumulatingAggregatorFactory.java
    • -44
    • +11
    ./operators/aggreg/NestedPlansRunningAggregatorFactory.java
    • -29
    • +33
    ./operators/aggrun/AbstractRunningAggregatePushRuntime.java
    • -172
    • +0
    ./operators/aggrun/AbstractWindowPushRuntime.java
    • -152
    • +0
    ./operators/aggrun/MaterializingWindowPushRuntime.java
    • -5
    • +4
    ./operators/aggrun/RunningAggregatePushRuntime.java
    • -13
    • +14
    ./operators/aggrun/RunningAggregateRuntimeFactory.java
    • -54
    • +0
    ./operators/aggrun/SimpleWindowPushRuntime.java
    • -65
    • +0
    ./operators/aggrun/WindowRuntimeFactory.java
    • -0
    • +25
    ./operators/meta/PipelineAssembler.java
    • -0
    • +173
    ./operators/win/AbstractWindowPushRuntime.java
    • -0
    • +50
    ./operators/win/AbstractWindowRuntimeFactory.java
    • -0
    • +118
    ./operators/win/WindowAggregatorDescriptorFactory.java
    • -0
    • +170
    ./operators/win/WindowMaterializingPushRuntime.java
    • -0
    • +56
    ./operators/win/WindowMaterializingRuntimeFactory.java
  1. … 840 more files in changeset.
[ASTERIXDB-2466][FUN] Implement window functions

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Implement window functions with SQL syntax:

function() OVER ((PARTITION BY expr1, expr2, ...)? ORDER BY exprA, exprB, ...)

- Where supported functions are:

ROW_NUMBER(), RANK(), DENSE_RANK(), PERCENT_RANK(), NTILE()

Change-Id: Ia28af8773cb11049c38d440c51b9c3cd1ed2bab4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3002

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +30
    ./base/IWindowAggregateEvaluator.java
    • -0
    • +111
    ./operators/aggrun/AbstractRunningAggregatePushRuntime.java
    • -0
    • +172
    ./operators/aggrun/AbstractWindowPushRuntime.java
    • -0
    • +152
    ./operators/aggrun/MaterializingWindowPushRuntime.java
    • -0
    • +42
    ./operators/aggrun/RunningAggregatePushRuntime.java
    • -0
    • +59
    ./operators/aggrun/RunningAggregateRuntimeFactory.java
    • -0
    • +54
    ./operators/aggrun/SimpleWindowPushRuntime.java
    • -0
    • +65
    ./operators/aggrun/WindowRuntimeFactory.java
    • -140
    • +0
    ./operators/std/RunningAggregateRuntimeFactory.java
  1. … 159 more files in changeset.
[NO ISSUE][RT] Follow IFrameWriter protocol in AbstractOneInputPushRuntime

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Many implementations of AbstractOneInputPushRuntime didn't

follow the IFrameWriter protocol causing many unexpected

runtime exceptions.

- This change ensures that all of the subclasses implement the

protocol correctly.

Change-Id: I5133007f298366f58b53acc9f48bc553724dd7b5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2884

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -11
    • +1
    ./operators/aggreg/AggregateRuntimeFactory.java
    • -13
    • +9
    ./operators/base/AbstractOneInputOneOutputOneFramePushRuntime.java
    • -1
    • +18
    ./operators/base/AbstractOneInputPushRuntime.java
    • -10
    • +0
    ./operators/base/AbstractOneInputSourcePushRuntime.java
    • -1
    • +2
    ./operators/meta/SubplanRuntimeFactory.java
    • -6
    • +18
    ./operators/sort/InMemorySortRuntimeFactory.java
    • -17
    • +1
    ./operators/std/AssignRuntimeFactory.java
    • -7
    • +4
    ./operators/std/EmptyTupleSourceRuntimeFactory.java
    • -5
    • +0
    ./operators/std/NestedTupleSourceRuntimeFactory.java
    • -17
    • +1
    ./operators/std/RunningAggregateRuntimeFactory.java
    • -1
    • +1
    ./operators/std/StreamLimitRuntimeFactory.java
    • -1
    • +1
    ./operators/std/StreamProjectRuntimeFactory.java
    • -11
    • +1
    ./operators/std/StreamSelectRuntimeFactory.java
    • -1
    • +4
    ./operators/std/StringStreamingRuntimeFactory.java
    • -1
    • +1
    ./operators/std/UnnestRuntimeFactory.java
  1. … 1 more file in changeset.
[ASTERIXDB-2375][RT] Evaluate constant experession in SELECT only once

- user model changes: none

- storage format changes: none

- interface changes: new method added to ComparisonHelper to handle

cached value.

Currently, in datascan case, the constant value is evaluated on per

tuple bases. We could cache the value to avoid the deserialization of

the constant value each time.

Change-Id: Iae4e78928da2bd63b2984b3624b88baed9b7cd73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2621

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 27 more files in changeset.
[ASTERIXDB-2393][COMP][RT] Add source location to error messages

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add source locations to compiler and runtime error messages

- Add source location enforcement to the test framework

and enable it for SqlppExecutionTest

Change-Id: Ie279ce345d1edcb5dea8e55cdb0233151c7bfd66

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2659

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -2
    • +2
    ./operators/aggreg/NestedPlansAccumulatingAggregatorFactory.java
    • -2
    • +2
    ./operators/aggreg/NestedPlansRunningAggregatorFactory.java
    • -2
    • +1
    ./operators/base/AbstractOneInputOneOutputRuntimeFactory.java
    • -0
    • +35
    ./operators/base/AbstractPushRuntimeFactory.java
    • -2
    • +1
    ./operators/base/SinkRuntimeFactory.java
    • -1
    • +1
    ./operators/std/AssignRuntimeFactory.java
    • -2
    • +2
    ./operators/std/EmptyTupleSourceRuntimeFactory.java
    • -2
    • +2
    ./operators/std/NestedTupleSourceRuntimeFactory.java
    • -2
    • +2
    ./operators/std/PrinterRuntimeFactory.java
    • -2
    • +2
    ./operators/std/SinkWriterRuntimeFactory.java
    • -2
    • +2
    ./operators/union/MicroUnionAllRuntimeFactory.java
  1. … 669 more files in changeset.
[NO ISSUE] Incremental cleanup of deprecated exception ctors

Change-Id: I1e7c3655828fc6530cef83ea502a6cfbf41acddf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2533

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -1
    • +1
    ./aggregators/TupleCountAggregateFunctionFactory.java
    • -1
    • +1
    ./aggregators/TupleCountRunningAggregateFunctionFactory.java
    • -1
    • +1
    ./operators/std/SinkWriterRuntimeFactory.java
    • -1
    • +1
    ./operators/std/SplitOperatorDescriptor.java
    • -1
    • +1
    ./operators/std/StringStreamingRuntimeFactory.java
    • -1
    • +1
    ./operators/std/UnnestRuntimeFactory.java
    • -1
    • +1
    ./writers/SerializedDataWriterFactory.java
  1. … 234 more files in changeset.
[NO ISSUE][RT] follow IFrameWriter protocol in SplitOperatorDescriptor

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Previously, the SplitOperatorDescriptor didn't follow the

IFrameWriter protocol in case of failure which lead to having

some open resources after the job.

- This caused so many failures in Cancellation tests.

- This change also increases the rate of cancellation during the

cancellation tests to ensure that similar problems are found.

Change-Id: I3166895589e1ab7355d689397f676f7da5c9809f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2399

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -12
    • +31
    ./operators/std/SplitOperatorDescriptor.java
  1. … 13 more files in changeset.
[ASTERIXDB-2244][RT] Implement micro union-all operator

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- implement support for binary micro operators in subplans

- implement micro union-all operator

- fix free variables visitor

Change-Id: I11be926f175889978c144dd4483ec565d3d86e2d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2277

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Till Westmann <tillw@apache.org>

Integration-Tests: Till Westmann <tillw@apache.org>

Tested-by: Till Westmann <tillw@apache.org>

    • -1
    • +1
    ./operators/aggreg/NestedPlansAccumulatingAggregatorFactory.java
    • -1
    • +1
    ./operators/aggreg/NestedPlansRunningAggregatorFactory.java
    • -2
    • +2
    ./operators/base/AbstractOneInputOneOutputRuntimeFactory.java
    • -4
    • +3
    ./operators/base/SinkRuntimeFactory.java
    • -2
    • +9
    ./operators/meta/AlgebricksMetaOperatorDescriptor.java
    • -9
    • +30
    ./operators/meta/PipelineAssembler.java
    • -77
    • +155
    ./operators/meta/SubplanRuntimeFactory.java
    • -3
    • +3
    ./operators/std/EmptyTupleSourceRuntimeFactory.java
    • -2
    • +2
    ./operators/std/NestedTupleSourceRuntimeFactory.java
    • -2
    • +2
    ./operators/std/PrinterRuntimeFactory.java
    • -2
    • +2
    ./operators/std/SinkWriterRuntimeFactory.java
    • -0
    • +112
    ./operators/union/MicroUnionAllRuntimeFactory.java
  1. … 28 more files in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -2
    • +2
    ./operators/aggreg/SerializableAggregatorDescriptorFactory.java
    • -2
    • +2
    ./operators/meta/SubplanRuntimeFactory.java
    • -4
    • +3
    ./operators/sort/InMemorySortRuntimeFactory.java
    • -2
    • +2
    ./serializer/ResultSerializerFactoryProvider.java
  1. … 981 more files in changeset.
[NO ISSUE] Misc logging cleanup

Change-Id: I2b951e11a9716eb11832d18eba57b124faf146b1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2295

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -7
    • +1
    ./operators/meta/AlgebricksMetaOperatorDescriptor.java
  1. … 1 more file in changeset.
[ASTERIXDB-2149] Enable multiple normalized keys in sort

- user model changes: no

- storage format changes: no

- interface changes: yes. The interface of sort is changed.

Currently, during the (in-memory) sort, we use an int normalized keys to

speed up comparisions by avoiding random memory accesses. However, this

technique is inefficient if the first 4 bytes of the sorting keys are

not distinctive. From performance point of view, it's better to use

longer normalized keys when it's possible (2-3x improvements).

This is enabled by this patch by:

- Allowing multiple normalized keys during sort, and the length of each

normalized key can be longer (multiple integers).

- Enable memory budgeting of pointer directories as well during sort

(but for performance, we still use int[], instead of byte[] from frame).

The next patch will enable the AsterixDB layer to use this feature to

speed up sort performance.

Change-Id: I4354242ff731b4b006b8446b58f65873047dde78

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2127

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -5
    • +13
    ./operators/sort/InMemorySortRuntimeFactory.java
  1. … 29 more files in changeset.
[ASTERIXDB-2169][STO][TX] Unblock modifications during full scan

- user model changes: no

- storage format changes: no

- interface changes: yes

- added ILSMHarness.replaceMemoryComponentsWithDiskComponents

details:

- During a long running query aka full scan, two things block

incoming modifications:

1) Memory component gets full, is flushed but can't be recycled

because of the search operation inside the component.

2) Read latches on the memory component not being released and

the memory component search cursor is not advancing.

The two cases are addressed in this change for the LSMBTree but

not yet addressed for other indexes.

The proposed solution for case (1) is to poll memory components

states every n records during the search operation. If a memory

component was found to have been flushed, its cursor is moved

to the corresponding disk component allowing the memory

component to be recycled.

The proposed solution for case (2) is to check memory component

cursor every n records. If the cursor has not advanced and the

component has writers, then the latches over the leaf page are

released, and the cursor re-do the operation entering from the

tree root.

- Added a test case.

- Added performance traces for enter and exit components.

Change-Id: I37ba52f6324ed1c5a78465c3a8cbcd351f1ed5bc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2166

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

    • -2
    • +14
    ./operators/base/AbstractOneInputOneOutputOneFramePushRuntime.java
    • -17
    • +32
    ./operators/meta/AlgebricksMetaOperatorDescriptor.java
  1. … 50 more files in changeset.