Clone Tools
  • last updated 23 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][COMP] Temporarily disable some property matching

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- ORDERED_PARTITIONED can only satisfy UNORDERED_PARTITIONED

if both use the same partitioning function. Currently there is no

mechanism for reasoning about partitioning functions so this case

has to be disabled until it is in place.

Change-Id: I7f280505f4fa02283fc14a0398760162e1150dc9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3532

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 7 more files in changeset.
[NO ISSUE][COMP] Static range map in full parallel sort

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Support static range map in full parallel sort

- Add static range map option to ORDERED_PARTITIONED

structural property

- Add parseExpression() methods to SQL++ and AQL parsers

- Simplify RangeMapBuilder.parseHint()

Change-Id: I7eab6e6ede8c2dbb714a27801a76ad64bd9be1b8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3513

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 22 more files in changeset.
[NO ISSUE][COMP] Improve IPartitioningProperty.substituteColumnVars()

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- IPartitioningProperty.substituteColumnVars() should return

a new instance of IPartitioningProperty if variable

substitution was performed for given variables

- Compute schema for a new Select operator introduced by

PullSelectOutOfEqJoin rule

- Refactor delivered properties computation in

AbstractPreclusteredGroupByPOperator

Change-Id: Iee7ef7de26f9d960b205d6d43e2820aaa396cb15

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3499

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -6
    • +10
    ./UnorderedPartitionedProperty.java
  1. … 3 more files in changeset.
[NO ISSUE][COMP] Make memory requirements an operator property

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Refactor how memory requirements are maintained in a query plan

- Introduce LocalMemoryRequirements class. Its instances are held

by each physical operator and could be altered by the optimizer.

- Introduce optimizer rule SetMemoryRequirementsRule that

initializes and configures memory requirements for each operator

Change-Id: I3481ddfe163c6ce786290c540cbd05db16a7f64f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3374

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +102
    ./LocalMemoryRequirements.java
  1. … 31 more files in changeset.
[ASTERIXDB-2286][COMP][FUN][HYR] Parallel Sort Optimization

- user model changes: yes

- storage format changes: no

- interface changes: yes

details:

- new plan for sort operation which includes sampling and

replicating the stream of data to be sorted. Sort-merge connector

is removed from the plan. The sorted result now is in multiple partitions.

- new optimization rule to check whether full parallel sort is applicable.

- new Forward operator to read the replicated sort input stream and

to receive the ouput of the sampling.

- new sequential merge connector to merge a globally ordered result residing

in multiple partitions (in addition to the connector's partition computer).

- "asterix-lang-aql/pom.xml" is changed as a result of refactoring

code related to the range map handling.

- new private sampling function to generate the range map object

(local & global functions) & their type computers.

user model changes:

- new compiler property is added to enable and disable parallel sort.

interface changes:

- "ILogicalOperatorVisitor.java" includes Forward Operator.

- "ITuplePartitionComputer.java" includes initialize() to enable partitioner

to do some initialization. FieldRangePartitionComputerFactory uses it to

pick a range map.

- "ITuplePartitionComputerFactory.java". createPartitioner() is changed to

createPartitioner(IHyracksTaskContext hyracksTaskContext). Context is needed

for transferring the range map throught the context.

Change-Id: I73e128029a46f45e6b68c23dfb9310d5de10582f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2393

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 356 more files in changeset.
[ASTERIXDB-2174] Use ListSet when clone partition property

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Use ListSet instead of HashSet when cloning partitioning properties

to ensure it has the consistent behavior as before.

Change-Id: Ib2b8c1715d75b387a73504f4709524d9ab9ce123

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2968

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2174] Fix partitioning improvement of group by

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fix the improvement of partitioning propagation of group by

introduced by https://asterix-gerrit.ics.uci.edu/#/c/2176/.

- The problem is that we cannot modify the partitioning property of the

input operator directly, since that property is owned by the input.

Otherwise, the property of the child operator would be modified even

before the group by operator, and this caused some tests regarding

common sub operator extration to fail (introducing REPLICATE).

Change-Id: I81a74ab1242772c790175414281e3f200537b386

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2966

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 4 more files in changeset.
[NO ISSUE][ING] Allow external UDF to use runtime parallelism

- user model changes: no

- storage format changes: no

- interface changes:

Details:

1. Enable UDF in feed to use the runtime parallelism.

2. Fix the DefaultNodeDomain where the nodes should be mutliSet but not

list, for comparison purpose.

Change-Id: Ic3b54617be115f51b6a48b9a61581c26b5be8d9d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2398

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 7 more files in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -41
    • +43
    ./IPartitioningRequirementsCoordinator.java
  1. … 981 more files in changeset.
[NO ISSUE][HYR][*DB] Minor refactoring / address SonarQube comments

Change-Id: Icf10b6df0fdc006675d8f0da6fd06d50200c6b6a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2098

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 60 more files in changeset.
Supports flexible degree of parallelism.

Detailed list of changes include:

- Add the IClusterInfoCollector interface and feed that to the compiler;

- Add the number of cores into the heartbeat;

- Add partition calculation in APIFramework;

- Fix partitioning property requirements in binary operators such as Join and UnionAll;

- Fix PushSubplanIntoGroupByRule for type propagation;

- Fix InlineLeftNtsInSubplanJoinFlatteningVisitor for the returned top join operator;

- Add unit test for partition calculation in APIFrameworkTest

- Add integration test SqlppExecutionFullParallelismIT, SqlppExecutionLessParallelismIT,

AqlExecutionFullParallelismIT, and AqlExecutionLessParallelismIT for different

parallelism settings;

- Refactor AQLExecutionTest and SqlppExecutionTest;

- Separate one cluster state test out from AqlExecutionTest.

Change-Id: If70faf52bed995c7098cc343f2fabd1b6c8d96e7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1398

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 115 more files in changeset.
Cleanup FileSplit and FileReference

This change gives FileSplit and FileReference specific meaning to

avoid confusion of an absolute vs relative, local vs global, inside

an IO device vs outside IO devices.

In addition, it enables better abstraction of global partitions and

delegate the responsibility of choosing which partition goes to which

IO device to the IO Manager through the introduction of FileDeviceComputer

In details:

Previously, the LocalResource in Hyracks had partition (storage partition)

and there is no such thing in Hyracks. This scope leak is bad. In addition

The local resource had a name and a path. they were always the same and so

the name was removed.

The storage partition was instead moved to asterixdb implementation of the

serialized object in the local resource.

With all of these changes, the cluster controller (compiler) only needs to

know about partitions and relative paths. It doesn't need to worry about

heterogenous Node setups and different io device configurations. For File

assignment to IO devices, a new interface (IFileDeviceComputer) was

introduced which can be overriden by applications to have their own

strategy for distributing files among IO devices.

Change-Id: I4fac508bf9af5a3bed41a3cf4464d2cbfecf2f61

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1352

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 287 more files in changeset.
merge master

  1. … 21 more files in changeset.
ASTERIXDB-1592: fix and regression tests.

Change-Id: Id59ae2fd79302211973cfb3b82829eb3836da4bc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1110

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 9 more files in changeset.
ASTERIXDB-1571: fix and add regression tests.

- also fix the left outer type propagation policy.

Change-Id: I45d6dbabdef709c9ba3cb883412301ab7fd69a85

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1075

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 19 more files in changeset.
Finalize the range in context.

  1. … 30 more files in changeset.
Clean up GROUP BY and WITH clause.

- fix ASTERIXDB-971, i.e., the expression to plan translator for FLWOGR without a "for";

- fix the scoping for nested WITH clause;

- fix the scoping for nested GROUP BY clause;

- fix default fields in group variables;

- fix/unify the expression substition AST visitor;

- enhance RemoveUnusedAssignAndAggregateRule to be able to remove unused GROUP BY

decoration varaibles;

- clearly sperate SQL++ aggregate functions and SQL-92 aggregate functions, i.e.,

SQL-92 aggregate sugars cannot apply to SQL++ aggregate functions;

- fix PushAggregateIntoGroupbyRule for nested plan deletions;

- enhance RemoveCartesianProductWithEmptyBranchRule;

- add ExtractGroupByDecorVariablesRule;

- add an aggregate function first-element;

- fix ASTERIXDB-1560, an error message issue.

Change-Id: I62fca7f937aa007d97ed87c75cef19f6aa3e5ade

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1050

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 235 more files in changeset.
snapshot for range state transition

    • -14
    • +12
    ./OrderedPartitionedProperty.java
  1. … 69 more files in changeset.
Support UNION ALL.

- support heterogenous input types for UNION ALL and the output is resolved to a mimimally generalized type;

- fix the partitioning property inference for UnionAllPOperator;

- fix for ASTERIXDB-1354, a bug in IsomorphismVariableMappingVisitor.

Change-Id: Ifadf1707bb2b6bed22f8fc5792c635e87291a468

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1035

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -1
    • +1
    ./IPartitioningRequirementsCoordinator.java
  1. … 72 more files in changeset.
ASTERIXDB-1531: fix ORDER BY with primary keys as non-prefix sort keys.

- clean up the local property inference implemenation;

- avoid side-effects for data properties during property matching.

Change-Id: Iee7fcdd6eb1279e8ee14ba75ee31ac118b00c806

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1022

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 13 more files in changeset.
Sonar fixes and other clean up items.

Change-Id: Ieb72a0fef3c181282cc4c71b881a7cb52da86818

Reviewed-on: https://asterix-gerrit.ics.uci.edu/999

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 67 more files in changeset.
merge with master

  1. … 12 more files in changeset.
Support implicit variable name and column name.

1. Support WITH clause for general cases;

2. Fixed open/closed type optimization;

3. Fixed scoping for nested aggregations;

4. Fixed side effects during partition property analysis;

5. Fixed type propagation policy for Subplan operator.

Change-Id: I409b9ba139c9f000a6b9b84d519d172d0069e4bb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/950

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 225 more files in changeset.
sonar update

  1. … 45 more files in changeset.
ASTERIXDB-1228: Add MISSING into the data model.

1. MISSING repsents the value of a non-existing field in a record

or an out-of-bound index access of a collection;

2. NULL represents that the value of an optional field in a record

is unknown or the value of existing collection entry is unknown.

3. Unit tests for all missing/null-in-missing/null-out scalar functions.

Change-Id: Ia49ed8474bfc5d6604231819065117468c5b0897

Reviewed-on: https://asterix-gerrit.ics.uci.edu/846

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 551 more files in changeset.
great merge - hyracks

  1. … 63 more files in changeset.
Move Hyracks to subfolder

    • -0
    • +90
    ./AbstractGroupingProperty.java
    • -0
    • +64
    ./BroadcastPartitioningProperty.java
    • -0
    • +70
    ./DefaultNodeGroupDomain.java
    • -0
    • +76
    ./FileSplitDomain.java
    • -0
    • +41
    ./FilteredVariablePropagationPolicy.java
    • -0
    • +57
    ./FunctionalDependency.java
    • -0
    • +54
    ./ILocalStructuralProperty.java
    • -0
    • +25
    ./ILogicalPropertiesVector.java
    • -0
    • +91
    ./IPartitioningProperty.java
    • -0
    • +102
    ./IPartitioningRequirementsCoordinator.java
    • -0
    • +47
    ./IPhysicalPropertiesVector.java
    • -0
    • +27
    ./IPropertiesComputer.java
    • -0
    • +27
    ./IStructuralProperty.java
    • -0
    • +107
    ./LocalGroupingProperty.java
  1. … 4413 more files in changeset.