Clone
 

ali alsuliman <ali.al.solaiman@gmail.com> in asterixdb

[ASTERIXDB-2634][COMP][RT] String functions to return NULL on mismatch

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Make string functions return NULL on mismatch instead of raising

an exception and issue a warning.

- consolitated ExceptionUtil from org.apache.asterix.runtime.exceptions

into org.apache.asterix.om.exceptions.

- AbstractStringTypeComputer does not check arguments type now

and does not throw mismatch exception.

Change-Id: I51db8ae6ba301757c45c41cfd02bd2b083436970

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3562

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 30 more files in changeset.
[NO ISSUE][ING][EXT] Handle filters on meta() fields for feeds

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This fix is to handle filters on meta() fields where the feed name

is not supplied as an arguement to the meta() when connecting the feed.

Both of these statement should work:

CONNECT FEED ds_feed TO DATASET ds WHERE meta(ds_feed).id LIKE "%WEB%"

CONNECT FEED ds_feed TO DATASET ds WHERE meta().id LIKE "%WEB%"

Change-Id: I1dbf40525601584efa29eb3adf4ba41062ebe53a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3557

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +14
    /asterixdb/asterix-app/data/csv/people2.csv
  1. … 20 more files in changeset.
[ASTERIXDB-2631][COMP][RT] Limit number of reported warnings

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Limit number of reported warnings.

- added a new compiler property "COMPILER.RUNTIME.WARNINGS"

with default value set to 0.

- updated Stat class to have a field for the count of runtime

warnings.

- updated few classes to propagate the warnings limit number

to Hyracks Task class.

- added new Option to parse unsigned long values.

- TaskProfile has a field for the count.

- updated TestExecutor to allow specifying parameters in

the body of the POST method.

Change-Id: Ie8756f1b9dcb7bbce92bd77692490fbada9a2482

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3535

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 43 more files in changeset.
[NO ISSUE][COMP] Fix index selection for datasets with meta

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Fix index selection for datasets with meta. Access method rule should

check where the field is coming from (dataset record or meta record)

and then determine if the field matches the keys in the index based

on their names and sources.

This patch also fixes resolving PK field accesses to the primary key

variable (e.g. $ds.getField("id") is turned into $13 where id is a PK).

The fix considers whether the PK is coming from the data record or the

meta record.

The patch also includes fixing rewriting of meta() references and

replacing them with their corresponding meta variables. Now nested plans

are visited also when looking for meta() references to take care of cases

where the data scan producing the meta variable and the meta() references

happen to be inside the nested plans.

MetaFunctionToMetaVariable() is fired also after the rules which eliminate

subplans to allow for rewriting of the meta() if it couldn't be replaced

when the meta() reference was in the subplan but referring to meta variable

outside the nested plan.

ReinferAllTypesRule() is now fired before ByNameToByIndexFieldAccessRule()

to allow the latter rule to get the up-to-date types in the whole plan.

Change-Id: I0503f64cd51153896e2d7d7abc465c679f82e2fd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3545

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Contrib: Till Westmann <tillw@apache.org>

    • -0
    • +9
    /asterixdb/asterix-app/data/csv/people.csv
    • -0
    • +9
    /asterixdb/asterix-app/data/csv/people3.csv
  1. … 78 more files in changeset.
[ASTERIXDB-2626][*DB] Ignore dup fields in record construction, issue warning

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

Change the current behavior of throwing exceptions on duplicates fields

on a record to just picking one and return a warning when possible.

- added source location for auto generated name expressions.

- removed duplicate test cases from AQL test suite since SQL++ alraedy has it.

- updated test cases that used to throw exception on duplicate fields.

- updated an AST test case to reflect the new behaviour of picking one field.

- clean-ups in testsuite_sqlpp:

moved flwor let33 test case to its proper group.

moved ObjectsQueries, GeoQueries xml references up to where other xml refs.

Change-Id: I2d773b194f0e0f35720bb3fd63f1997239800762

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3519

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 39 more files in changeset.
[NO ISSUE][IDX] Add tests to make sure no nulls in index

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Added test cases to test that nulls do not make it

to secondary indexes.

Change-Id: I6d9a9a072f8378f2af262f49728001ce4e10bdbb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3530

Reviewed-by: Michael Blow <mblow@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +3
    /asterixdb/asterix-app/data/data1.adm
[NO ISSUE][COMP] Meta() after group by

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Using meta() after group-by requires meta() be assigned to

a variable in a LET clause to expose meta().

- added source location to IFunctionManager.lookupFunction()

- added test cases

Change-Id: I2b173e23352fa80ef57154f08910586c6ee196b8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3528

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[NO ISSUE][COMP] Fix InlineUnnestFunction to return false if not fired

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Fix InlineUnnestFunctionRule to return false if there are no

changes in the plan.

- minor clean-ups.

Change-Id: Ib2b69ae3ad9712d1443078e0ba0b254b46376d43

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3526

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2611][*DB] Fix GroupAll with empty input

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This patch is to fix the issue with group by all where

some partitions are empty. The patch in general handles

propagating outer scope variables through group by all by using

subplans instead of using decor variables when translating

the AST.

- changed IsomorphismVariableMappingVisitor to stop re-visiting

the same nested tuple operator pair again

- moved getFdColumns() up in hierarchy to be shared by others

- added a check to prevent group-all with decoration fields

Change-Id: I56916fc60c696f6c702717210f108f2247b2a397

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3521

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 23 more files in changeset.
[ASTERIXDB-2615][*DB] Enable constant folding rule to determine when to fold

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Currently, some functions are excluded from being evaluated at compile time in

constant folding rule. Those functions are maintained in a set. The reason for

excluding them is because they can produce records/lists in their "open" format

and constant folding them will make them "closed" which would lead to incorrect

results. Any time a new function is implemented, the developer would have to pay

attention and add their functions if they should be excluded, too.

The exclusion set should be removed. The constant folding rule should determine

from the output type of the function whether it can/should be folded or not.

- fixed an issue with StaticTypeCastUtil where it would not open up a field

when that field's value is a function call.

- modified casting items of lists to avoid casting open the item when it is

already in the opened up format.

- modified the type computer of object_remove_fields to handle constant

arguments (the 2nd argument which is the name of the fields) since the

argument can be constant folded now.

- modified field-access-by-index to handle evaluating it at compile time

(by constant folding rule) in the case where the object accessed is missing

(or non-object)

Change-Id: I3964aa8accaaae3b4c5c7ddf928e6cbd73d517f6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3509

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 35 more files in changeset.
[ASTERIXDB-2613][*DB] Fix serialization of schemaless records

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

When serializing schemaless records, the record serializer should

continue serializing the fields as schemaless fields to handle fields

that are also records (nested records).

Change-Id: Ibc7257987d24905e82d5e81f27840fba81281706

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3501

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2610][RT][*DB] Improve deep comparison runtime

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Avoid data copies during comparison (no visitable pointables).

Allow comparison of non-tagged values to avoid copying the tag

when accessing an array element or record field.

- added TaggedValueReference.

- changed ILogicalBinaryComparator to use TaggedValueReference

instead of IPointable.

- removed numberOfItems() from ListAccessorUtil and used

AOrderedListSerializerDeserializer.getNumberOfItems().

- removed storage & IPointable pools from AbstractAGenericBinaryComparator

& LogicalComplexBinaryComparator and used TaggedValueReference pool.

- removed compareNumbers() from ComparatorUtil since IPointable's

have been replaced with TaggedValueReference.

- adapted record comparison logic in LogicalComplexBinaryComparator

to match the logic in the physical comparator.

Change-Id: Id9ece93c704f566d7bdb7fd17b1ba92713c917d5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3466

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 5 more files in changeset.
[ASTERIXDB-2607][RT][*DB] Fix detecting duplicate fields in record construction

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Fix detecting duplicate field names in record construction.

Change-Id: Ib11647cee4e6dd07f04b9e91ce03cb72f293bcb2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3479

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2575][RT][FUN] Remove usages of ComparisonHelper

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Remove usages of ComparisonHelper and switch to logical or physical

comparators.

- updated RecordReplace test cases that used to throw an exception

when comparing complex types.

- cleaned up interval classes.

- fixed getStartOffset() in AIntervalPointable.

- removed ComparisonHelper

Change-Id: Ib2a05f35c752e1d2ecfdbabbf5d303ea73c7c9db

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3470

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2458][COMP] Fix min/max functions with group by

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

In a sort-group-by operator, an aggregate function has a merging

aggregate function that will be used when merging the run files

if they were generated. The merging aggregate function of the local

min/max aggregate function should be different from the merging

aggregate function of the global min/max. For local min/max, the

merging aggregate function should propagate system_null if the

aggregation is system_null. Also, the global min/max should handle

finishPartial() different from finish(). finishPartial()

should not output NULL if the aggregation is system_null since the

aggregation is still going on.

- added functions to be intermediate steps for local aggregation.

- implemented logic for finishPartial()

- added test cases for strict_min/max with group by

Change-Id: Ie0551b091b9adbbbd51158dbd36124a7184bdce0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3434

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 33 more files in changeset.
[ASTERIXDB-2574][COMP] Fix min/max functions

- user model changes: no

- storage format changes: no

- interface changes: no

This change is mainly for 2 things. The first thing is to not throw

an exception when the type of the aggregated field is invalid for min/max

(e.g. record or rectange) or min/max get incompatible data like string

and int. The result in this case would be NULL. The second thing is to

enable comparing ARRAYs correctly by using logical comparison. When

a partition runs into type invalidity, it will output NULL. The global

aggregator interprets NULL received from a partition as type invalidity

and outputs NULL as the final result. Both SQL and SQL++ will do that.

Special treatment is needed for scalar and distinct version of SQL since

SQL min/max ignores NULL values and continue aggregation and the scalar

and distinct version of SQL are normally setup as a global aggregator

since they behave like the global aggregator in a two-step aggregation.

Currently, there is only a local min and max functions. The other

min/max functions are used for everything, the global function of

two-step aggregation, and for scalar and distinct min/max. In order to

differentiate, a global min/max functions are added that will be used

for the two-step aggregation.

Details:

- fixed listify to open up elements when adding them to the collection

and the collection item type is of type ANY and changed the type inferer

of listify to enable that.

- fixed AbstractCollectionType to make sure itemType is never null.

- changed MinMaxAggTypeComputer to not throw an exception but return

NULL for invalid types.

- changed min/max descriptors to implement inferer to propagate the

type of the field and pass that when getting a comparator.

- switched min/max comparison to the logical comparison.

- refactored method inequalityUndefined to be shared by logical comparison

and min/max functions.

- added global max/min functions to enable differentiating between

scalar min/max, distinct min/max and two-step min/max (global & local).

- code clean-up for LogicalScalarBinaryComparator; created two INSTANCES

and re-used.

Change-Id: I1231cfe558099d167bae0b2fa7fc4879b756baf0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3427

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 137 more files in changeset.
[ASTERIXDB-2547][COMP] Disallow passing UNION tag to get comparator

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

UNION should not be allowed when getting a comparator by tag.

The comparator provider returns a generic comparator with types

ANY when UNION is passed as a tag. This could cause problems if

the actual type is a complex type. UNION should not be allowed.

Change-Id: Id8816a0dc5584f0a27410c512f3a44ccfc6c3151

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3404

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2458][COMP] Fix InjectTypeCastForFunctionArgumentsRule

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

InjectTypeCastForFunctionArgumentsRule is for functions that

can potentially return any of their arguments. switch and

if_null(expr1, expr2, ...) are examples. All the arguments

need to be casted (opened) to the type that the function

will return which is the generalized type of all arguments.

Some functions like if_null can determine the exact expression

they will return, e.g. if_null(1, {"id": 3}) in which case

the return type is always integer. The rule tries to cast

th 2nd argument, the record, to integer and fails. In such

cases, these functions do not need to cast their arguments.

If the function determines its output type to be ANY, then

all arguments need to be casted (opened). If the function

determines its output to be a dervied type, then casting is

also needed since that output type should be the generalized

type of all arguments.

Change-Id: I2fee234d883b59319e4ec4df58d61ecd498373fd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3406

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2288][COMP] Fix field access with UNION

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

For UNION, if there is an assign operator above UNION, the optimizer

will move the assign operator under UNION into both branches. If

the assign operator has field access by index, the index is based on

the output record of the union operator which could be different from

the field index in left and right branches. The fix is to map the

index to the correct index of the input operator where the assign

is copied to.

Change-Id: I4d65374aca6959fd5e70279fac8f7f0a6c7c64c5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3400

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2289][COMP] Fix field access with CASE

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This patch fixes field access in the presense CASE and JOIN.

This is a scenario where push-down-field-access rule throws an

exception if the field access has potentially two sources and

it could not push down the field access to left or right branch.

Don't throw an exception and just return false

(i.e. field access was not pushed) instead of throwing an exception.

Change-Id: I911e4e9018c15e8f226e46fa610d222eb2301fcd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3399

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2567][RT] Fix serializing nullable record fields

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

When serializing a record, a field should be checked if it's nullable.

The value to be serialized should be checked if it's null or value to

serialize it correctly. Exceptions could happen when the record is in

an array if record fields are not checked.

Change-Id: I42105965e69c60bc4b123337296476b4d75f0ad8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3398

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 3 more files in changeset.
[ASTERIXDB-2564][RT] Too many objects created in min() and max()

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

During min() and max() aggregation, the functions keep track of

the aggregation type in order to handle heterogeneous lists.

It promotes the aggregation type if needed (e.g. encountered double).

Don't switch to new aggregation type and create a new comparator

when the new input value type is the same as the previously

aggregated values. That is because canPromote(agg_type, new_val_type)

will always return true for same types.

Change-Id: I0bb9f0715985ae555de00bbf3173c80371d8968b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3391

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2555][RT][COMP] Make hash join use logical comparison

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This patch changes the hash join operator to use the join condition

to evaluate if tuples are equal when joining. Binary physical comparators

have been removed. The join condition evaluator is in TuplePairEvaluator.

- extraced TuplePairEvaluatorFactory out of nested loop join class

into a separate class so that it is shared among nested loop and

hash join.

- switched from FrameTuplePairComparator to ITuplePairComparator in

in OptimizedHybridHashJoin and InMemoryHashJoin.

- moved debugging code from OptimizedHybridHashJoin into a separate

class, JoinUtil.

- temporarily made the logical comparison of multisets use raw binary

comparison instead of returning null until the logic is implemented.

- made IBinaryBooleanInspector a functional interface and updated

the implementations.

- updated record and array test cases to reflect the new

behaviour of hash join where logical comparison could produce null.

Also, updated sorting, group by and distinct test cases since

the input data has been modified.

- added two new input files arrays1nulls.adm & arrays2nulls.adm

to be used by the open dataset. previous arrays1.adm & arrays2.adm

are used by the closed dataset since it cannot accept arrays with

null values.

Change-Id: If1834967fdd913fdc76003f09636b2450d07cd5e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3387

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -0
    • +23
    /asterixdb/asterix-app/data/complex/arrays1nulls.adm
    • -0
    • +18
    /asterixdb/asterix-app/data/complex/arrays2nulls.adm
  1. … 34 more files in changeset.
[NO ISSUE][HYR][RT] Handle calls to close() properly in hash join

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Hash join build activity assumes that upon calling close()

the join state must have been allocated in open() and

therefore, it will directly call "close build" using the join

state. However, if the build activity throws an exception

in open(), then it could happen that the join state has

not been initialized. Calling close() on the build activity

will lead to a NPE since it will use the join state directly.

This patch checks if the join state has been initialized. If

so, it will close the build as usual.

Change-Id: I41fff7deed5e56818bfefb4d36b8018a9f512cbb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3375

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[ASTERIXDB-2552][RT] Reuse objects on reseting buffer manager

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

The "VariableFrameMemoryManager" releases the physical frames

objects, the logical frames objects and the free slots objects

on reset(). This patch is to retain those objects and only

release them on close() (as also stated in the interface).

- added close() to IFrameFreeSlotPolicy to make the interface similar

to the other interfaces (IFrameBufferManager, IFramePool, ... etc)

where close() is supposed to release while reset() is supposed to only

reset but retain resources.

- renamed "InMemorySortRuntimeFactory" to "MicroSortRuntimeFactory".

Change-Id: I88bea27e5024c621412ef609475e2a7ba1913afa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3372

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Till Westmann <tillw@apache.org>

[NO ISSUE][FUN] Move negative tests to seperate test unit

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Moved array functions negative tests to a seperate test unit.

Change-Id: I0a3bfce402e4b7091c40483b5802db7b27afa717

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3371

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 22 more files in changeset.
[ASTERIXDB-2552][RT] Implement micro external sort for subplans

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This patch is to change the in memory sort used in subplans to

be a micro external sort to avoid out of memory error.

- added reset() to the runs merger to allow reusing the runs merger.

- renamed "InMemoryStableSortPOperator" to "MicroStableSortPOperator".

- changed the tag from "IN_MEMORY_STABLE_SORT" to "MICRO_STABLE_SORT".

- added test cases.

Change-Id: I930849d644c60d461d2869c9773b85e49b46fbdb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3353

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 19 more files in changeset.
[ASTERIXDB-2552][RT] Refactor runs generator and merger

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Moved the writer and sorter out of the merger to allow

micro external sort to use the run generator and merger

plus minor clean-ups.

Change-Id: Idda31c92cbcddba5ebef8bbbf7855b9c8293dd51

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3363

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2554][HYR] Add UTF8 and byte array comparator factories

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Add comparator factories for UTF8StringPointable, UTF8StringLowercasePointable,

UTF8StringLowercaseTokenPointable and ByteArrayPointable instead of using

PointableBinaryComparatorFactory, a wrapping factory that will create a factory

each time (which also creates a comparator each time).

Change-Id: Ied6a29210a3dc1ba9fd553fb0a67ff4340e4571f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3355

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 34 more files in changeset.
[ASTERIXDB-2516][RT] Prevent passing null type to comparator provider

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This change is to disallow passing null as IAType to the comparator

provider when asking for a comparator. It also includes few fixes

for issues appearing as a result of this change.

- changed the generic comparator to check whether the tag of data

at runtime is a valid tag and throw an exception if not.

- modified the comparator provider to return non-tagged comparator

for IAType SHORTWITHOUTINFOTYPE which is a short without tag.

SHORTWITHOUTINFOTYPE should not use the generic comparator since

the input data has no tag.

- fixed Dataset class to consider external dataset when getting

the IAType of the primary keys. The primary keys for external

datasets are different from regular datasets. They are not

part of the record type. Previously, null would be returned.

This would cause a failure when getting a comparator for the

primary keys of an external dataset since the type passed is null.

- fixed the TypeTraitProvider to give the correct lengths for

duration, day_time_duration, and year_month_duration.

Change-Id: I37767a3f3d1e3b29597d2a4998c0b60005cadb09

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3336

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>