Clone Tools
  • last updated 29 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Separate index build from index access

This change separates index build from index access.

All indexes now have a single dataflow helper which

uses the index path to locate the resource on the nc

to read the resource from memory or disk.

Existing resource metadata and dataflow helpers were

combined into resource builders eliminating lots of

duplicated code.

Change-Id: Ie4ea3aaa63dff8d246fa43ca7c7359729bc8cf47

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1728

Integration-Tests: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 571 more files in changeset.
Introduce IStorageComponentProvider

Change-Id: If86750cdb2436c713f6598e54d4aaaf23d9f7bbf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1451

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -400
    • +0
    ./SecondaryBTreeOperationsHelper.java
    • -585
    • +0
    ./SecondaryIndexOperationsHelper.java
    • -383
    • +0
    ./SecondaryInvertedIndexOperationsHelper.java
    • -513
    • +0
    ./SecondaryRTreeOperationsHelper.java
    • -0
    • +78
    ./StorageComponentProvider.java
  1. … 419 more files in changeset.
Remove tinyaccessor dependency from production.

Change-Id: Ib2dfb36b59ba4a77884c30c2b11c2a846c4890fa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1459

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 22 more files in changeset.
ASTERIXDB-1711: remove "Aql"

- AQLAPIServlet -> FullAPIServlet

- AqlPrimitiveValueProviderFactory -> PrimitiveValueProviderFactory

- remove AqlLogicalPlanAndMetadataImpl

- remore AqlMetadataImplConfig

Change-Id: I8e371941ba509d10193649aaed48a74a08d2d1f5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1447

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 9 more files in changeset.
ASTERIXDB-1711: remove more mentions of Asterix

Change-Id: I21666fd54075ac56de8aab76bb7d5cdbec020121

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1387

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

    • -23
    • +23
    ./SecondaryBTreeOperationsHelper.java
    • -18
    • +19
    ./SecondaryIndexOperationsHelper.java
    • -17
    • +17
    ./SecondaryInvertedIndexOperationsHelper.java
    • -23
    • +23
    ./SecondaryRTreeOperationsHelper.java
  1. … 564 more files in changeset.
Remove Append Only Flag

Change-Id: Id5d6917db8ab29aa01521596f556006e25a502fe

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1385

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -3
    • +7
    ./SecondaryInvertedIndexOperationsHelper.java
  1. … 208 more files in changeset.
ASTERIXDB-1711: remove some more Aql-prefixes

Change-Id: I041341fe76e9338b7c568a9eacaf826f8fa0ed46

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1360

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -14
    • +14
    ./SecondaryIndexOperationsHelper.java
  1. … 231 more files in changeset.
Continue Cleaning Up File References and Splits

1. Make FileSplit an abstract class with two subclasses;

Managed and Unmanaged. A Managed FileSplit can be mapped

in a new subclass MappedFileSplit that maps a relative path to an

IO device. UnmanagedFileSplit is for files outside the io devices.

2. Remove all usages of absolute paths in file split in test cases. The

only remaining place is the write statement.

3. Fix some of the hidden issues in the tests that were working because

of our use of the absolute paths.

4. Revert the decision of selecting the IO device to the CC.

Change-Id: I166af8f9b3a2257f94d7b05db94888fb7cb4c79e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1359

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 253 more files in changeset.
Cleanup FileSplit and FileReference

This change gives FileSplit and FileReference specific meaning to

avoid confusion of an absolute vs relative, local vs global, inside

an IO device vs outside IO devices.

In addition, it enables better abstraction of global partitions and

delegate the responsibility of choosing which partition goes to which

IO device to the IO Manager through the introduction of FileDeviceComputer

In details:

Previously, the LocalResource in Hyracks had partition (storage partition)

and there is no such thing in Hyracks. This scope leak is bad. In addition

The local resource had a name and a path. they were always the same and so

the name was removed.

The storage partition was instead moved to asterixdb implementation of the

serialized object in the local resource.

With all of these changes, the cluster controller (compiler) only needs to

know about partitions and relative paths. It doesn't need to worry about

heterogenous Node setups and different io device configurations. For File

assignment to IO devices, a new interface (IFileDeviceComputer) was

introduced which can be overriden by applications to have their own

strategy for distributing files among IO devices.

Change-Id: I4fac508bf9af5a3bed41a3cf4464d2cbfecf2f61

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1352

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -3
    • +3
    ./SecondaryInvertedIndexOperationsHelper.java
  1. … 284 more files in changeset.
ASTERIXDB-1711: remove some Aql-prefixes

Change-Id: Ia0b64ffa7c50cd62fc3303fdb44eb769f56c978a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1314

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -3
    • +3
    ./SecondaryInvertedIndexOperationsHelper.java
    • -13
    • +13
    ./SecondaryRTreeOperationsHelper.java
  1. … 78 more files in changeset.
Refactor Messaging

In this refactoring, each message implementation includes

a handle method. This avoids bloating of message brokers

and enable better extensibility for messaging.

Change-Id: I7c918bf504058c98ecf89f5b019503278e9aa01f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1128

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -1
    • +1
    ./SecondaryInvertedIndexOperationsHelper.java
  1. … 120 more files in changeset.
Support CASE expression in SQL++.

- support both simple CASE expression and searched CASE expression;

- support heterogenous return types of a CASE expression;

- remove IfElseExpression in SQL++;

- remove IfElseToSwitchCaseFunctionRule.java since the plan pattern it tries to identify is no longer right;

- unify CastRecordDescriptor and CastListDescriptor into CastTypeDescriptor;

- fix IS NULL for the case that the input is MISSING;

- let CastTypeDescriptor have a generated runtime so as to handle NULL/MISSING correctly.

Change-Id: Id9de7c3df70be43456e38a61ce767cc14f09e661

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1015

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -27
    • +27
    ./SecondaryBTreeOperationsHelper.java
    • -20
    • +19
    ./SecondaryIndexOperationsHelper.java
    • -23
    • +23
    ./SecondaryInvertedIndexOperationsHelper.java
    • -50
    • +52
    ./SecondaryRTreeOperationsHelper.java
  1. … 112 more files in changeset.
Fix Indexing on Open fields and Meta fields

This change fix the following cases:

1. Build a secondary index on a meta field: success

2. Build an open index on a meta field: failure

3. Build a secondary index on an open field in record part: success

Testing ingestion and querying are working correctly for these cases.

Change-Id: I6195149940f150250a65f2515e9ac9d6de2a33f9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/930

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -7
    • +26
    ./SecondaryBTreeOperationsHelper.java
    • -13
    • +36
    ./SecondaryIndexOperationsHelper.java
    • -14
    • +16
    ./SecondaryInvertedIndexOperationsHelper.java
  1. … 73 more files in changeset.
ASTERIXDB-1228: Add MISSING into the data model.

1. MISSING repsents the value of a non-existing field in a record

or an out-of-bound index access of a collection;

2. NULL represents that the value of an optional field in a record

is unknown or the value of existing collection entry is unknown.

3. Unit tests for all missing/null-in-missing/null-out scalar functions.

Change-Id: Ia49ed8474bfc5d6604231819065117468c5b0897

Reviewed-on: https://asterix-gerrit.ics.uci.edu/846

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 551 more files in changeset.
ASTERIXDB-1434: fixed RecoveryIT failure test

- Fixed Recovery IT failure test. There has been a lurking bug

which always set the isPrimaryIndex flag to true incorrectly for

secondary LSMBTree indexes. The bug surfaced out from the

recent change which removed a Bloom filter from secondary

LSMBTree indexes.

Change-Id: I85bc3c8445c10da8f2a8c2f334d1eeefb1c0662a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/845

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <michael.blow@couchbase.com>

Set LSMRTreeWithAntiMatterTuples as the default RTree for interal dataset

Change-Id: I19323ef38335409bc914265502ae56655db6d106

Reviewed-on: https://asterix-gerrit.ics.uci.edu/838

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

    • -27
    • +40
    ./SecondaryRTreeOperationsHelper.java
  1. … 5 more files in changeset.
MBR size optimization for point data in R-Tree

Change-Id: Ic48304a221f462c66bb1417f291b0ce4f2ce7581

Reviewed-on: https://asterix-gerrit.ics.uci.edu/831

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

    • -25
    • +38
    ./SecondaryBTreeOperationsHelper.java
    • -18
    • +11
    ./SecondaryIndexOperationsHelper.java
    • -2
    • +2
    ./SecondaryInvertedIndexOperationsHelper.java
    • -57
    • +141
    ./SecondaryRTreeOperationsHelper.java
  1. … 42 more files in changeset.
Merge branch 'master' into hyracks-merge2

  1. … 483 more files in changeset.