src

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- In many cases, we would like to associate a value with a key

in MetadataProvider to be accessed during the compilation

of jobs. However, currently, there is no place to store

such values, so we ended up storing them in the config

map.

- The config map is a <String, String> map and so, we would

write our values as a string and then parse them when needed.

- To avoid this, and to avoid introducing a new map, we simply

change the config stored in MetadataProvider from <String,String>

to <String, Object>.

Change-Id: I55b392ad199d74b0f3cffdc38b54593b12ec1a06

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2795

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Murtadha Hubail <mhubail@apache.org>

- user model changes: no

- storage format changes: no

- interface changes: yes

+ IClientRequest: used to represent a client request

that can be cancelled.

Details:

- Introduce IClientRequest to allow for multiple types of requests

to be cancellable.

Change-Id: I8f65da1744ea7ecf26ea3f8a576ebaf4472ccd62

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2774

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Include NC local counters in the NC startup tasks

completion completion message.

- Ensure no NC is marked as active without its local

counters reported.

- Remove the need for individual NC local counters

message.

- Clean up ITxnIdFactory and IResourceIdManager APIs.

Change-Id: Ief8b9d43783ea22810c6fdb29947a1284e32daee

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2767

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

- user model changes: no

- storage format changes:

This change replaces the use of Java serialization for persisted objects

such as dataset/index metadata, checkpoints, ect...

This will break backward compatibly with any existing AsterixDB instance.

However, the change is needed to enable future backward compatibility support

for persisted objects.

- interface changes:

IJsonSerializable: contains API to serialize a class as a JsonNode.

IPersistedResourceRegistry: contains a mapping between an IJsonSerializable

class and a unique type id. An IPersistedResourceRegistry is responsible

for generating the class identifier in the JSON output.

The class identifier will always contain the following attributes:

@type: a unique type id that identifies the object type.

@version: the version of the serialized class.

@class: the serialized class full name.

Any registered class with PersistedResourceRegistry must provide

a static fromJson(IPersistedResourceRegistry, JsonNode) method for

deserialization. This is ensured during the class registration process.

Change-Id: I5b103e06eab6627dbfe9d531caae1a3ac4b296da

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2752

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Murtadha Hubail <mhubail@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduce utils to convert pointables into adm nodes.

- Such functions can be useful for tuple translation of

Metadata entities, reading constant records in with

clauses in DDLs, and for debugging runtime.

Change-Id: I705e75d573a7f2c888dd7d83b4c24a2c7050eb88

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2733

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add source locations to compiler and runtime error messages

- Add source location enforcement to the test framework

and enable it for SqlppExecutionTest

Change-Id: Ie279ce345d1edcb5dea8e55cdb0233151c7bfd66

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2659

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

-Implement LIMIT pushdown for primary index scan/lookup. In case of

a select operator, we also push the select condition to the priamry

search operator to ensure correctness.

Change-Id: I824fcad79995325e12a1a81d629160025294b915

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2541

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

- user model changes: no

- storage format changes: yes

- renamed AbstractLSMIOOperationCallbackFactory

to LSMIOOperationCallbackFactory

- useless classes have been removed.

- LSMBTreeIOOperationCallbackFactory

- LSMBTreeWithBuddyIOOperationCallbackFactory

- LSMInvertedIndexIOOperationCallbackFactory

- LSMRTreeIOOperationCallbackFactory

- interface changes: yes

Details:

- Previously, flushes have different lifecycle depending

on the memory component state

- not allocated

- allocated

- modified

- In certain cases, flush operations are skipped alltogether

- IO Operation callbacks became complicated and difficult

to maintain since calls are done differently in different

cases.

- In certain cases, afterFinalize is called on the IO

Operation callbacks even if beforeOperation was never

called.

- In this change, flushes go through the same lifecycle

events regardless of the state of the memory component.

- In addition, primary and secondary memory components

would reside in different virtual buffer caches due

to skipped flushes, or due to having the secondary

index created when the primary index's memory component

is residing on the virtual buffer cache with index !=0.

- Moreover, when flushes are lagging and all memory

components are being flushed, search operations assumes

the oldest of the memory component is the newest and

produces incorrect results.

- In addition, in case of a failed flush of a component,

the IO scheduler would skip it and flush the next

component. This would produce a bad state on disk.

- In this change, a failed flush can be retried. otherwise,

all future flushes of the component fail due to the failure

of the previously failed flush.

- Previously, when a component fails to modify an index due

to flush failures, it assumes disk is full.

- With this change, the modification failure reports the

original cause of the failed flush.

Change-Id: I29f7992ec6c0f71c5b63d45800b2fb590d651e4b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2584

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

Change-Id: Iff6678efff502b566020896320ecdf170ade720a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2576

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Remove legacy metadata latch that was used before

the metadata entity locking was introduced.

Change-Id: I5b6b7153c8bc7bb6b489242c5d651879a09b57e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2556

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fix the creation of indexes on optional nested field

by handling AUnionType properly.

- Fix the nullability check of primary keys.

Change-Id: If098ff4a45db3c4e8b65d098cbb0940c8b7ab845

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2477

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Change-Id: I1e7c3655828fc6530cef83ea502a6cfbf41acddf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2533

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

- A new geometry data type that represents standard OGC objects

- A new set of spatial functions that operate on geometry objects including

spatial analysis, spatial predicates, and spatial aggregate functions

- Parse both Well-Known Text (WKT) and GeoJSON standard formats

- All geometry functions are implemented through the open source library

Esri Geometry API

Change-Id: I9cddeffea42e85469b6fc38f361bd98e64025289

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2056

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Remove invalid findDataset from MetadataManager.

- Add test case for dropping and recreating a dataverse

with indexes and data.

Change-Id: I59b084c85279d926aa60fb7bfcd1a9f6e5935f2a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2482

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- All metadata locking is performed by the CC in

MetadataLockManager. Having the locking API in

MetadataNode is only causing confusion.

Change-Id: I9fae8bd32c8312ef69b38016ad00f170fca9a1c5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2474

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Make all metadata indexes modifications as

force modifications.

- Do not decrement ops of atomic transactions

until they fully commit or abort to prevent

flushing partial records.

- Do not schedule flush if a force modification

starts before the flush log is written to disk.

- Unify code path for completing operations

after commit/abort in op tracker.

- Remove unneeded update log commit notification.

- Add test case for failing flush due to force

modification.

Change-Id: If8d5df630f1d9119002ef91da5c282da18901acc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2456

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fix leak of one id at start of each allocated block

- Increase block size from 100 -> 512

Change-Id: Ia97d8472f30ef0f68c51830b4cbb4660184d7393

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2465

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

- user model changes: no

- storage format changes: no

- interface changes: yes

Added parameters to LibraryFunction in library.xsd.

Details:

By enabling init function in external UDF, a user can reuse the same UDF

implementation with different paramters. One example can be to reuse the

same Machine Learning algorithm with different model files by assigning

different file paths to the UDF parameter.

Change-Id: I567ce0bcac288267595b2565e53fea61e16fbd65

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2107

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

- user model changes: no

- storage format changes: no

- interface changes:

Details:

1. Enable UDF in feed to use the runtime parallelism.

2. Fix the DefaultNodeDomain where the nodes should be mutliSet but not

list, for comparison purpose.

Change-Id: Ic3b54617be115f51b6a48b9a61581c26b5be8d9d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2398

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

- user model changes: yes

Add syntax support for specifying predicate in connect feed

- storage format changes: no

- interface changes: no

Details:

In some use cases, a user may want to filter the incombing data with

certain attributes. One example can be only store the incoming tweets

with geo locations. This patch enables the <WHERE> clause in connect

feed statement. User can subset the incoming data using following

syntax:

connect feed feeds.TweetFeed to dataset Tweets3 using policy `Basic`

WHERE id NOT LIKE 'nc1:10%' OR username = 'BronsonMike';

Change-Id: I0b3cc6fe9d7fb5f5645dd9c759da448bfe1e88f1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2255

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Flushes in metadata datasets are triggerred by entity update

logs, unlike regular transactions where flushes are triggerred

by entity commit logs.

- Because entity update logs can be writting to disk before the

operation completes, there is a chance that an operation that

caused the component to be full exits after the log is flushed

and so, a flush operation is not scheduled.

- This change proposes a simple fix. The fix is that metadata

operation will also check if a flush is needed and will schedule

one if needed.

Change-Id: I07a18840dc54fe052b7bd294595f816f6d8a4d2f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2413

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

- user model changes: no

- storage format changes: no

- interface changes: IAccessMethod, ILSMIndexOperationContext,

IIndexAccessor

Details:

- Implement an index-only plan

- Add a SET option that disables the index-only plan

Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1866

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Change-Id: I912ccef43c498e09fc9df4dbd5a3b6961d90aa33

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2395

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Also, enable resource id generation when only pending removal nodes are absent

from the cluster

Change-Id: I15cfb74bc345680102cedafa99f7ff4f144860bc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2389

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Murtadha Hubail <mhubail@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

- user model changes: no

- storage format changes: no

- interface changes: yes

- IIndexCursor.close() is now idempotent and can be called on

a closed cursor.

- IIndexCursor.destroy() is now idempotent and can be called

on a destroyed cursor.

- Add IIndexAccessor.destroy() letting the accessor know it is

safe to destroy its reusable cursors and operation contexts.

- Add IIndexOperationContext.destroy() letting the context

know that the user is done with it and allow it to release

resources

details:

- Previously, implementations of the IIndexCursor interface

didn't enforce the interface contract. This change enforces

the contract for all the implementations.

- With the enforcement of the contract, all the users of the

cursors are expected to follow and enforce the expected lifecycle.

- Test cases were added.

Change-Id: I98a7a8b931eb24dbe11bf2bdc61b754ca28ebdf9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2324

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Previously, when the cluster recovers and tries to notify the

active events handler to resume recovery tasks, the notification

is performed on the listener but the task waits on itself.

- After this change, the task waits on the listener and so

the notification wakes up the recovery task correctly.

Change-Id: I5fe3a2013a4cb92313648fd04ff2ef3c417270fb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2385

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

- user model changes: no

- storage format changes: no

- interface changes: yes

The metadata entity for a function now accepts a FunctionSignature.

It used to accept a dataverse, a function name, and an arity.

Change-Id: If00a914a1e56190ecd12c2687c32b4edc2895018

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2380

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Change-Id: Ide17ab0ea7553a6e8e22487b52ba0af379431f81

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2372

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Change-Id: I88f14fb351976db239ed752693e59882da62d588

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2368

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Michael Blow <mblow@apache.org>