Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE] Fix type inference for ANY in UDFs

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

When data type is available, we should use defined data type. If not, we

should try to infer the data type from the data.

Change-Id: I6a03749fe041ec6f43de230d3e784122f7664545

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3405

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 1 more file in changeset.
[NO ISSUE] Error codes + NCService getter + Record metadata getter

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Task, added getter for NodeControllerService.

- RecordWithMetadataAndPK, added getter for metaTypes.

- Added extra error codes.

Change-Id: Ic3fcc6b771b4393542988cd0aa6fd992ccb81028

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3423

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 3 more files in changeset.
[NO ISSUE][FUN] Introduce IEvaluatorContext

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduce IEvaluatorContext - a context for evaluators

that is available both at runtime and during constant

folding at compile time

- IEvaluatorContext provides access to IHyracksTaskContext

which is only available at runtime. At compile time this

accessor returns null

- Instantiate evaluators with IEvaluatorContext

instead of IHyracksTaskContext

Change-Id: Iff90c7739e46d6602fcd0609e67f6a8cd9714ca5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3481

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 687 more files in changeset.
[ASTERIXDB-2597] Load UDFs via HTTP

- POST existing UDF format to /admin/udf/$DATAVERSE/$LIBNAME

- DELETE against that URL to remove UDFs

Change-Id: I6be9fef54c010bdb32f5c78af9b973f9843f442f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3386

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 37 more files in changeset.
[NO ISSUE][FUN] Fix type inference and casting in UDFs

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

The current UDF framework handles the argument types in a sloppy way. It

takes in the arguments and reads them as the expected data types in the

configuration. This could cause exception at runtime when passing in

arguments with unexpected datatypes. When setting the arguments, it did

a type casting for numeric values only to make sure the int64 from query

interface can be evaluated properly. However, this is not robust enough.

This patch fixes the type inference for UDFs during the complation time.

The ExternalTypeComputer is refactored to return defined data type, and

meanwhile checks the argument data types. Also, the

IntroduceDynamicTypeCastForExternalFunctionRule is modified to cover the

type castings for data types besides record type.

Change-Id: I40506fcca3cd8f14bbd6412359683433256c4c1f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3270

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Xikui Wang <xkkwww@gmail.com>

  1. … 30 more files in changeset.
[NO ISSUE][FUN] Add Function To Return Secondary Index Tuples

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add a new private function (dump_index) that can be used

to query all tuples of a secondary index. This function

can be used for testing to validate the expected contents

of secondary indexes.

- Add test case.

Change-Id: Iefae2a798126b9efb049596417b93499d075a608

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3325

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 11 more files in changeset.
[NO ISSUE][RT] Fix Secondary Indexes Bulkload/Upsert

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Prevent null values from being upserted into

secondary indexes.

- Use secondary index tuple filter when bulkloading

to prevent null/missing values from being inserted

into indexes of optional fields.

Change-Id: I7b847aece06a4387cc41389b0566d0c0dc98487d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3304

Reviewed-by: Luo Chen <cluo8@uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 8 more files in changeset.
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +1
    ./apache/asterix/external/feed/api/IFeed.java
  1. … 611 more files in changeset.
[NO ISSUE][FUN] Fix type inference in JListAccessor

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

As the list element type is available, there is no need to deserialize

element types from the data. Doing so, for example, will cause the

object type to be fully open type and closed fields to be recognized as

open fields. This will further cause NPE in UDFs. Modified the upperCase

test case to check this issue.

Change-Id: I4f437857779e5c0af175c695938fbd7208da244d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3264

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 9 more files in changeset.
[NO ISSUE][OTH] Support log redaction

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Support log redaction.

Change-Id: I602c833ba2a055da8fbe8782ec62be683ff4581b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3105

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -9
    • +10
    ./apache/asterix/external/util/FeedLogManager.java
  1. … 29 more files in changeset.
[NO ISSUE][EXT] Fix JRecord setField for open fields

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

setField method should be able to handle open fields.

Change-Id: I4ec185d62e78e8100215334bc656586268314797

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3068

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 1 more file in changeset.
[NO ISSUE][ING] TweetParserFactory fix

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Fix the TweetParserFactory so it returns the correct record class.

Change-Id: If330bf060423027e01787224d1b2e93fd4d33d71

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3050

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wail Alkowaileet <wael.y.k@gmail.com>

[NO ISSUE][ING] Tweets ingestion related refactoring

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

As the incoming Tweets are merely JSON records, the Twitter adapter

should be able to use JSON parser. For that purpose, we changed the

record record to be consistent with JSONDataParser which uses char[]

instead of String. Also, "JSON" should also be a valid way to specify

using JSON parser.

Change-Id: Id6d656a4af974499ef0df2e389152ca205c2d078

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3032

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wail Alkowaileet <wael.y.k@gmail.com>

  1. … 2 more files in changeset.
[ASTERIXDB-2480][ING] Fix http-feed compatibility with JSON parser

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

The generic record used in current http-feed returns length of "-1"

which will cause the JSON parser to fail. For compatibility

consideration, we should use CharArrayRecord instead.

Change-Id: I719c98376583a3316dbc291696df65a97dd4b26e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3026

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wail Alkowaileet <wael.y.k@gmail.com>

  1. … 9 more files in changeset.
[ASTERIXDB-2286][COMP][FUN][HYR] Parallel Sort Optimization

- user model changes: yes

- storage format changes: no

- interface changes: yes

details:

- new plan for sort operation which includes sampling and

replicating the stream of data to be sorted. Sort-merge connector

is removed from the plan. The sorted result now is in multiple partitions.

- new optimization rule to check whether full parallel sort is applicable.

- new Forward operator to read the replicated sort input stream and

to receive the ouput of the sampling.

- new sequential merge connector to merge a globally ordered result residing

in multiple partitions (in addition to the connector's partition computer).

- "asterix-lang-aql/pom.xml" is changed as a result of refactoring

code related to the range map handling.

- new private sampling function to generate the range map object

(local & global functions) & their type computers.

user model changes:

- new compiler property is added to enable and disable parallel sort.

interface changes:

- "ILogicalOperatorVisitor.java" includes Forward Operator.

- "ITuplePartitionComputer.java" includes initialize() to enable partitioner

to do some initialization. FieldRangePartitionComputerFactory uses it to

pick a range map.

- "ITuplePartitionComputerFactory.java". createPartitioner() is changed to

createPartitioner(IHyracksTaskContext hyracksTaskContext). Context is needed

for transferring the range map throught the context.

Change-Id: I73e128029a46f45e6b68c23dfb9310d5de10582f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2393

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 356 more files in changeset.
[ASTERIXDB-2264][ING] Introduce Http Feed Adapter

- user-model changes: add http_adapter for feed.

- storage format changes: no

- interface changes: no

Details:

1. Added http feed. User may use following syntax to create a http feed:

create feed TweetFeed with {

"adapter-name" : "http_adapter",

"addresses" : "asterix_nc2:10002,asterix_nc1:10001",

"address-type" : "NC",

"type-name" : "TweetMessageType",

"format" : "adm"

};

2. Refactored insert-feed test case to avoid waiting for 10s.

3. Refactored some constants in feeds.

Change-Id: I3c197a3df557ecc01f07f0907688c4ea81379e40

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2994

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 14 more files in changeset.
[NO ISSUE] User-defined Function Documentation update

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

1. Updated the UDF documentation to be consisten with current master.

2. Cleaned default UDF package to remove useless UDFs.

3. Added the example in documentation as a test case for IT.

4. Reorganized the documentation to keep up with the new structure.

5. Minor changes to other documentation pages to keep style consistent.

Change-Id: I17b1b4d639ca38689298ce88145257e794eb90e1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2804

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 27 more files in changeset.
[NO ISSUE][RT] Improve debuggability of runtime functions

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- This change introduce util methods to facilitate debugging

functions runtime. Runtime functions read and write

values from and into byte arrays.

- To allow easier inspection of such values, we provide

a method that takes a pointable and produces an adm

node representing the value pointed to by the pointable.

Change-Id: I33257507e9bdd3be80355218e4bafecd1c71c650

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2926

Reviewed-by: Till Westmann <tillw@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 4 more files in changeset.
[NO ISSUE][OTH] Remove Unnecessary Null Checks

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Remove unnecessary null checks before instanceof.

Change-Id: I870df76623f5f983a90b5437abf7e3993a54c744

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2895

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 14 more files in changeset.
[NO ISSUE][OTH] Remove Unused Imports

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: Iafff39073d0fedaff74a26ef7e3260008a79ff0c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2893

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 59 more files in changeset.
[NO ISSUE][OTH] Cleanup Unnecessary Semicolons

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: I73141e6a5d7950c074a80ea664fa6109ff94cdc4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2888

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 55 more files in changeset.
[NO ISSUE][OTH] Cleanup Redundant Type Cast

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: I9c93ce90ad4aa789ae46284431e172ae4777bf55

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2887

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 30 more files in changeset.
[NO ISSUE][FUN] Bug fixes in JObjectAccessors

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

1. Align the deserialization start of several datatypes.

2. Assign the open fields in JRecord properly in JRecordAccessor.

Change-Id: I131f1bf70f01cd763cfd008abcba704e757c864e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2799

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

[NO ISSUE][ING] Add timeout to stop active message

Change-Id: Ie0416d76670e945cb958f5a1c235201a3e016009

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2762

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 9 more files in changeset.
[NO ISSUE][ING] Cleanup active listener

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Merge STOPPED and PERMANENT_FAILURE into a single

state since they are equivalent.

- Allow timeout of start and stop of active entities.

- Add test cases for timeout scenarios.

- Refactor common code into the abstract listener class.

Change-Id: I7f3b14aec46728fbe8b256b915d0e30992b2fe47

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2618

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 11 more files in changeset.
[ASTERIXDB-2401][SQLPP] Support parameterized queries

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Support statement parameters: named ($name) and positional ($1 or ?)

- Enhance query service API to accept these parameters in the request

- Remove [?] index accessor from SQL++ grammar because it conflicts

with positional parameters ([0] can be used instead)

- Add testcases for parameterized queries

Change-Id: Ia612f731cd2370fccd54c4796bd9787fbea16766

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2707

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 104 more files in changeset.
[NO ISSUE][ING] Allow Readers to add their own stats

- user model changes: no

- storage format changes: no

- interface changes: yes

- Add IRecordReader.getStats()

Details:

- Allow record reader to contribute their own stats.

Change-Id: Ib064bd97b33f2d9da330bac3a49bd4c6831faaca

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2642

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 1 more file in changeset.
[ASTERIXDB-2393][COMP][RT] Add source location to error messages

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add source locations to compiler and runtime error messages

- Add source location enforcement to the test framework

and enable it for SqlppExecutionTest

Change-Id: Ie279ce345d1edcb5dea8e55cdb0233151c7bfd66

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2659

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 679 more files in changeset.
[NO ISSUE][RT] Make start and cancel job uninterruptible

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Previously, a Hyracks data connection that starts

a job might be interrupted before it receives the

job id. This would leak resources since that job

will run even though, no one is going to read its

result.

- Similarly, job cancellation can be interrupted and

so the job which was meant to be cancelled will

continue running.

- To avoid this, a new thread is added to Hyracks

Client Connection which takes care of starting

and cancelling of jobs. The thread submitting these

requests will be un-interruptible until those calls

return.

Change-Id: I27b2aaae902b19829bd2df2ae04c5e704f5ca8e8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2639

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 2 more files in changeset.
[NO ISSUE][STO] Misc Storage Fixes and Improvements

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- This change introduces some improvements to storage

operations.

- Local RecoveryManager is now extensible.

- Bulk loaders now call the IO callback similar to

Flushes, making them less special and creating a

unified lifecycle for adding an index component.

- As a result, The IndexCheckpointManager doesn't need

to have a special treatment for components loaded

through the bulk load operation.

- Component Id have been added to the index checkpoint

files.

- Cleanup for the code of local recovery for failed flush

operations.

- Ensure that after local recovery of flushes, primary

and secondary indexes have the same index for mutable

memory component.

- The use of WAIT logs to ensure in-flight flushes

are scheduled didn't work as expected. A new log type

WAIT_FOR_FLUSHES was introduced to acheive the expected

behavior.

- The local test framework was made Extensible to support

more use cases.

- Test cases were added for component ids in checkpoint files.

The following scenarios were covered:

- Primary and secondary both have values when a flush is

shceduled.

- Primary have values but not secondary when a flush is

scheduled.

- Primary is empty and an index is created through bulk

load.

- Primary has a single component and secondary is created

through bulk load.

- Primary has multiple components and secondary is created

through bulk load.

- Each primary opTracker now keeps a list of ongoing flushes.

- FlushDataset now waits only for flushes only and

not all io operations.

- Previously, we had many flushes scheduled on open datasets.

This was not detected but after this change, a failure

is thrown in such cases.

- Flush operations dont need to extend the comparable

interface anymore since they are FIFO per index.

Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2632

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 86 more files in changeset.