Clone Tools
  • last updated 28 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE] Fix logging for ncservice-based tests

Change-Id: Id264aede0f62558ad6e34355047c623a1d594692

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3311

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 35 more files in changeset.
[NO ISSUE][FUN] Add Completed_Requests Function

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add completed_requests to get recently completed

requests.

- Use enum for requests state.

- Add new config to specify the requests archive

size.

- Add test case for completed_requests function.

Change-Id: I3f47d523c683c3879ec52ce5bdaf16ce338e8e46

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3301

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 23 more files in changeset.
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 625 more files in changeset.
[NO ISSUE][OTH] Add API To Ensure Request Requirements

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Refactor IRequestParameters as ICommonRequestParameters

and IRequestParameters to break cyclic dependencies.

- Add new API to ensure request can be scheduled for execution.

Change-Id: Ifb0513e0baf2b473006d4aa23040c86751fbb4fc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3181

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Murtadha Hubail <mhubail@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

  1. … 8 more files in changeset.
[ASTERIXDB-2518][RT] Introduce Request Tracker

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Introduce IReceptionist to generate request references.

- Track all requests by uuid.

- Add more information to active_requests response.

- Replace StatementExecutorContext by RequestTracker.

- Deprecate StatementExecutorContext (to be removed)

- Allow extensions to set optional parameters in query service.

- Return forbidden when a cancellation is attempt on a request

that is not cancellable.

Change-Id: If08ecd91c55881743b2ecf40a628fa3d4166c554

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3163

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -8
    • +16
    ./QueryCancellationServletTest.java
  1. … 40 more files in changeset.
[NO ISSUE][MISC] Introduce IClientRequest

- user model changes: no

- storage format changes: no

- interface changes: yes

+ IClientRequest: used to represent a client request

that can be cancelled.

Details:

- Introduce IClientRequest to allow for multiple types of requests

to be cancellable.

Change-Id: I8f65da1744ea7ecf26ea3f8a576ebaf4472ccd62

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2774

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 15 more files in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 982 more files in changeset.
[NO ISSUE][STO] Adapt Storage Structure To Rebalance

- user model changes: no

- storage format changes: no

- interface changes: yes

-- Added IResource#setPath to use for the resource

storage migration.

Details:

- Unify storage structure to support dataset rebalance:

Old format:

./storage/partition_#/dataverse/datasetName_idx_indexName

New format:

./storage/partition_#/dataverse/datasetName/rebalanaceNum/indexName

- Adapt recovery and replication to new storage structure.

- Add old structure -> new structure NC migration task.

- Add CompatibilityUtil to ensure NC can be upgraded during

NC startup.

- Centralize the logic for parsing file path to its components in

ResourceReference/DatasetResourceReference.

- Add storage structure migration test case.

- Add test case for recovery after rebalance.

Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2181

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 40 more files in changeset.
[ASTERIXDB-2171][STO] Remove Temporary Datasets

- user model changes: no

- storage format changes: no

- interface changes: yes

- Removed all reference to temporary datasets.

Details:

- Remove temporary datasets grammar but reserve

<TEMPORARY> token.

- Remove special callbacks for temp datasets.

- Remove metadata garbage collector thread.

- Remove isTemp flag from datasets.

- Remove special storage path for temp datasets.

- Remove temp datasets test cases.

Change-Id: Ib50ea27e2ec9c114508cd4a109cf646e2f20dcb0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2172

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 369 more files in changeset.
[ASTERIXDB-2058][HYR] Only Complete job cancellation after cleanup

- user model changes: no

- storage format changes: no

- interface changes: yes

--IJobManager.cancel now takes a callback

details:

- Job cancellation now completes only after the job cleanup work

has completed and not merely when the abort tasks are executed.

- The NCQueryServiceServlet actively cancels requests that passes

5 minutes.

- Cancellation of timedout jobs is not done through the Http API

but through message broker.

- Typically, requests might timeout when the servers are

overloaded. When that is the case, there is a high chance http

requests are to be rejected including requests to cancel

previously submitted queries. This is the reason for using

Message broker for this task.

- ExecuteStatementRequest used to execute the statement in

a different executor thread even though it is itself is being

executed in an executor thread and is not blocking anyone.

This was fixed as well.

Change-Id: I14b4bbd512cc88e489254d8bf82edba0fd3a3db5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1961

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 37 more files in changeset.
[ASTERIXDB-1992][ING] Suspend/Resume active entities

- user model changes: wait for completion only returns

when the entity becomes inactive.

- storage format changes: no

- interface changes:

01) Introduce:

IMetadataLockManager for entity locking.

IActiveNotificationHandler for handling active events.

IRetryPolicy for recovery of failed active jobs.

IActiveEntityController for controlling active entities.

02) IJobLifecycleListener.notifyJobFinish now passes the

JobStatus and Exceptions.

03) IActiveEntityEventsListener.isActive() returns true,

if entity is active, false otherwise.

04) IActiveEntityEventsListener.unregister() removes the

listener upon entity deletion.

05) IActiveEntityEventsListener.getJobFailure() returns

the job failure if the entity is in a failed state.

06) IStatementExecutor.getComponentProvider() returns

the storage component provider.

07) IStatementExecutor.getApplicationContext() returns

the application context.

08) IMetadataManager.upsertEntity to perfrom metadata

entities' upsert operations.

09) IMetadataNode.upsertEntity to perfrom metadata

entities' upsert operations

10) ICcApplicationContext.getMetadataLockManager() returns

the lock manger.

11) ICcApplicationContext.getClusterStateManager() returns

the cluster state manager.

details:

- Starting and stopping of active entities now go through

their listeners rather than having the listener indicating

active entity and not having it indicating inactive entity.

This facilitates suspend/resume operations of long running

active jobs for the sake of DDL operations or topology

changes.

- Unit tests for the vast majority of code paths and

different possible scenarios have been added.

Change-Id: Ifeac8c73e6bad39a13663b84a52121356e3c6b40

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1875

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 140 more files in changeset.
Extensible exception handling in QueryServiceServlet

Change-Id: If8037a97f3d0b0febb8caf68e099f1fd24e0ac49

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1836

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 11 more files in changeset.
Make Default Dir a Command Line Option

- Avoid using static field to hold default directory, in favor of

an option.

- Improve output on failed regexadm match (indicate where actual

and expected diverge)

Change-Id: I7855c8f344eea9c9b6a394d85413a062a3ddb609

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1751

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 15 more files in changeset.
Remove static cc application context instance

Change-Id: Ia2e250405967ec880e7af6387aa981f39b3392c0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1606

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 180 more files in changeset.
Ensure Metadata locks are acquired for SQL++ queries

Change-Id: I5f468599897a37cbcb12d8577d072f340f0d949c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1642

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 36 more files in changeset.
Add a REST endpoint for query cancellation.

This change also includes the following parts:

- Fix failure handling and potentially thread leakage in

MaterializingPipelinedPartition;

- Fix failure handling in PartitionDataWriter;

- Add a new test suite: SqlppExecutionWithCancellationTest.

Change-Id: I2936ac83f71bbef533e2695ed0a2b220c23fc483

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1564

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +92
    ./QueryCancellationServletTest.java
  1. … 45 more files in changeset.
Configuration Revamp

- Ini section of node / cc details now returns ini param names instead of

managix option names

- Normalized command line -vs- ini file configuration parameter names

- Eliminated unused parameters

- Ini validation

- Migrate *DB parameters out of [app] and into nc / cc sections as

appropriate

- Eliminate [app] section. Cluster-wide configuration lives in [common]

- Sort properties alphabetically when returned by HTTP api

Change-Id: I95b7e0bd4538ef42817c8826e76412150074b754

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1487

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 174 more files in changeset.
Improve reading from and writing to Metadata pages

This change introduces a new interface IComponentMetadata.

Each LSM component is associated with a metadata object

which can be used to read and write arbitrary data to

the metadata pages of components. When flushing a

component, data in its metadata component is automatically

flushed to the disk component. For merge operations,

the IO Callback is responsible for merging the components'

metadata pages.

Change-Id: Id95ef33c0a0bc1abb3fc3ecdea5611ee4acd6dfa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1476

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Integration-Tests: Ian Maxon <imaxon@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +180
    ./ConnectorApiServletTest.java
    • -0
    • +120
    ./VersionApiServletTest.java
  1. … 149 more files in changeset.
Introduce IStorageComponentProvider

Change-Id: If86750cdb2436c713f6598e54d4aaaf23d9f7bbf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1451

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 424 more files in changeset.
Use Chunked Http Response

Change-Id: I249180f58e92058dd3b264ea17c4196b4baf4348

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1474

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 36 more files in changeset.
Replace Servlets with Netty Based HTTP Servers

Change-Id: I3d552d4eb8c868535ca4c41cbcf7e352217b18ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1429

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +178
    ./ConnectorApiLetTest.java
    • -0
    • +46
    ./QueryServiceLetTest.java
    • -0
    • +120
    ./VersionApiLetTest.java
  1. … 62 more files in changeset.
Cleanup AsterixDB Dependencies

Enforce maven dependencies, to ensure that all declared dependencies are

used, and that all used dependencies are declared.

Change-Id: I3cc9d4db57b2138931baa3ef51ea569c45b79a2f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1421

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 43 more files in changeset.
ASTERIXDB-1714: Eliminate dependency on org.json

Change-Id: Ie9c5400fd134ae75d43385255af7794e968b1c7e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1392

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 132 more files in changeset.
Supports flexible degree of parallelism.

Detailed list of changes include:

- Add the IClusterInfoCollector interface and feed that to the compiler;

- Add the number of cores into the heartbeat;

- Add partition calculation in APIFramework;

- Fix partitioning property requirements in binary operators such as Join and UnionAll;

- Fix PushSubplanIntoGroupByRule for type propagation;

- Fix InlineLeftNtsInSubplanJoinFlatteningVisitor for the returned top join operator;

- Add unit test for partition calculation in APIFrameworkTest

- Add integration test SqlppExecutionFullParallelismIT, SqlppExecutionLessParallelismIT,

AqlExecutionFullParallelismIT, and AqlExecutionLessParallelismIT for different

parallelism settings;

- Refactor AQLExecutionTest and SqlppExecutionTest;

- Separate one cluster state test out from AqlExecutionTest.

Change-Id: If70faf52bed995c7098cc343f2fabd1b6c8d96e7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1398

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 114 more files in changeset.
ASTERIXDB-1711: remove more mentions of Asterix

Change-Id: I21666fd54075ac56de8aab76bb7d5cdbec020121

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1387

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

  1. … 570 more files in changeset.
Continue Cleaning Up File References and Splits

1. Make FileSplit an abstract class with two subclasses;

Managed and Unmanaged. A Managed FileSplit can be mapped

in a new subclass MappedFileSplit that maps a relative path to an

IO device. UnmanagedFileSplit is for files outside the io devices.

2. Remove all usages of absolute paths in file split in test cases. The

only remaining place is the write statement.

3. Fix some of the hidden issues in the tests that were working because

of our use of the absolute paths.

4. Revert the decision of selecting the IO device to the CC.

Change-Id: I166af8f9b3a2257f94d7b05db94888fb7cb4c79e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1359

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 253 more files in changeset.
Cleanup FileSplit and FileReference

This change gives FileSplit and FileReference specific meaning to

avoid confusion of an absolute vs relative, local vs global, inside

an IO device vs outside IO devices.

In addition, it enables better abstraction of global partitions and

delegate the responsibility of choosing which partition goes to which

IO device to the IO Manager through the introduction of FileDeviceComputer

In details:

Previously, the LocalResource in Hyracks had partition (storage partition)

and there is no such thing in Hyracks. This scope leak is bad. In addition

The local resource had a name and a path. they were always the same and so

the name was removed.

The storage partition was instead moved to asterixdb implementation of the

serialized object in the local resource.

With all of these changes, the cluster controller (compiler) only needs to

know about partitions and relative paths. It doesn't need to worry about

heterogenous Node setups and different io device configurations. For File

assignment to IO devices, a new interface (IFileDeviceComputer) was

introduced which can be overriden by applications to have their own

strategy for distributing files among IO devices.

Change-Id: I4fac508bf9af5a3bed41a3cf4464d2cbfecf2f61

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1352

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 287 more files in changeset.
ASTERIXDB-1711: remove some Aql-prefixes

Change-Id: Ia0b64ffa7c50cd62fc3303fdb44eb769f56c978a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1314

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 84 more files in changeset.
Refactor Messaging

In this refactoring, each message implementation includes

a handle method. This avoids bloating of message brokers

and enable better extensibility for messaging.

Change-Id: I7c918bf504058c98ecf89f5b019503278e9aa01f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1128

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 126 more files in changeset.
Extension Points For Servlets

Change-Id: Iff794b1c21296c50867b7430a0652924b5cf31d2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1019

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 14 more files in changeset.