Clone Tools
  • last updated 28 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][COMP] Reduce memory requirements for monitoring functions

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Queries that only invoke monitoring/metadata datasource functions

should run with minimal memory requirements

- The functions are: active_requests(), completed_requests(),

dataset_resources(), storage_components(), jobs(), and ping()

Change-Id: If7de53a5b476c8ce0d3fe485bf526a5d425e4c37

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3548

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 27 more files in changeset.
[ASTERIXDB-2593][FUN] TPC-DS always parallelize + gen all tables

- user model changes: yes

- user can call the tpcds_datagen() by passing only the scaling

factor, and that will automatically generate all the tables.

- storage format changes: no

- interface changes: no

Details:

- Added support to a second version of the tpcds_datagen()

function to take only a single parameter that will result in

generating the data for all the tables in one go. The user

now can generate the data for a single table at a time, or

all the tables in one go.

- Overridden the behavior for activating the parallelism.

Before, the library wouldn't activate parallelism unless

the tables are big, and the number of data generated per

table is over 1,000,000. With this change, the parallelism

is always activated, regardless of the table size or the

data size being generated.

- Added a new test, TPCExecutionTest, to execute the long

tests for the TPC. Some of the TPC tests take longer than

5 minutes to finish and could result in high usage of

disk space.

Change-Id: Iff199b0c533d22bcae1caf5057788b257ba4e486

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3437

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 20 more files in changeset.
[NO ISSUE][FUN] Introduce data generator function

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduced data generator function to generate data with accordance

to the TPC Benchmark DS specifications.

- Data generator function accepts 2 arguments, table name and scaling

factor, this makes the same function reusable for all the desired

tables and scaling factors to be generated.

- Added test cases for the data generator functions.

Change-Id: Idb6bd6f697628395c70008e6f730bc5ca403da5e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3360

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 14 more files in changeset.
[NO ISSUE][FUN] Add Function To Return Secondary Index Tuples

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add a new private function (dump_index) that can be used

to query all tuples of a secondary index. This function

can be used for testing to validate the expected contents

of secondary indexes.

- Add test case.

Change-Id: Iefae2a798126b9efb049596417b93499d075a608

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3325

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 11 more files in changeset.
[NO ISSUE][FUN] Add Completed_Requests Function

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add completed_requests to get recently completed

requests.

- Use enum for requests state.

- Add new config to specify the requests archive

size.

- Add test case for completed_requests function.

Change-Id: I3f47d523c683c3879ec52ce5bdaf16ce338e8e46

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3301

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 23 more files in changeset.
[NO ISSUE] Add internal function jobs() to retrieve job information

- user model changes: added function jobs()

- storage format changes: no

- interface changes: no

Change-Id: I80d786899d6dd5970c8faf24041bc60da881a365

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3115

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 9 more files in changeset.
[ASTERIXDB-2513][FUN] Add Active_Requests Function

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add a datasource function (active_requests) which

returns the active jobs that the user specified

client_context_id for.

- This function runs on a single NC and uses messaging

to get the currently running jobs from CC.

- Currently, the function returns the following fields:

-- clientContextId: the user specified clientContextId.

-- requestTime: a timestamp at which the request reference

was created.

-- jobId: optionally, the job id that belongs to this request.

- The function may be improved later to return all jobs and it may

return additional fields such as (request uuid, statement,

executionTime, elapsedTime, nodeAddress, userAgent, etc..)

- Add test case.

- Do not allow cancellation test to cancel queries with

clientContextId to avoid intermittent failures.

Change-Id: I95962742161ed18c4cf2e09c8541c8ad3b35356c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3136

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 24 more files in changeset.
[NO ISSUE][MISC] Add ping function

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add a ping function that takes 0 parameters

and simply returns a single record from each node

which contains the node id.

Change-Id: Ic077a11b61dec782756e2dede33ca8220893a2de

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2764

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 7 more files in changeset.
[ASTERIXDB-2195][REPL] Replace Static Replication

- user model changes: no

- storage format changes: no

- interface changes: yes

- Redesigned all replication interfaces

Details:

- Replace static replication and fault tolerance by

dynamic storage API.

- Remove static based fault tolerance strategies.

- Redesign replication APIs and classes to smaller

maintainable parts.

- Clean up replication properties.

- Unify logic for checkpoints when replication is

enabled.

- Remove static replication test cases.

- Add replication runtime test cases for:

- Bulkload component replication.

- Memory component recovery.

- Flushed component replication.

- Add replication integration test for:

- Resync failed replica.

Change-Id: Ic5c4b0ac199a4530c807e558c8aebb1eb1284048

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2252

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 252 more files in changeset.
[ASTERIXDB-2194][COMP] Introduce datasource functions

- user model changes: yes

Some functions can be datasources

- storage format changes: no

- interface changes: yes

- Add IDatasourceFunction: A function that is also a datasource

- Add IFunctionToDataSourceTransformer: transform an unnest

function into a datascan during compilation

Details:

- Currently, functions are location agnostic and are run on

parameters that are either passed through them during compile

time or runtime.

- An exception to this is the dataset function which has

an associated location constraints running on the nodes

which host the dataset.

- In this change, we introduce a general framework that allows

creation of new functions similar to the dataset function.

- Such functions are called datasource Functions.

- A datasource function takes constant parameters and run on

a set of partitions similar to the dataset function.

- The first example of such functions is the DatasetResources

function.

- The DatasetResources function takes two parameters, a dataverse

and a dataset. It is then run on all nodes and returns a set

of dataset resources.

- Test cases are added for this function.

Change-Id: Ibcf807ac713a21e8f4d59868525467386e801303

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2216

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +60
    ./MetadataBuiltinFunctions.java
  1. … 41 more files in changeset.
[ASTERIXDB-1564][CONF] Consistently Use Log4j2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace java.util.logging by Log4j2.

- Excluded classes due to their tests:

- IoUtil

- Tracer

Change-Id: Ic137571292f45de1f1994c61d328b97185012197

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2226

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 390 more files in changeset.
[ASTERIXDB-1723] Delete managix

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Delete managix installer method and all dependencies

- Remove xml-based configuration

Change-Id: I5c710b6ec870bedd49dd46b74ed96a3665670e9b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2155

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1378 more files in changeset.
[ASTERIXDB-2107][CLUS] Prevent Invalid UNUSABLE State in Dynamic Topology

- user model changes: no

- storage format changes: no

- interface changes: yes

Renamed IClusterStateManager add/Remove NCConfig methods

to notifyNode join/failure.

Details:

- Mark node as participant when it completes its startup and

not when it joins the cluster.

- Allow partitions to be added with pending activation state.

- Remove the use of static MetadataProperties for reporting number of nodes.

- Add test cases.

Change-Id: I7a0db2d66cf44650dcc673b3f2de537816cb84c7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2029

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 10 more files in changeset.
Configuration Revamp

- Ini section of node / cc details now returns ini param names instead of

managix option names

- Normalized command line -vs- ini file configuration parameter names

- Eliminated unused parameters

- Ini validation

- Migrate *DB parameters out of [app] and into nc / cc sections as

appropriate

- Eliminate [app] section. Cluster-wide configuration lives in [common]

- Sort properties alphabetically when returned by HTTP api

Change-Id: I95b7e0bd4538ef42817c8826e76412150074b754

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1487

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 174 more files in changeset.
Exceptions Cleanup for Replication/FaultTolerance Strategies

Change-Id: I8f28b8db42bf7c8537ff2da22cbd2a97e243f32f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1529

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 18 more files in changeset.
Introduce Strategy Based Replication and Fault-Tolerance

This change includes the following:

- Introduce new APIs for Replication and Fault-Tolerance Strategies.

- Add configuration in cluster description file for high-availability.

- Add built-in replication strategies (Metadata_Only, Chained_Declustering)

- Add built-in fault-tolerance strategies (Auto, Metadata_Node)

- Remove none cluster state functionality from ClusterStateManager.

- Add customizable NC startup sequence

Change-Id: I1d1012f5541ce786f127866efefb9f3db434fedd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1405

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +67
    ./FaultToleranceUtil.java
  1. … 111 more files in changeset.
Introduce IStorageComponentProvider

Change-Id: If86750cdb2436c713f6598e54d4aaaf23d9f7bbf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1451

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 425 more files in changeset.
Implements concurrent query management support.

The following changes are included:

-- factor out JobManager, NodeManager, and ResourceManager from ClusterControllerService;

-- let each application plugin its own IJobCapacityController implementation;

-- let each job specify its required cluster capacity;

-- add a required cluster capacity estimation visitor for optimized query plans;

-- add admission control and queuing for queries, but always executes DDLs and DMLs immediately;

-- add tests for JobManger, NodeManager, ClusterCapacity, ClusterCapacityVisitor, and IJobCapacityController;

-- enlarge the -Xmx setting for ManagixSqlppExecutionTest.

Change-Id: I8fb6fda57efa139114dd234e08cc7de7129468c8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1424

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +70
    ./ResourceUtils.java
  1. … 119 more files in changeset.
ASTERIXDB-1711: remove more mentions of Asterix

Change-Id: I21666fd54075ac56de8aab76bb7d5cdbec020121

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1387

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

  1. … 570 more files in changeset.
ASTERIXDB-1711: remove some Aql-prefixes

Change-Id: Ia0b64ffa7c50cd62fc3303fdb44eb769f56c978a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1314

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 84 more files in changeset.
Refactor Messaging

In this refactoring, each message implementation includes

a handle method. This avoids bloating of message brokers

and enable better extensibility for messaging.

Change-Id: I7c918bf504058c98ecf89f5b019503278e9aa01f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1128

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 126 more files in changeset.