Clone
 

yingyi bu <yingyi@couchbase.com> in asterixdb

[ASTERIXDB-2051][COMP] Fix PushSubplanIntoGroupByRule for complex cases.

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- re-implement PushSubplanIntoGroupByRule and let it handle general cases;

- add an option to LogicalOperatorDeepCopyWithNewVariablesVisitor for

not re-mapping free variables in the given plan subtree.

Change-Id: I969c40112be0506981357a9c41bf9675ae12ffb9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1992

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 3 more files in changeset.
[ASTERIXDB-1812][RT] Budget the memory usage for pre-clustered group-by.

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- let pre-clustered group-by consider memory budget.

Change-Id: I670269b0b8f446d06d8dd73202194574aa524e85

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1940

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 13 more files in changeset.
[ASTERIXDB-2044][COMP] Eliminate listify for complex group-by

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fix EliminateSubplanWithInputCardinalityOneRule to handle recursive

subplans;

- Fix various places that assumes the nested plans inside a group-by

operator cannot be empty;

- Added regression tests.

Change-Id: Ida9aa8d89a89f90256e54c8c1806af9b4a162d21

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1946

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

[ASTERIXDB-2032]Let stop-sample-cluster.sh to kill processes.

Change-Id: I44d75e518cca604273790aa1d862ca5764fa6395

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1933

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[ASTERIXDB-1897][COMP][RT] Fix complex group-by.

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fix type computer for numeric aggregations;

- Fix error reporting for SubplanRuntimeFactory;

- Add a negative test query.

Change-Id: Iebb393820a8edd0c54d80248b2a33c77d4f6fd7b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1923

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[ASTERIXDB-2000][RT] Fix roundtrip-ability for INF in JSON format.

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Use INF to represent infinity.

Change-Id: I13687822433744afe1ad3439029b2c0887563f9b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1901

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -0
    • +1
    /asterixdb/asterix-app/data/infinity.adm
  1. … 20 more files in changeset.
[DOC] Reorganize data model documentations.

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Re-organize data model documentations;

- Align results in the function doc with JSON;

- Fix a few minor issues in the function doc.

Change-Id: I6a2306747631c25d46c5246643abc1e147cf7c21

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1898

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

[Doc]reorganize DDL documentations.

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Re-organize DDL documentations;

- Move declaration to queries;

- Fix several links;

- Add an example for non-enforced index.

Change-Id: I269750827ef529ac0e474fa0862635cc1f3aefed

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1892

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +81
    /asterixdb/asterix-doc/src/main/markdown/sqlpp/5_ddl_dml.md
    • -0
    • +37
    /asterixdb/asterix-doc/src/main/markdown/sqlpp/5_ddl_head.md
[ASTERIXDB-1664][FUN][DOC] Fix position/regexp_postion to be 1-based

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Change function position and regexp_position to be 1-based;

- Re-organize function documentations.

Change-Id: I415e0ba0ac4dfceba95913ff27af95c0c1af5fee

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1888

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-1982][FAIL] Unify runtime error reporting

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Unify error reporting in result distribution and operator pipeline;

The fact that ASTERIXDB-1982 is sporadic is because the order of

DatasetDirectoryService.reportJobFailure(...) and

DatasetDirectoryService.reportResultPartitionFailure(...)

is not deterministic and the latter can override the former;

- Make the order of setException and addWaiter irrelevant to

avoid sporadically slipped exception;

- Fix the exception list in Task to be thread-safe.

Change-Id: I36f243c98876ff40e2539ca9241ff6d19fee929f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1883

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

[ASTERIXDB-1985][CLUS] Add rebalance callback

Change-Id: I9a90ba975467c136371236195f82d48430d8319d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1863

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

[ASTERIXDB-1986][DOC] Remove auto plural documentation.

Change-Id: If18db7bea1a2b51f6f75f73ed5970eaa1c6e0437

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1878

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

[ASTERIXDB-1753][SQL] Disable auto-plural for group-by.

- user model changes: do not perform auto-scalar-to-plural

transition for variables that are defined prior to a group-by

group clause after the group-by clause.

- storage format changes: no

- interface changes: no

details:

- migrated existing test queries that use auto-plural;

- added a negative test query.

Change-Id: I5d6383fc7453fe7537fc291c3483708b59df3871

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1860

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 247 more files in changeset.
[ASTERIXDB-1943][API][STO] Make rebalance idempotent.

- user model changes:

added rebalance cancellation HTTP API.

- storage format changes: no

- interface changes: no

Details:

- add a HTTP API for cancelling a rebalance request;

- clean up leftover states at the beginning of a

rebalance request;

- add tests for rebalance cancellation.

Change-Id: I0d14a07978e106cd497cc35538fafef318b2fcf7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1821

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 3 more files in changeset.
[ASTERIXDB-1945][STO] Cleanup Buffer Cache API

- user model changes: no

- storage format changes: no

- interface changes: yes

INcApplicationContext

- removed IFileMapProvider getFileMapManager();

to hide FileMapManager from other components;

IStorageManager

- IFileMapProvider getFileMapProvider(INCServiceContext ctx);

to hide FileMapManager from other components;

IFileHandle

- added FileReference getFileReference();

to avoid unnecessary casts;

IIOManager

- public void deleteWorkspaceFiles() throws

HyracksDataException;

added throws;

ILSMIndexFileManager

- void createDirs() throws HyracksDataException;

added throws;

IInvertedIndex

- added void purge() throws HyracksDataException;

a. InvertedIndexes don't implement the ITreeIndex interface.

b. when we deactivate a disk component, we need to purge it so

the buffer cache doesn't go through each page.

c. this need to be revisited, ASTERIXDB-1944

IFileMapManager

- int registerFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- FileReference unregisterFile(int fileId) throws

HyracksDataException;

return value added for future refernece of the file;

IBufferCache

- int createFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- void deleteFile(int fileId) throws HyracksDataException;

remove the dirty page flag since there's no dirty page;

- int openFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- added void deleteFile(FileReference file) throws

HyracksDataException;

we used to have this public methods in both BufferCache

and VirtualBufferCache. Now we lifted it into the interface.

AbstractLSMIndex

- removed protected abstract void

destroyMemoryComponent(ILSMMemoryComponent c)

throws HyracksDataException;

It is because turned out when we deactivate, we actually

destroy them. However, because of the not well defined API,

double destroy was okay and so we used to do double destroy.

Details:

This change fixes the buffer cache to follow the API such that:

1. createFile creates the file.

2. deleteFile deletes the file.

3. openFile opens the file.

4. closeFile closes the file.

5. creates existing file is not allowed.

6. deletes deleted file is not allowed.

7. open non existing file is not allowed.

In addition, we hide the file map from all other components.

Change-Id: I0a973c2adb2e7fdcbbf18c7b888af3de5f0acc74

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1843

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 147 more files in changeset.
Cancel the on-going job if waitForCompletion is interrupted.

Change-Id: I3417271660e815a13fd706e1cc057bca6a625c37

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1825

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Rename TaskUtil.getInSharedMap(..) -> TaskUtil.get(..).

Change-Id: Ic795c988ca63db7c760a2c8e7638451945cb72aa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1817

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Miscellaneous minor cleanups.

Change-Id: I08911bc62c573da483174a71b71949162ca229f2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1815

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Clean up exceptions in runtime functions.

Change-Id: Ie08d6de31d9e928e888137680d9c7af5d66b85ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1814

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 15 more files in changeset.
Remove default node group.

In this way, CREATE DATASET statement can adjust to dynamic

cluster topology.

When we create a dataset:

- if the node group name is not given, we create a new node group

using all currently available nodes;

- if the node group name is give, we use the given node group for

the dataset.

When we drop a dataset:

- if no other dataset depends on the node group of the dataset to

be dropped, we also drop the node group.

Change-Id: If68dc6a7c1270ab1f5049c9334e3318425fd8287

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1799

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 37 more files in changeset.
Support rebalancing all datasets or a given dataverse.

Change-Id: Iad2740fd53b36bf122fd469beeca759d887e40fb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1793

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 53 more files in changeset.
Support rebalancing datasets with indexes.

- Remove type arguments from the methods in IndexUtils

that generate index operation (e.g., create, load, compact)

jobs. Do type extraction inside SecondaryIndexOperationsHelper.

Change-Id: I9c0720382440ae44441a8f8847e75649a3822fa2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1790

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 21 more files in changeset.
Add a dataset rebalance REST API.

- Failures during rebalance are not handled;

- Indexes are not built for the rebalance target.

Change-Id: Ibda35252031fc4940972f0f19bbf796cadfa53d6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1768

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 61 more files in changeset.
Use JRE for AWS installation.

Change-Id: I25ca959f53c49c7aa1e2841227d27d7c254578be

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1776

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -0
    • +26
    /asterixdb/asterix-server/src/main/opt/aws/yaml/install_jre.yml
ASTERIXDB-1918: support scientific math representation.

Change-Id: Ic0b6661416751a82a552cebc6248596b4eeff500

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1777

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Improve metadata two-phase locking error message.

- Enforce two-phase locking through a state;

- Clear the lock list in unlock().

Change-Id: I6fd8ddf62e5cbd5500e84a3acdbdb680ac068748

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1769

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Reviewed-by: Michael Blow <mblow@apache.org>

ASTERIXDB-1915: make dataset files uniformly distributed among io devices.

Change-Id: I2dd9e17e96c1d4ef55e29d0a0f8feadf8ce321ed

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1770

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Disable TPC-H q7_variant.

Change-Id: I5ad257ac170823417e9aa434d74ca4e586162afa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1750

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Remove unused methods in DatasetUtil, IndexUtil, and MetadataProvider.

Change-Id: I60352d052f684a65fd823a1c7aa33b794bd7f476

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1741

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Update README.

Change-Id: I9c5c7434faa96875554316741d7caffbfe52c0b1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1744

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>