asterixdb

Clone Tools
  • last updated 12 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
ASTERIXDB-1556, ASTERIXDB-1733: Hash Group By and Hash Join conform to the memory budget

- External Hash Group By and Hash Join now conform to the memory budget (compiler.groupmemory and compiler.joinmemory)

- For Optimzed Hybrid Hash Join, we calculate the expected hash table size when the build phase is done and

try to spill one or more partitions if the freespace can't afford the hash table size.

- For External Hash Group By, the number of hash entries (hash table size) is calculated based on

an estimation of the aggregated tuple size and possible hash values for the given field size in that tuple.

- Garbage Collection feature has been added to SerializableHashTable. For external hash group-by,

whenever we spill a data partition to the disk, we also check the ratio of garbage in the hash table.

If it's greater than the given threshold, we conduct a GC on Hash Table.

Change-Id: I2b323e9a2141b4c1dd1652a360d2d9354d3bc3f5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1056

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 27 more files in changeset.
Use a file to store the map from error codes to error messages.

Change-Id: I344903a9961469b62a951a9095514bf671ff11ee

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1410

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Remove vestiges of Virtual pages

This concept was altered into confiscated pages, but some of the old

vestiges stuck around. This is just to remove them as the code is

not used and the terminology is very easy to conflate with the

terms used in the LSM wrapper.

Change-Id: I9ce8ea25a021d069e02eb15fcb80d703c06a883c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1413

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Full-text implementation step 3

- Implemented index-based full-text contains() function.

- Fixed a bug in the full-text code that can't process join cases well.

- Introduced full-text type index syntax. The functionality was already

in the codebase but it was not being used. So, using this syntax, we

can utilize the given index type - Not the length partitioned keyword index.

- Added the support for SQL++

- Default option will be set if a user doesn't provide any option.

Change-Id: I1087854ac7cf5b6ef5094e27a1646f12f6a8653f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1388

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 149 more files in changeset.
ASTERIXDB-1755: add UPSERT in SQL++.

Detailed list of changes included:

- Fix AQL RETURNING clause to use an expression instead of a query;

- Add a rule to check whether a RETURNING expression contains dataset accesses;

- Perform AST-level rewrites for INSERT/UPSERT as a whole, including the

body query and the returning expression.

Change-Id: I02b2be7ff2653573eccb48037895f5c8c4bc8c74

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1401

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 89 more files in changeset.
ASTERIXDB-1650: fix undefined variable reference resolution.

The order of resolving a reference to an undefined variable is:

-- If the reference to an undefined variable is the binding expression for FROM/JOIN/UNNEST/Quantifier,

then it can only be resolved to a dataset;

-- Otherwise, it is firstly resolved as a field-access. If the system couldn't find a candidate

field access, it is then resolved to a dataset.

Change-Id: I24e4c1b38e53c97380cfb3e2e9b61cdd05fe7002

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1408

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 72 more files in changeset.
Add NCService documentation for compiler.parallelism.

Change-Id: I00ffcc402238479d67f22a81e97081ea313d9f99

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1407

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

ASTERIXDB-1758: NCService documentation fix.

Change-Id: I2c74f272ba7d207f09c777853291e951195913c8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1406

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

ASTERIXDB-1711: remove more Aql-prefixes

AqlMissingWriterFactory -> MissingWriterFactory

AqlPositionWriter -> PositionWriter

AqlMergeAggregationExpressionFactory -> MergeAggregationExpressionFactory

AqlMissableTypeComputer -> MissableTypeComputer

AqlPartialAggregationTypeComputer -> PartialAggregationTypeComputer

AqlOptimizationContextFactory -> OptimizationContextFactory

Change-Id: I72f3051ff5c76ccde54e8b3b33941eb936870194

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1386

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Cleaning up typos and clarifying examples in string function doc.

Change-Id: I944d9dd1d1e23608b6a596190c3d89e19cfacead

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1403

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Supports flexible degree of parallelism.

Detailed list of changes include:

- Add the IClusterInfoCollector interface and feed that to the compiler;

- Add the number of cores into the heartbeat;

- Add partition calculation in APIFramework;

- Fix partitioning property requirements in binary operators such as Join and UnionAll;

- Fix PushSubplanIntoGroupByRule for type propagation;

- Fix InlineLeftNtsInSubplanJoinFlatteningVisitor for the returned top join operator;

- Add unit test for partition calculation in APIFrameworkTest

- Add integration test SqlppExecutionFullParallelismIT, SqlppExecutionLessParallelismIT,

AqlExecutionFullParallelismIT, and AqlExecutionLessParallelismIT for different

parallelism settings;

- Refactor AQLExecutionTest and SqlppExecutionTest;

- Separate one cluster state test out from AqlExecutionTest.

Change-Id: If70faf52bed995c7098cc343f2fabd1b6c8d96e7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1398

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 101 more files in changeset.
ASTERIXDB-1754: fix NPE in CopyLimitDownRule.

Change-Id: Ia9e26912a048c680aa9db7a12cd3933a7acbde37

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1399

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

ASTERIXDB-1751: Spilled partition files will be cleaned properly.

- Properly closes the spilled parition file during the build phase of hash join

so that the file will be deleted after the hash join finishes.

Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1395

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Introduce CheckpointManager API

This change includes the following:

- s/CheckpointObject/Checkpoint

- Add AsterixDB storage version to checkpoints.

- Prevent any txn log access when a storage version mismatch is detected.

- Introduce CheckpointManager API and CheckpointProperties.

- Properly stop checkpointing thread on instance shutdown.

- Separate checkpointing logic when replication enabled/disabled.

Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1380

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 6 more files in changeset.
Move ExceptionTest to Integration Tests

Change-Id: Iff3afa39bddf4b1e75f345afcf2a55842755ca74

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1391

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Fix ASTERIXDB-1255

This changes enable dynamic hashtable size in SimilarityJaccardEvaluator

Change-Id: I08c550d35d864df08792369d38ff81012e3976af

Reviewed-on: https://asterix-gerrit.ics.uci.edu/586

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Remove Incubator Disclaimer

Change-Id: Ib68d1c6a925559c1ab369ca26debaeef1e961b99

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1390

Reviewed-by: Till Westmann <tillw@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +0
    /asterixdb/asterix-external-data/pom.xml
    • -1
    • +0
    /asterixdb/src/main/assembly/source.xml
  1. … 6 more files in changeset.
ASTERIXDB-1711: remove more mentions of Asterix

Change-Id: I21666fd54075ac56de8aab76bb7d5cdbec020121

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1387

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

  1. … 556 more files in changeset.
Fixing some doc bugs related to numeric functions.

Change-Id: I692586bb408f4d939b4d16bdaf784822c60c9aca

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1312

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Full-text implementation step 2

- Parameter checking during the compilation is now applied.

Change-Id: Idec6b602ff7797846fd237a924005031c2395346

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1383

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Remove Append Only Flag

Change-Id: Id5d6917db8ab29aa01521596f556006e25a502fe

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1385

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 199 more files in changeset.
ASTERIXDB-1711: rename asterix_transaction_log

- s/asterix_transaction_log/transaction_log/

- update storage version

- remove some dead code

Change-Id: Ieed464a1993a6bbff798e18e4c94245755a447eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1368

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Some Code Reduction/Cleanup

Change-Id: I179a16fc183bd50d5d58cc12321234df1615abfd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/916

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Extract common code of [Optimizer|Smoke]ParserTest

Change-Id: Iab916e71539ffac7b952ab6795614c00039ca7ff

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1379

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

ASTERIXDB-1749: fix breaking of lines using '\r'

Change-Id: Ica5ce0b82f1d2c6f2033be2ce20bf56a563fb57b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1382

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

reduce code duplication a tiny bit

Change-Id: I3f3bf5fb423cd55bd7fc168ac7a9014a24bc516c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1365

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Full-text implementation step 1

- Introduced ftcontains() that conducts the full-text search

- This version doesn't support index-based full-text search yet.

After this step, the functionality will be added as a separate patch.

- Added BinaryHashSet derived from BinaryHashMap class.

- Parameter checking during the compilation is not merged yet.

Change-Id: If00a871a8241d6aa6931f97b694d65f164d3ab8c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1228

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 59 more files in changeset.
Removed some new byte[] creation and implemented explicit resource release

- Removed some new byte[] creation (mainly for toByteArray() method)

- Implemented Explicit resource release during a hash join

- Refactorered Hash-join code to remove repetitive same condition check

Change-Id: I55195696a3db09c14b8debdd78f5f68d701b9129

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1378

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Cleanup type exceptions and error messages in the compiler.

Change-Id: I2bdb21251e53ac552e85ba30b8105f534bf9f4f3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1371

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 37 more files in changeset.
Fix for ASTERIXDB-1738: Change feed fails to delete record with meta PK

Change-Id: I5ce8d9c69f96593ee305cfad4b44e486c9f1d6cc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1351

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>