asterixdb

Clone Tools
  • last updated 15 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Merge commit 'eaa38d8c2c' into 'master'

Change-Id: Ic03c45b9a79784b357ea66502956778bf105b9b3

[NO ISSUE][FUN] Clean up of warn utility method

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

clean up of warnTypeMismatch methods to avoid confusion

betweent the two methods.

Change-Id: I7ae7c85fa3cc549cfd622f179f36486e016d9643

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3581

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][TXN] Remove Synchronization on TransactionContext

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently the flag indicating if a transaction timed out

synchronizes on the transaction context which makes it

a hot spot when multiple threads on the same transactions

attempting to access that flag. This change removes that

synchronization and replaces it with a volatile variable.

Change-Id: Ib14c07a051d27a5b47c62ff35769670e6ab01fd2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3579

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Merge commit '819c57b440'

Change-Id: I82594526b6f921af058982f450efbdfdad78b106

Merge commit 'e8e59d0e6e' from 'stabilization-f69489'

Change-Id: I73ad57791bb7264a670bb36f4eb6f50640b4a86f

[NO ISSUE] Update Guava, Jackson dependencies

Change-Id: I1a415367b0e403237deab782dd99214b927138db

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3576

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[ASTERIXDB-2541][STO] Introduce GreedyScheduler

- user model changes: yes.

Add new option: storage.io.scheduler (async/greedy)

- storage format changes: no.

- interface changes: yes.

Introduce IIndexCursorStats

Details:

- Introduce GreedyScheduler that always executes the merge

operation with the smallest number of remaining pages to minimize

the number of disk components

- Introduce IIndexCursorStats to collect the statistics of index scans.

This allows GreedyScheduler to know the remaning pages of merge

operations.

- Extend AbstractIoOperation so that GreedyScheduler can pause/resume

merge operations if needed.

Change-Id: I38fe394d1180d4e3f6796064c0e6c6630b6ad303

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3284

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 59 more files in changeset.
[ASTERIXDB-2647][COMP] Random partitioner shouldn't preserve input's properties

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Random partitioner operator currently always propagates

the local properties of its input. If the data of the

input operator is partitioned, the random partitioner

should not propagate the local properties of the input

since the random partitioner destroys them.

Change-Id: I0378597451b3e6b25e8b45295159efec262abae2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3575

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2635][*DB] Fix JSON plan pretty printing

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Follow up change to fix printing "expressions" field

Change-Id: Ia994c0e0001cee2eca31f0bdf51ddb52320ae752

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3574

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[NO ISSUE] Fix comment of concurrent merge policy test

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fix the comment of the test case using-concurrent-merge-policy.

Change-Id: I363f5739ac4bb2a9dd14df7be97372eb1e838dd6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3572

Reviewed-by: Luo Chen <cluo8@uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[NO ISSUE][TEST] Increase Test Buffer Cache Memory Budget

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Increase test buffer cache memory budget to 128MB

to avoid test failures due to small buffer cache.

Change-Id: I76d29bf7d81de189e2157d902382a60e6c9772e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3571

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][COMP] Fix type computers of some array functions

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Fix the type computer of some array functions to return a nullable

type if the function might return NULL.

Change-Id: I21b747f3116e78f4dcbec079a38972c1f1abc2ce

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3570

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Contrib: Till Westmann <tillw@apache.org>

[ASTERIXDB-2635][*DB] Fix JSON plan pretty printing

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Use Jackson to pretty print the logical plan as JSON.

- fixed JsonLogicalPlanTest to validate JSON plans

produced by optimizer tests.

Change-Id: Ic5ada2f31afebc3f219b0584b5ae527ddf9e326e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3567

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 7 more files in changeset.
[NO ISSUE][*DB] pass max-warnings via session config

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Pass max-warnings from servlet to APIFramework

and QueryTranslator through the session config instead

of creating an internal parameter (REQUEST_MAX_WARNINGS)

and storing it in the metadata provider.

Change-Id: I4145bcccc2702fae4b0c33872f1b98c47ab299f7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3569

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[NO ISSUE][TEST] Make test framework generic with respect to endpoint

Refactoring of test framework to configure query parameters, charset, and endpoint

Change-Id: Iaeb217a0cc49e839d7a0c6cbae90d7eb964c3322

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3515

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Till Westmann <tillw@apache.org>

[ASTERIXDB-2600][STO] Introduce ConcurrentMergePolicy

- user model changes: yes. Add a new merge policy and make it as default

- storage format changes: no.

- interface changes: no.

Details:

- Introduce ConcurrentMergePolicy that performs concurrent merges

without the maximum component size.

- Make this merge policy as the default merge policy in AsterixDB since

the PrefixMergePolicy has made some wrong design decisions.

Change-Id: I2ed79847584b9fe846d62ad56ee094863538a2a2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3463

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 18 more files in changeset.
[ASTERIXDB-2540][STO] Optimize Storage Disk I/O

- user model changes: yes. Add a new storage option:

storage.disk.force.bytes (default 16MB),

- storage format changes: no.

- interface changes: yes.

Introduced IPageWriteCallback to LSM indexes

Details:

- Bypass all queuing (from BufferCache and IOManager) for disk writes.

This queuing is unnecessary but destroys fairness among multiple

writers.

- Introduce IPageWriteCallback to control the behavior of disk page

writes. Currently, this interface is used to perform disk forces

regularly for each writer thread.

Change-Id: I1f618dc7c186623e860239b4d97640fe3528e75b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3285

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 128 more files in changeset.
[ASTERIXDB-2631][COMP][RT] Report count of all warnings

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Currently, only runtime warnings are counted. Include

parser & compile-time warnings, as well.

- removed the warn limit as a compiler option and made it

a request parameter.

- the warning collector of the parser collects all warnings

but allow pulling warnings up to a maximum desired.

- the warning collector of the compiler/executor collects up

to the request max-warnings and allow pulling up to a max desired.

- servlets collect from the parser and query compiler &

executor up to the request max-warnings.

Change-Id: Ia7e559313d32a8ba6f450f254a3a7b4b1e662a50

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3564

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 76 more files in changeset.
[ASTERIXDB-2634][COMP][RT] String functions to return NULL on mismatch

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Make string functions return NULL on mismatch instead of raising

an exception and issue a warning.

- consolitated ExceptionUtil from org.apache.asterix.runtime.exceptions

into org.apache.asterix.om.exceptions.

- AbstractStringTypeComputer does not check arguments type now

and does not throw mismatch exception.

Change-Id: I51db8ae6ba301757c45c41cfd02bd2b083436970

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3562

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 30 more files in changeset.
[ASTERIXDB-2532][RT] per-operator profiling

Enables profiling in queries at the operator-level when the analyze

variable is set in a query.

Change-Id: Ie16f3901ae5b32920d8552d5fd1ec8bb6e2ec8ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3226

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 46 more files in changeset.
Merge commit '82b067c' from stabilization-f69489

Change-Id: I4891035d517387148e86027a733afa4a21a5c75a

[ASTERIXDB-2633][FUN] Fix LIKE to regex translation

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: I6138f62fdcd60bcc641f662d1fdb05b46dafbebf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3558

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Merge commit '8930f0d' from stabilization-f69489

Change-Id: I64cf862bc5ebe19a24acbccf821f68556699222e

  1. … 9 more files in changeset.
[NO ISSUE][ING][EXT] Handle filters on meta() fields for feeds

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This fix is to handle filters on meta() fields where the feed name

is not supplied as an arguement to the meta() when connecting the feed.

Both of these statement should work:

CONNECT FEED ds_feed TO DATASET ds WHERE meta(ds_feed).id LIKE "%WEB%"

CONNECT FEED ds_feed TO DATASET ds WHERE meta().id LIKE "%WEB%"

Change-Id: I1dbf40525601584efa29eb3adf4ba41062ebe53a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3557

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +14
    /asterixdb/asterix-app/data/csv/people2.csv
  1. … 20 more files in changeset.
[ASTERIXDB-2631][COMP][RT] Limit number of reported warnings

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Limit number of reported warnings.

- added a new compiler property "COMPILER.RUNTIME.WARNINGS"

with default value set to 0.

- updated Stat class to have a field for the count of runtime

warnings.

- updated few classes to propagate the warnings limit number

to Hyracks Task class.

- added new Option to parse unsigned long values.

- TaskProfile has a field for the count.

- updated TestExecutor to allow specifying parameters in

the body of the POST method.

Change-Id: Ie8756f1b9dcb7bbce92bd77692490fbada9a2482

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3535

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 43 more files in changeset.
[NO ISSUE][COMP] Remove listify() when iterating over aggregate

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Optimizer must remove listify() when iterating

over an aggregate function that returns a list

Change-Id: I7a8919595be7e4a44a5816ac7f281842d03ecc1f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3556

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

[NO ISSUE][RT] Add Thread-Based Stats Collector

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add infra to allow collecting thread-based stats during

runtime for any thread that belongs to a task.

- Collect number of pinned pages per thread and report it

in the TaskProfile.

- Aggregate pinned pages counters from all job tasks and

report it as diskIoCount in the metrics field in the json

response. The plan is to move this stats to the profile

field when it is introduced.

- Collecting pinned pages stats is currently enabled by

default for any job with IndexSearchOperatorNodePushable.

The plan is to allow enabling/disabling as part of the

job profiling change.

- Add test case for diskIoCount metric.

- Remove unused IndexSearchOperatorNodePushable constructor.

Change-Id: I44dfcedcadb3d0f48815b521e7d495e473b02e3d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3555

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 13 more files in changeset.
[NO ISSUE][COMP] Increase memory budget for window operator

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Increase default memory budget for a window operator

from 4MB to 32MB

Change-Id: I6902d5e730232892f22e36598389309e8bc909c3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3551

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

[NO ISSUE][COMP] Reduce memory requirements for monitoring functions

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Queries that only invoke monitoring/metadata datasource functions

should run with minimal memory requirements

- The functions are: active_requests(), completed_requests(),

dataset_resources(), storage_components(), jobs(), and ping()

Change-Id: If7de53a5b476c8ce0d3fe485bf526a5d425e4c37

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3548

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 13 more files in changeset.
[NO ISSUE][COMP] Fix index selection for datasets with meta

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Fix index selection for datasets with meta. Access method rule should

check where the field is coming from (dataset record or meta record)

and then determine if the field matches the keys in the index based

on their names and sources.

This patch also fixes resolving PK field accesses to the primary key

variable (e.g. $ds.getField("id") is turned into $13 where id is a PK).

The fix considers whether the PK is coming from the data record or the

meta record.

The patch also includes fixing rewriting of meta() references and

replacing them with their corresponding meta variables. Now nested plans

are visited also when looking for meta() references to take care of cases

where the data scan producing the meta variable and the meta() references

happen to be inside the nested plans.

MetaFunctionToMetaVariable() is fired also after the rules which eliminate

subplans to allow for rewriting of the meta() if it couldn't be replaced

when the meta() reference was in the subplan but referring to meta variable

outside the nested plan.

ReinferAllTypesRule() is now fired before ByNameToByIndexFieldAccessRule()

to allow the latter rule to get the up-to-date types in the whole plan.

Change-Id: I0503f64cd51153896e2d7d7abc465c679f82e2fd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3545

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Contrib: Till Westmann <tillw@apache.org>

    • -0
    • +9
    /asterixdb/asterix-app/data/csv/people.csv
    • -0
    • +9
    /asterixdb/asterix-app/data/csv/people3.csv
  1. … 78 more files in changeset.