asterixdb

Clone Tools
  • last updated 14 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Merge branch 'gerrit/mad-hatter' into 'master'

Change-Id: I892dfdc0f2f61cb20e17995d797598b420e8c554

[NO ISSUE][STO] Persist Bloom Filter Existence in Index Metadata

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- To clearly identify whether an index has a bloom filter or

not for BTree indexes, persist this information in the index's

metadata stored on each NC.

- For backward compatibility, when reading an index's metadata that

was created before adding the hasBloomFilter field, default its

value based on whether or not the index is a primary key index.

- Remove unused special readObject from LSMBTreeLocalResource.

Change-Id: Icec570d490987de401c036790ee9567238a60301

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3804

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 18 more files in changeset.
[NO ISSUE][RT] Report Disk IO Per Operator

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not collect per thread stats when runtime

profiling is disabled.

- Include per operator disk IO in task profile.

- Remove aggregated disk IO count from the query

response metrics.

- Adapt test cases.

Change-Id: Iced174ae5d4fa9ca13f597325cbb66ff80819e71

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3783

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

  1. … 4 more files in changeset.
Merge branch 'gerrit/mad-hatter' into 'master'

Change-Id: I7e921b559f30adfab704143818dd7ee59e194eb4

    • -16
    • +7
    ./asterix-lang-aql/src/main/javacc/AQL.jj
    • -6
    • +7
    ./asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
[ASTERIXDB-2648][COMP] Fix dataset order in broadcast hint

- user model changes: yes

- storage format changes: no

- interface changes: no

details:

- Provides hash-bcast hint to replace bcast hint which makes

broadcast join consistent with other types of joins for choosing

the broadcasting dataset based on dataset order in FROM clause.

- Chooses the build dataset to get broadcast based on the order

in the FROM clause not the WHERE clause (which was the case for

bcast hint).

Change-Id: I505fc8e0bc32c760ae0471c6b302bbfd8499be4a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3566

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 37 more files in changeset.
[ASTERIXDB-2655][STO] Default storage compression to enabled

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Use 'snappy' as default 'storage.compression.block' value

- Update tests to reflect the new default

Change-Id: I62428c01ec24bc72bdf3cdf903655b3d0d7acbdd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3683

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Michael Blow <mblow@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

[ASTERIXDB-2651][RT][FUN] Handle matching a literal backslash properly in LIKE

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- fixed an issue in the build() of RegExpMatcher where the "flags" argument gets ignored if the flags

argument does not change in the subsequent evaluation of the regexp_* functions over the tuples of dataset.

- added the missing Java regex meta characters.

- split test case "regexp_replace" into two, "regexp_replace" & "regexp_replace_with_flag".

Change-Id: I4f218f87f0c1736e2d8b5de07221d7efa0f9aef1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3614

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 86 more files in changeset.
[NO ISSUE] Fix type inference for ANY in UDFs

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

When data type is available, we should use defined data type. If not, we

should try to infer the data type from the data.

Change-Id: I6a03749fe041ec6f43de230d3e784122f7664545

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3405

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

[NO ISSUE][NET] Log ChannelReadInterface Stats on Failure

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- When the read buffers of a FullFrameChannelReadInterface

are exceeded, log its stats to help in debugging the issue.

- Warn when a partial frame is being written over a full

frame write channel.

- Rename ThreadSafetyGuaranteedBy -> GuardedBy.

- Annotate networking calls with their synchronization

guards.

Change-Id: I89eed0c06dbf4b0e86747538bff286dc37853957

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3663

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Al Hubail <mhubail@uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 12 more files in changeset.
[NO ISSUE][API] Add ability to identify secure servlet requests

Change-Id: Ia693e29a47b513a63fdce80383da90ba165c28d6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3603

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 6 more files in changeset.
[NO ISSUE][COMP] Cleanup Identifier constructors

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Remove default constructors from

Identifier, VarIdentifier, VariableExpr

- Make Identifier immutable

Change-Id: I3a0b4da0e2b621d309b8d9aa3c47540eb18566eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3607

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -18
    • +15
    ./asterix-lang-aql/src/main/javacc/AQL.jj
    • -14
    • +12
    ./asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
[ASTERIXDB-2634][COMP] String functions return null on data/type errors

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Functions changed:

substring(string, start_idx),

repeat(string, num_times),

string_join([string], string_separator),

replace(string, search_string, string_replacement, num_times),

regexp_replace(string, string_pattern, string_replacement, num_times)

Those functions (except string_join) can return NULL even if the

arguments types are valid since the argument values could be not

valid at runtime. Their type computer is always nullable.

- int argument can be double/float on the condition that its value

is integer

- clean-ups:

UnaryStringInt64TypeComputer & AbstractStringTypeComputer.

ExceptionUtil:

- changed signature of toExpectedTypeString(). Overloading

with varargs with Object made it confusing.

- changed some args to Supplier to allow code sharing.

- removed StringIntToStringTypeComputer since now the instances

are not being used by the above functions.

- reorganized/renamed test cases

Change-Id: Ia85a0d08888021ae439a1d9f2f5858bcd52c79f3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3605

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 49 more files in changeset.
[NO ISSUE][DOC] Improve window functions documentation

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Minor improvements in the documentation

for window functions and OVER clause

Change-Id: I349d9b93edca3f3a7ffb340b016402e9ae8320fc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3606

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -20
    • +19
    ./asterix-doc/src/main/markdown/sqlpp/3_query.md
[ASTERIXDB-2649][FUN] TPC-DS datasource function, generate proper data types

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Changed the TPC-DS datasource function to generate

the TPC-DS data with proper data types according to

the TPC-DS schema.

- Updated the TPC-DS test cases.

Change-Id: I20f6b8d043906ad62652d098e09ab70eb1d78b1b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3604

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

[NO ISSUE][FUN] Clean up of warn utility method

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

clean up of warnTypeMismatch methods to avoid confusion

betweent the two methods.

Change-Id: I7ae7c85fa3cc549cfd622f179f36486e016d9643

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3581

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][TXN] Remove Synchronization on TransactionContext

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently the flag indicating if a transaction timed out

synchronizes on the transaction context which makes it

a hot spot when multiple threads on the same transactions

attempting to access that flag. This change removes that

synchronization and replaces it with a volatile variable.

Change-Id: Ib14c07a051d27a5b47c62ff35769670e6ab01fd2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3579

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Merge commit 'e8e59d0e6e' from 'stabilization-f69489'

Change-Id: I73ad57791bb7264a670bb36f4eb6f50640b4a86f

  1. … 1 more file in changeset.
[NO ISSUE] Update Guava, Jackson dependencies

Change-Id: I1a415367b0e403237deab782dd99214b927138db

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3576

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 1 more file in changeset.
[ASTERIXDB-2541][STO] Introduce GreedyScheduler

- user model changes: yes.

Add new option: storage.io.scheduler (async/greedy)

- storage format changes: no.

- interface changes: yes.

Introduce IIndexCursorStats

Details:

- Introduce GreedyScheduler that always executes the merge

operation with the smallest number of remaining pages to minimize

the number of disk components

- Introduce IIndexCursorStats to collect the statistics of index scans.

This allows GreedyScheduler to know the remaning pages of merge

operations.

- Extend AbstractIoOperation so that GreedyScheduler can pause/resume

merge operations if needed.

Change-Id: I38fe394d1180d4e3f6796064c0e6c6630b6ad303

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3284

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

    • -0
    • +1
    ./asterix-app/src/main/resources/cc.conf
  1. … 71 more files in changeset.
[NO ISSUE] Fix comment of concurrent merge policy test

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fix the comment of the test case using-concurrent-merge-policy.

Change-Id: I363f5739ac4bb2a9dd14df7be97372eb1e838dd6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3572

Reviewed-by: Luo Chen <cluo8@uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[NO ISSUE][TEST] Increase Test Buffer Cache Memory Budget

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Increase test buffer cache memory budget to 128MB

to avoid test failures due to small buffer cache.

Change-Id: I76d29bf7d81de189e2157d902382a60e6c9772e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3571

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -1
    • +1
    ./asterix-app/src/main/resources/cc.conf
    • -1
    • +1
    ./asterix-app/src/main/resources/cc2.conf
    • -1
    • +1
    ./asterix-app/src/main/resources/cc3.conf
    • -1
    • +1
    ./asterix-app/src/test/resources/cc-single.conf
    • -1
    • +1
    ./asterix-app/src/test/resources/cc-ssl.conf
    • -1
    • +1
    ./asterix-app/src/test/resources/cc.conf
[NO ISSUE][COMP] Fix type computers of some array functions

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Fix the type computer of some array functions to return a nullable

type if the function might return NULL.

Change-Id: I21b747f3116e78f4dcbec079a38972c1f1abc2ce

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3570

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Contrib: Till Westmann <tillw@apache.org>

[ASTERIXDB-2635][*DB] Fix JSON plan pretty printing

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Use Jackson to pretty print the logical plan as JSON.

- fixed JsonLogicalPlanTest to validate JSON plans

produced by optimizer tests.

Change-Id: Ic5ada2f31afebc3f219b0584b5ae527ddf9e326e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3567

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 18 more files in changeset.
[NO ISSUE][*DB] pass max-warnings via session config

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Pass max-warnings from servlet to APIFramework

and QueryTranslator through the session config instead

of creating an internal parameter (REQUEST_MAX_WARNINGS)

and storing it in the metadata provider.

Change-Id: I4145bcccc2702fae4b0c33872f1b98c47ab299f7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3569

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[NO ISSUE][TEST] Make test framework generic with respect to endpoint

Refactoring of test framework to configure query parameters, charset, and endpoint

Change-Id: Iaeb217a0cc49e839d7a0c6cbae90d7eb964c3322

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3515

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Till Westmann <tillw@apache.org>

[ASTERIXDB-2600][STO] Introduce ConcurrentMergePolicy

- user model changes: yes. Add a new merge policy and make it as default

- storage format changes: no.

- interface changes: no.

Details:

- Introduce ConcurrentMergePolicy that performs concurrent merges

without the maximum component size.

- Make this merge policy as the default merge policy in AsterixDB since

the PrefixMergePolicy has made some wrong design decisions.

Change-Id: I2ed79847584b9fe846d62ad56ee094863538a2a2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3463

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 18 more files in changeset.
[ASTERIXDB-2540][STO] Optimize Storage Disk I/O

- user model changes: yes. Add a new storage option:

storage.disk.force.bytes (default 16MB),

- storage format changes: no.

- interface changes: yes.

Introduced IPageWriteCallback to LSM indexes

Details:

- Bypass all queuing (from BufferCache and IOManager) for disk writes.

This queuing is unnecessary but destroys fairness among multiple

writers.

- Introduce IPageWriteCallback to control the behavior of disk page

writes. Currently, this interface is used to perform disk forces

regularly for each writer thread.

Change-Id: I1f618dc7c186623e860239b4d97640fe3528e75b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3285

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 134 more files in changeset.
[ASTERIXDB-2631][COMP][RT] Report count of all warnings

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Currently, only runtime warnings are counted. Include

parser & compile-time warnings, as well.

- removed the warn limit as a compiler option and made it

a request parameter.

- the warning collector of the parser collects all warnings

but allow pulling warnings up to a maximum desired.

- the warning collector of the compiler/executor collects up

to the request max-warnings and allow pulling up to a max desired.

- servlets collect from the parser and query compiler &

executor up to the request max-warnings.

Change-Id: Ia7e559313d32a8ba6f450f254a3a7b4b1e662a50

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3564

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 76 more files in changeset.
[ASTERIXDB-2634][COMP][RT] String functions to return NULL on mismatch

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Make string functions return NULL on mismatch instead of raising

an exception and issue a warning.

- consolitated ExceptionUtil from org.apache.asterix.runtime.exceptions

into org.apache.asterix.om.exceptions.

- AbstractStringTypeComputer does not check arguments type now

and does not throw mismatch exception.

Change-Id: I51db8ae6ba301757c45c41cfd02bd2b083436970

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3562

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 30 more files in changeset.
[ASTERIXDB-2532][RT] per-operator profiling

Enables profiling in queries at the operator-level when the analyze

variable is set in a query.

Change-Id: Ie16f3901ae5b32920d8552d5fd1ec8bb6e2ec8ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3226

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 46 more files in changeset.