hyracks-fullstack

Clone Tools
  • last updated 24 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][COMP][RT] New library deployment model

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Deploy external libraries using Hyracks jobs

- Library manager is no longer available on CC

- Add IServlet.init() which is invoked during web server startup

- External libraries can no longer provide implementations of

IExternalDataSourceFactory because external libraries are

not available on CC

- Added testcase for an external adapter

Change-Id: If64f99f6a15b81b1e426239bde63360f5ef57059

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6863

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 80 more files in changeset.
[NO ISSUE][COMP] Limit operator is not a map

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- LimitOperator.isMap() should return false

because Limit operator is not a map

Change-Id: Ib17a2268ac0576956a8c1bbc6b5be1cb7bacc2ab

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6984

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

[NO ISSUE][HYR][*DB][MISC] Fix whispered logging, reduce ActiveManager info logging

Change-Id: I95fa2a8a87237500f4cdad0e0146a0ada2a08056

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6945

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1 more file in changeset.
[ASTERIXDB-2751][EXT] Issue a warning if external dataset configuration returns no files

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Issue a warning when creating an external dataset if the dataset

configuration returns no files.

- Issue a warning when querying an external dataset if no files

are returned.

Change-Id: I0b2be49bf3c69c3e055d942a5c3f0106b3069d52

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6843

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 49 more files in changeset.
[NO ISSUE][MISC] IRetryPolicy requires non-null throwable

Change-Id: I4835266435fc31f5973f4312e6ddcbdb85d1e839

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6923

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 1 more file in changeset.
[NO ISSUE][HYR] +=ThrowingIntConsumer, unchecked wrapper helpers

Change-Id: I43915b87f5a10dd024f5618c9e8da26c5ad5c496

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6903

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Merge from branch 'gerrit/stabilization-f69489'

Change-Id: I388f3af0307d5482da4cfd559a7373fa4887c425

  1. … 2 more files in changeset.
[NO ISSUE] update log4j -> 2.13.3

Change-Id: Ic447ac0a092af188e4a420d68d726367efd19514

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6783

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

  1. … 2 more files in changeset.
[ASTERIXDB-2730][STO] Optimize flush in GVBC

- user model changes: no

- storage format changes: no

- interface changes: yes. Introduce a new storage property:

storage.memorycomponent.max.concurrent.flushes (default 0)

Details:

- Introduce a new storage property to allow concurrent flushes

by GVBC. The default value is 0, which means that the flush concurency

will be the same as the number of NC partitions.

- Move cleaning up of a memory component out of the synchronization block

on op tracker because this may take a relatively long time (a full scan over

all GVBC pages).

- Introduce a minor fix to make sure the memory component is unwritable

before requesting flushing it by GVBC

Change-Id: Id8867fa3ac65da319723b804cc1e39dc8eb6bde5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6624

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 3 more files in changeset.
[NO ISSUE][MISC] HyracksException.getParams() type, += Throwing helpers

Change-Id: I0aa740452d40719da33bcb25ecec6c09e45c9dd1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6703

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 1 more file in changeset.
Merge branch 'gerrit/mad-hatter'

Change-Id: Icb703199fd6eb3e41e19e288192f2b72215b4b06

[NO ISSUE] HTTP, halt improvements

- don't propagate close to httpclient response stream, as it will block indefinitely

- only wait for 60s to collect thread dump at halt; ensure we always halt once asked to

Change-Id: I46de779e2f3ab0ff670ebec2966f364abcef0ecd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6623

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[NO ISSUE] Improve handling of udf parameter/return types

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Prohibit parameter/return type specification for inline udfs

- Prohibit parameter/return type quantifier (?) for external udfs

- Allow external udfs to refer to types from other dataverses

- Do not store type quantifiers in udf metadata

- Align names of anonymous types automatically created by

QueryTranslator for datasets and udfs

- Reserve user-defined type names starting from '$' for system use

- Add testcases

- Prohibit UdfServlet operations if cluster is not active

- Return correct path from IOManager.getWorkspacePath()

- Test framework: support multiple commands in .lib.sqlpp files

- Remove CC files when deleting old instance data in

AsterixHyracksIntegrationUtil

Change-Id: I09bdfd8c7b26750d12339034d2143a16102d1212

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6545

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 35 more files in changeset.
Merge commit 'a548307b9f' from 'gerrit/mad-hatter'

Change-Id: I57eb5d96af8f227e5a9550381418f9468eb400d8

  1. … 8 more files in changeset.
[NO ISSUE][HYR][NET] NetworkUtil enhhancements

Change-Id: I9c0d24100b653015233fc5a08554983d0a0185e1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6544

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

[NO ISSUE][*HYR] Span enhancements

- add reset() to start the span over again

- add sleep() to sleep for the remainder of the span

Change-Id: I93799354f2e238b759d2ef11aed109ec96baaf21

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6483

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

[ASTERIXDB-2738][STO] Create Mask File Before Merge Operations

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Before starting a merge operation, create a mask file

(.mask_C_startSeq_endSeq) for the merged component to

indicate that this component isn't valid yet.

- On the merge operation successful completion, delete the

merged component mask file.

- In the case of any unexpected failure during the merge

operation, all files of the failed merged component will

be deleted on node startup/shutdown, including the mask

file.

- Halt on any IO opeartion failure.

- Add a test case that ensures only masked merged components

are deleted but not the original components that were

supposed to be merged.

Change-Id: I476dd3be5e75468e83044b3aaf0f6c2d8beadf1c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6425

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Luo Chen <cluo8@uci.edu>

  1. … 6 more files in changeset.
[NO ISSUE][EXT] Minor refactoring of external datasets details validation

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Minor refactoring of external datasets details validation.

- Added a missing error message in Hyracks.

Change-Id: I975228925366adc94f136bbbb014cb6817bf180e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6325

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 3 more files in changeset.
Merge commit 'aa7ff7b' from 'gerrit/mad-hatter'

Change-Id: Ibfc874ca45692420fb86038537d10e172894730a

  1. … 18 more files in changeset.
Merge commit 'f1cd417' from 'gerrit/mad-hatter'

Change-Id: I0f30399a6485708f4b154af09ea36377fe4d3dbf

  1. … 9 more files in changeset.
[ASTERIXDB-2729][LIC] Fix license warnings in newly added AWS libraries & dependencies

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added all missing licenses.

- Silenced all warnings being generated by the licenses.

- Updated the license check to accept <gavs> property for

convenience of use.

Change-Id: I0ab576d748799a59d23990e2379bdf6aed74a7e5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6284

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 4 more files in changeset.
[ASTERIXDB-2726][EXT] Report line number instead of record number in messages of parsers

- user model changes: no

- storage format changes: no

- interface changes: yes

IRecordReader:

added getLineNumber() to provide line number for parsers and converters.

IRecordConverter:

added configure() to pass the line number supplier to the record converter.

IRecordDataParser:

pass line number supplier from the Reader to the Parser.

Details:

Report line number instead of record number in messages of parsers.

- added getPreviousStreamName() to allow readers to report errors happening on

the previous stream when the underlying stream has already switched to a new one.

- changed the test executor to compare actual warnings issues by a test case with

the expected warnigns properly.

Change-Id: I00508d8eeca4d9bae95f55ab51ecfb0ce2ced6b0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6245

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

  1. … 39 more files in changeset.
[NO ISSUE][HYR][HTTP] Utility method to enable interruptable http requests

Change-Id: Id880af7e1e60195c53a043752d9339818c002a2c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6164

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

[ASTERIXDB-2176] Python UDFs

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Allow UDFs to be shiv-packaged python modules with their

dependencies

- Use pyro for python RPC

- Maven build for pyro package into server

- Remove JObject spatial types

- Simpler conversion from java primitive/standard types and

collections

to JObject equivalents

Change-Id: Ibea23a2e9308132f343d80eff04ede9a235aa021

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5526

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 105 more files in changeset.
[ASTERIXDB-2715][STO] Dynamic Memory Component Architecture

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Introduce a dynamic memory component architecture that uses a global

virtual buffer cache to manage the write memory for all LSM-trees.

- When the overall write memory is nearly full, we flush a dataset

partition at a time using a round-robin way. Additionally, we allow

users to configure the maximum size of filtered memory components

to provide better pruning capability.

- Clean up legacy code for statically allocating write memory to each

dataset.

- Remove the following parameters:

storage.metadata.memorycomponent.numpages

storage.max.active.writable.datasets

- Add the following parameters:

storage.memorycomponent.flush.threshold (default: 0.9)

storage.filtered.memorycomponent.max.size (default 0)

Change-Id: Ia6a0f4de020acd7af89ef630322526c4be5076e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5824

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 30 more files in changeset.
[NO ISSUE] Fix duplicate dependency build warnings

Change-Id: I00bb170557d3beff786dda8cfb27ed980ec94d53

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5986

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[NO ISSUE] Ensure 'pom' modules are test-free

Change-Id: I2c938f163a6327d16062390ee9f661cfa81df3dd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5985

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 1 more file in changeset.
[ASTERIXDB-2713][EXT] CSV & TSV support for external dataset p4

- user model changes: no

- storage format changes: no

- interface changes: yes

- IValueParser

Details:

- added parameter "null" to allow user to specifiy what string represents

a null value. Only nullable fields whose values match this string will

produce NULL. Otherwise, no NULLs will be produced.

- empty fields:

- for string fields, produce empty string "".

- for non-string fields, issue a warning an ignore the record.

- changed IValueParser to return boolean to allow some implementations

to return true or false instread of throwing an exception.

- added parameter "redact-warnings" to allow user to specifiy if parser

warnings should exclude information like file name.

- changed the Integer and Long parsers to consider spaces properly and

also to handle overflow and underflow.

- changed the boolean parser to align with other parsers that ignore

leading and trailing spaces.

- added test cases.

Change-Id: Ib6aed0095a472510b69cc29a3e444e7de5a2c1ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5963

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 60 more files in changeset.
[NO ISSUE][HTTP] Http Encoding refactoring / cleanup

Change-Id: I1e6eef71f4f17a7563f182400dd1e815e4a6b8ff

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5943

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Contrib: Michael Blow <mblow@apache.org>

  1. … 2 more files in changeset.
[NO ISSUE][COMP] Fix AbstractIntroduceGroupByCombinerRule

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- AbstractIntroduceGroupByCombinerRule did not always

restore the plan to its original state when failing

Change-Id: I9cada7f324a271ff017c64b69326b9d4a1a32a85

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5925

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 5 more files in changeset.