Clone
 

hussain towaileb <hussainht@gmail.com> in asterixdb

Merge commit 'c25c36d1b0' from 'gerrit/mad-hatter'

Change-Id: I8a78950cb81f0a29ceae94fa439eec700c17d06d

[NO ISSUE] Pass item type when creating external dataset properties

Change-Id: I8b6693d229a36358eda584ddfb95d1819f76ce5c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6963

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

[ASTERIXDB-2751][EXT] Issue a warning if external dataset configuration returns no files

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Issue a warning when creating an external dataset if the dataset

configuration returns no files.

- Issue a warning when querying an external dataset if no files

are returned.

Change-Id: I0b2be49bf3c69c3e055d942a5c3f0106b3069d52

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6843

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 39 more files in changeset.
Merge commit '0163af68d0' from 'gerrit/mad-hatter'

Change-Id: I5b53384961e42fe070d2e1b885e762428f6f8c45

[ASTERIXDB-2743][EXT] Skip not found files when querying S3 external dataset

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- When querying an S3 external dataset, if some files are not found

due to possiblity getting deleted while querying, skip the filies

and continue to the next ones instead of failing.

Change-Id: I6d3e691a1714228f1844faf05095cf8aa439ea1d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6723

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

[ASTERIXDB-2741][EXT] Support include/exclude filters for S3 external datasets

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added support for include/exclude filters.

- Added test cases for include/exclude.

Change-Id: I21dbc5c08456f95a2cad1a8fd9147472fad15376

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6604

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 75 more files in changeset.
Merge commit 'a548307b9f' from 'gerrit/mad-hatter'

Change-Id: I57eb5d96af8f227e5a9550381418f9468eb400d8

    • -0
    • +8
    /asterixdb/asterix-external-data/pom.xml
[ASTERIXDB-2722][EXT] Require minimum permissions + improve error reporting

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Creating an external dataset requires S3:ListBucket permission.

- Querying an external dataset requires S3:ListBucket and

S3:GetObject permissions.

- Improved error reporting.

- Added test cases for the above mentioned items (where applicable).

Change-Id: Idc266cf63b8f92a07af7341118d2636673913160

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6463

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +8
    /asterixdb/asterix-external-data/pom.xml
  1. … 4 more files in changeset.
Merge commit '0f407d9224' from 'gerrit/mad-hatter'

Change-Id: I89eaf1bace87130d8d35ec0ce8e0c263ac87b59a

[ASTERIXDB-2725][COMP] Check if dataverse exists before continuing the DDL execution

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added dataverse check at multiple places before proceeding

with the execution of the DDL.

- Added a test cases for each newly added check.

Change-Id: I8d54b0b10741d65df91cbb4b7596d363f7ddbd97

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6203

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2736][EXT] Ensure retrieving all objects if an S3 bucket has more than 1000 objects

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fixed the AWS reader to handle reading more than 1000

objects coming in a single request.

- Added a test case for the above mentioned item.

Change-Id: Ic7891aa86852e07dfad9ce41de908b34f86bdb42

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6344

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

[ASTERIXDB-2729][LIC] Fix license warnings in newly added AWS libraries & dependencies

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added all missing licenses.

- Silenced all warnings being generated by the licenses.

- Updated the license check to accept <gavs> property for

convenience of use.

Change-Id: I0ab576d748799a59d23990e2379bdf6aed74a7e5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6284

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -0
    • +16
    /asterixdb/src/main/licenses/content/raw.githubusercontent.com_aws_aws-sdk-java-v2_2.10.83_NOTICE.txt
    • -0
    • +2
    /asterixdb/src/main/licenses/content/raw.githubusercontent.com_awslabs_aws-eventstream-java_7be2dd80e12f8835674c8ffb0f4a2efb64c7b585_NOTICE.txt
[ASTERIXDB-2724][EXT] Handle passing empty defintion to external datasets

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- When an empty string is passed as an external dataset's definition, then

no prefix is supplied to the AWS client (nothing to filter).

- Added a test case for the above mentioned item.

Change-Id: I500e7afb97aa076b690ef3b98ee83c8f5934a88f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6183

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Michael Blow <mblow@apache.org>

[ASTERIXDB-2719][EXT] Add .gz compressed test cases for CSV and TSV

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added test cases for compressed data for CSV and TSV.

- Addressed a comment to ensure closing the S3 stream in

case of a failure during streaming the compressed data.

Change-Id: I6160ee5f5aefa4ca60e5a88c3922bde40f4c65ec

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6123

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 20 more files in changeset.
[ASTERIXDB-2719][EXT] Default external datasets to read .gz and .gzip files as well

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added support to properly stream and read .gz files.

- Added test cases for .json.gz files, and a mix of .json and .gz files.

Change-Id: Ic16044966400954d0cb7c36b99839ad91267ff84

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6043

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 45 more files in changeset.
[ASTERIXDB-2718][EXT] Add more test cases to cover complex JSON formats

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added test cases for JSON single line, multiple lines, arrays, nested

objects, ... etc.

Change-Id: I3cdcd2a3f417b9a045067c2fb50249c3bdf02211

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5923

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 26 more files in changeset.
[NO ISSUE]: Removing unused fields

Change-Id: Ifaec20a267a2e0e388f32c2d543153bc96369489

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5563

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

[ASTERIXDB-2697]: Implementing AWS s3 as external data source

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Added an external reader for AWS S3.

- Updated query translator to include the WITH

parameters into the dataset details when creating

an external dataset.

- Added test case for AWS S3 using an S3 mocking server

to avoid using real credentials.

Change-Id: I71d89116c0bb404c9621b16f21a6a31cbf7bb7f6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5025

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 38 more files in changeset.
[ASTERIXDB-2689][FUN] Agg Avg: skip and warn for invalid values

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently, the aggregate avg function throws an exception

when an invalid (non-numeric but not null or missing) value

is encountered, this patch changes the behavior to skip

the invalid value (treat it as null) and issue a warning.

- Added test cases for serializable aggregate avg (when

using hash group by).

Change-Id: I32037e0f178a84555ac868547eed5762e980f010

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5183

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 113 more files in changeset.
[ASTERIXDB-2569][TEST] Refactor type computer tests

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Converted type computer tests to be parameterized.

- Moved the tests to appropriate packages to use the

log4j logger.

- Type computer instances check is based on the field

type instead of field name.

Change-Id: I4426efcc9e825ebb00e04e3783d18bb1cbb63a90

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3393

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2535][COMP] Fix uuid present in insert/upsert statement

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added a new record merge function, extending the old one, to handle

the merge of duplicate fields.

- Updated the record merge type computer to handle the merge of

duplicate fields properly at compile time.

- Added a new record merge descriptor and evaluator, extending

the old one, to handle the merge of duplicate fields properly

at runtime.

- Updated IntroduceAutogenerateIDRule to use the new record merge

function.

- Added test cases to test the newly added function for insert

and upsert statements.

Change-Id: I22100d3ff29864b8bfd54b0decb183e5056fdb4a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3370

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

  1. … 53 more files in changeset.
[NO ISSUE][FUN] Introduce data generator function

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduced data generator function to generate data with accordance

to the TPC Benchmark DS specifications.

- Data generator function accepts 2 arguments, table name and scaling

factor, this makes the same function reusable for all the desired

tables and scaling factors to be generated.

- Added test cases for the data generator functions.

Change-Id: Idb6bd6f697628395c70008e6f730bc5ca403da5e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3360

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[NO ISSUE] Introduce MissingNullInOutFunction annotation

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- This annotation is for functions that respect the missing/null in ->

missing/null out behavior.

- Removed all the hardcoded functions inside the NullMissingTest and

ExceptionIT test classes.

- Functions annotated with the MissingNullInOutFunction are

automatically included in the NullMissingTest and ExceptionIT tests.

- Renamed ExceptionIT class to NullMissingExceptionTest.

- Moved NullMissingTest and NullMissingExceptionTest to their

appropriate packages to read the Logger config properly.

Change-Id: I253373ae4c8e23d9f5ed38f5fbbdefe9680e04b4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3361

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 258 more files in changeset.
[NO ISSUE] Fix class name typo

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- CalendarDuartionFromDateDescriptor -> CalendarDurationFromDateDescriptor

typo fix.

Change-Id: Ice71fd27ccf9ffd39e4ff368063c0715e33959db

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3362

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[NO ISSUE][FUN] Code generator alternative

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Possible alternative to replace the generated code with a static

method that does the NULL and MISSING checking step.

- Missing and Null check uses static method and helper methods

instead of varargs as a method argument.

- NullMissingTest and ExceptionIT manually add the functions that need

to be tested, this behavior will be changed later, the functions

that need to be tested for the "missing/null in -> missing/null out"

behavior will have their own flagging mechanism (annotation or

a method) indicating that those methods will be tested, and

the manual addition of functions will be removed.

Change-Id: Icca2e2128c4b0f2bfd8675655cf5296cbbaeba88

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3043

Reviewed-by: Michael Blow <mblow@apache.org>

Contrib: Michael Blow <mblow@apache.org>

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 167 more files in changeset.
[NO ISSUE][COMP] Documenting BreakSelectIntoConjunctsRule

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added documentation for BreakSelectIntoConjunctsRule.

- Renamed some variables for easier readability and maintenance.

- No actual code/logic changes.

Change-Id: Iadc5dc41115f91caa835255396969eaf47e1356d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3230

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[NO ISSUE][LIC] Fix ASF License typo

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fix a typo in the ASF license in some files.

Change-Id: I8841434bb00f40833eaec73c76f44cb65fcfb3b0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3221

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

[NO ISSUE][COMP] Optim rules comments + minor changes

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- CheckFilterExpressionTypeRule checks if the optimization

context has the output type environment before computing it.

- Minor addition/changes to rule comments.

Change-Id: I6f5cb78125657ff2bb4658af40bfc9d9cbff184e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3201

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[NO ISSUE] Refactoring plan optimization method

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Refactoring plan optimization method.

Change-Id: I8275bca00b41969444824cffd12f542d802bf26f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3101

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

[NO ISSUE] Renaming AQL variables to more generic names

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Renaming AQL variables to more generic names.

Change-Id: If9cf5fc13e1da7c9e172c2b866643973fafaaf13

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3098

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>