asterix-external-data

Clone Tools
  • last updated 29 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE] Fix type inference for ANY in UDFs

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

When data type is available, we should use defined data type. If not, we

should try to infer the data type from the data.

Change-Id: I6a03749fe041ec6f43de230d3e784122f7664545

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3405

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 1 more file in changeset.
[NO ISSUE] Error codes + NCService getter + Record metadata getter

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Task, added getter for NodeControllerService.

- RecordWithMetadataAndPK, added getter for metaTypes.

- Added extra error codes.

Change-Id: Ic3fcc6b771b4393542988cd0aa6fd992ccb81028

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3423

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 3 more files in changeset.
[NO ISSUE][FUN] Introduce IEvaluatorContext

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduce IEvaluatorContext - a context for evaluators

that is available both at runtime and during constant

folding at compile time

- IEvaluatorContext provides access to IHyracksTaskContext

which is only available at runtime. At compile time this

accessor returns null

- Instantiate evaluators with IEvaluatorContext

instead of IHyracksTaskContext

Change-Id: Iff90c7739e46d6602fcd0609e67f6a8cd9714ca5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3481

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 687 more files in changeset.
[ASTERIXDB-2597] Load UDFs via HTTP

- POST existing UDF format to /admin/udf/$DATAVERSE/$LIBNAME

- DELETE against that URL to remove UDFs

Change-Id: I6be9fef54c010bdb32f5c78af9b973f9843f442f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3386

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 37 more files in changeset.
[NO ISSUE][FUN] Fix type inference and casting in UDFs

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

The current UDF framework handles the argument types in a sloppy way. It

takes in the arguments and reads them as the expected data types in the

configuration. This could cause exception at runtime when passing in

arguments with unexpected datatypes. When setting the arguments, it did

a type casting for numeric values only to make sure the int64 from query

interface can be evaluated properly. However, this is not robust enough.

This patch fixes the type inference for UDFs during the complation time.

The ExternalTypeComputer is refactored to return defined data type, and

meanwhile checks the argument data types. Also, the

IntroduceDynamicTypeCastForExternalFunctionRule is modified to cover the

type castings for data types besides record type.

Change-Id: I40506fcca3cd8f14bbd6412359683433256c4c1f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3270

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Xikui Wang <xkkwww@gmail.com>

  1. … 23 more files in changeset.
[NO ISSUE][FUN] Add Function To Return Secondary Index Tuples

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add a new private function (dump_index) that can be used

to query all tuples of a secondary index. This function

can be used for testing to validate the expected contents

of secondary indexes.

- Add test case.

Change-Id: Iefae2a798126b9efb049596417b93499d075a608

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3325

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 11 more files in changeset.
[NO ISSUE][RT] Fix Secondary Indexes Bulkload/Upsert

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Prevent null values from being upserted into

secondary indexes.

- Use secondary index tuple filter when bulkloading

to prevent null/missing values from being inserted

into indexes of optional fields.

Change-Id: I7b847aece06a4387cc41389b0566d0c0dc98487d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3304

Reviewed-by: Luo Chen <cluo8@uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 8 more files in changeset.
[NO ISSUE] Bump versions to 0.[39].4.2-SNAPSHOT

Change-Id: If0cf05002c8019c1e1af43a18e75e617c0f7d4c9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3299

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 101 more files in changeset.
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 611 more files in changeset.
[NO ISSUE][FUN] Fix type inference in JListAccessor

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

As the list element type is available, there is no need to deserialize

element types from the data. Doing so, for example, will cause the

object type to be fully open type and closed fields to be recognized as

open fields. This will further cause NPE in UDFs. Modified the upperCase

test case to check this issue.

Change-Id: I4f437857779e5c0af175c695938fbd7208da244d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3264

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 8 more files in changeset.
Advance version to AsterixDB 0.9.4.2-SNAPSHOT and Hyracks 0.3.4.2-SNAPSHOT

Change-Id:I47827d49e8a589ae885b134bf5c7557fae3b0620

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3209

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Ian Maxon <imaxon@uci.edu>

  1. … 101 more files in changeset.
Release Apache AsterixDB 0.9.4.1 and 0.3.4.1

Change-Id:I0af32fba00277c516b2671cc3567d61a2765f526

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3208

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Ian Maxon <imaxon@uci.edu>

  1. … 101 more files in changeset.
Merge commit '1ff69c0' from 'stabilization-f69489' into 'master'

Change-Id: Ic8aa90be3ba3645ad22d21cc54abc83ab7008598

  1. … 24 more files in changeset.
[NO ISSUE][OTH] Support log redaction

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Support log redaction.

Change-Id: I602c833ba2a055da8fbe8782ec62be683ff4581b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3105

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 29 more files in changeset.
[NO ISSUE][EXT] Fix JRecord setField for open fields

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

setField method should be able to handle open fields.

Change-Id: I4ec185d62e78e8100215334bc656586268314797

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3068

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 1 more file in changeset.
[NO ISSUE] Update third-party library versions

Change-Id: I752876c0d5144019f228cb5e758892560d2c042d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3090

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Michael Blow <mblow@apache.org>

  1. … 29 more files in changeset.
[ASTERIXDB-2495][ING] Avoid LET in applying functions to feeds

- user model changes: no

- storage format changes: no

- interface changes: no

LET is not necessary for applying functions to data feeds. We could

inline the function calls when constructing the pipeline query.

Change-Id: I65842f9ac84891b363d7e0a02425258d0df794e7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3072

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 8 more files in changeset.
[NO ISSUE][ING] TweetParserFactory fix

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Fix the TweetParserFactory so it returns the correct record class.

Change-Id: If330bf060423027e01787224d1b2e93fd4d33d71

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3050

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wail Alkowaileet <wael.y.k@gmail.com>

[NO ISSUE][ING] Tweets ingestion related refactoring

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

As the incoming Tweets are merely JSON records, the Twitter adapter

should be able to use JSON parser. For that purpose, we changed the

record record to be consistent with JSONDataParser which uses char[]

instead of String. Also, "JSON" should also be a valid way to specify

using JSON parser.

Change-Id: Id6d656a4af974499ef0df2e389152ca205c2d078

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3032

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wail Alkowaileet <wael.y.k@gmail.com>

[ASTERIXDB-2480][ING] Fix http-feed compatibility with JSON parser

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

The generic record used in current http-feed returns length of "-1"

which will cause the JSON parser to fail. For compatibility

consideration, we should use CharArrayRecord instead.

Change-Id: I719c98376583a3316dbc291696df65a97dd4b26e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3026

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wail Alkowaileet <wael.y.k@gmail.com>

  1. … 9 more files in changeset.
[ASTERIXDB-2286][COMP][FUN][HYR] Parallel Sort Optimization

- user model changes: yes

- storage format changes: no

- interface changes: yes

details:

- new plan for sort operation which includes sampling and

replicating the stream of data to be sorted. Sort-merge connector

is removed from the plan. The sorted result now is in multiple partitions.

- new optimization rule to check whether full parallel sort is applicable.

- new Forward operator to read the replicated sort input stream and

to receive the ouput of the sampling.

- new sequential merge connector to merge a globally ordered result residing

in multiple partitions (in addition to the connector's partition computer).

- "asterix-lang-aql/pom.xml" is changed as a result of refactoring

code related to the range map handling.

- new private sampling function to generate the range map object

(local & global functions) & their type computers.

user model changes:

- new compiler property is added to enable and disable parallel sort.

interface changes:

- "ILogicalOperatorVisitor.java" includes Forward Operator.

- "ITuplePartitionComputer.java" includes initialize() to enable partitioner

to do some initialization. FieldRangePartitionComputerFactory uses it to

pick a range map.

- "ITuplePartitionComputerFactory.java". createPartitioner() is changed to

createPartitioner(IHyracksTaskContext hyracksTaskContext). Context is needed

for transferring the range map throught the context.

Change-Id: I73e128029a46f45e6b68c23dfb9310d5de10582f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2393

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 356 more files in changeset.
[ASTERIXDB-2264][ING] Introduce Http Feed Adapter

- user-model changes: add http_adapter for feed.

- storage format changes: no

- interface changes: no

Details:

1. Added http feed. User may use following syntax to create a http feed:

create feed TweetFeed with {

"adapter-name" : "http_adapter",

"addresses" : "asterix_nc2:10002,asterix_nc1:10001",

"address-type" : "NC",

"type-name" : "TweetMessageType",

"format" : "adm"

};

2. Refactored insert-feed test case to avoid waiting for 10s.

3. Refactored some constants in feeds.

Change-Id: I3c197a3df557ecc01f07f0907688c4ea81379e40

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2994

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 12 more files in changeset.
Merge release-0.9.4 into master

Change-Id:Ied2b685f015e0f35563ebd0510231a5d663494f4

  1. … 38 more files in changeset.
[NO ISSUE] Enable Build/Run w/ Java 11

- upgrade hadoop libraries (2.2.0 -> 2.8.5)

Change-Id: I04377b91ea7535bd358c396fbd9fcb019c734814

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2986

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3030

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 17 more files in changeset.
[NO ISSUE] Enable Build/Run w/ Java 11

- upgrade hadoop libraries (2.2.0 -> 2.8.5)

Change-Id: I04377b91ea7535bd358c396fbd9fcb019c734814

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2986

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 17 more files in changeset.
[NO ISSUE][FUN] Add more data types for external library

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

1. Added more types for external library.

2. Added test case for data types.

3. Fixed minor bug where the parameter type is not trimmed in UDF.

Change-Id: I6ce73e791533b0617074536e0d841242d9e0ee31

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2871

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 10 more files in changeset.
[NO ISSUE] User-defined Function Documentation update

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

1. Updated the UDF documentation to be consisten with current master.

2. Cleaned default UDF package to remove useless UDFs.

3. Added the example in documentation as a test case for IT.

4. Reorganized the documentation to keep up with the new structure.

5. Minor changes to other documentation pages to keep style consistent.

Change-Id: I17b1b4d639ca38689298ce88145257e794eb90e1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2804

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 21 more files in changeset.
Advance version to AsterixDB 0.9.5-SNAPSHOT and Hyracks 0.3.5-SNAPSHOT

Change-Id:Ied2b685f015e0f35563ebd0510231a5d663494f7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2953

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 99 more files in changeset.
Release Apache AsterixDB 0.9.4 and 0.3.4

Change-Id:Ib015ae6e13f38eb726512a9c5ddeeda428ad1dff

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2952

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 100 more files in changeset.
[NO ISSUE][RT] Improve debuggability of runtime functions

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- This change introduce util methods to facilitate debugging

functions runtime. Runtime functions read and write

values from and into byte arrays.

- To allow easier inspection of such values, we provide

a method that takes a pointable and produces an adm

node representing the value pointed to by the pointable.

Change-Id: I33257507e9bdd3be80355218e4bafecd1c71c650

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2926

Reviewed-by: Till Westmann <tillw@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 4 more files in changeset.