asterixdb

Clone Tools
  • last updated 23 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Implemented the memory-bounded HashGroupby and HashJoin for BigObject

It contains both hash grouby and hash join changes.

The main change is

1. update the ExternalGroupby to Hash-based groupby

2. update the Join operators to use the Buffermanager.

The buffer manager part is moved from the Sort package to upper

level so that it can be shared by all the operators.

Change-Id: I248f3a374fdacad7d57e49cf18d8233745e55460

Reviewed-on: https://asterix-gerrit.ics.uci.edu/398

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 133 more files in changeset.
Remove Unnecessary Prints in Tests

Change-Id: I4def109c0a8b40343d59d5689f8a791249174478

Reviewed-on: https://asterix-gerrit.ics.uci.edu/655

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Coverity Fix for Bad Lock Object

CID 68477: Bad choice of lock object (BAD_LOCK_OBJECT)

- boxed_lock:

Boxing a primitive may or may not return a canonical boxed

representation depending upon the value of the primitive being boxed.

Thus, using a boxed primitive as a lock is dangerous.

Change-Id: Ib993d94bfae6b788b5b56d388fa7a33ec958dee4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/665

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

remove unused files in org/apache/asterix/common/api

Change-Id: I26ddd964e0caed9d8411c3e646f91b80b2a333ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/664

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

More Coverity Fixes

- CID 68210 (#1 of 1): Check of thread-shared field evades lock

acquisition (LOCK_EVASION)

6. thread1_overwrites_value_in_field: Thread1 sets index to a new

value. Now the two threads have an inconsistent view of index and

updates to fields of index or fields correlated with index may be

lost.

- CID 68209 (#1 of 1): Check of thread-shared field evades lock

acquisition (LOCK_EVASION)

6. thread1_overwrites_value_in_field: Thread1 sets index to a new

value. Now the two threads have an inconsistent view of index and

updates to fields of index or fields correlated with index may be

lost.

- CID 68208 (#1 of 1): Check of thread-shared field evades lock

acquisition (LOCK_EVASION)

6. thread1_overwrites_value_in_field: Thread1 sets index to a new

value. Now the two threads have an inconsistent view of index and

updates to fields of index or fields correlated with index may be

lost.

- CID 68435 (#1 of 1): UR: Uninitialized read of field in constructor

(FB.UR_UNINIT_READ)

1. defect: Uninitialized read of btreeCmp.

- CID 68416 (#1 of 1): UL: Lock not released on all paths

(FB.UL_UNRELEASED_LOCK_EXCEPTION_PATH)

1. defect:

org.apache.hyracks.storage.am.lsm.invertedindex.inmemory.PartitionedIn

MemoryInvertedIndex.updatePartitionIndexes(short) does not release

lock on all exception paths.

- CID 68415 (#1 of 1): UL: Lock not released on all paths

(FB.UL_UNRELEASED_LOCK)

1. defect:

org.apache.hyracks.storage.common.buffercache.BufferCache.sweepAndFlus

h(int, boolean) does not release lock on all paths.

- CID 68414 (#1 of 1): UL: Lock not released on all paths

(FB.UL_UNRELEASED_LOCK)

1. defect:

org.apache.hyracks.storage.common.buffercache.BufferCache.dumpState()

does not release lock on all paths.

- CID 68412 (#1 of 1): UL: Lock not released on all paths

(FB.UL_UNRELEASED_LOCK)

1. defect:

org.apache.hyracks.storage.am.lsm.common.impls.VirtualBufferCache.dele

teFile(int, boolean) does not release lock on all paths.

- CID 68337 (#1 of 1): RV: Bad use of return value

(FB.RV_ABSOLUTE_VALUE_OF_RANDOM_INT)

1. defect: Bad attempt to compute absolute value of signed random

integer.

- CID 68336 (#1 of 1): RV: Bad use of return value

(FB.RV_ABSOLUTE_VALUE_OF_RANDOM_INT)

1. defect: Bad attempt to compute absolute value of signed random

integer.

- CID 68335 (#1 of 1): RV: Bad use of return value

(FB.RV_ABSOLUTE_VALUE_OF_RANDOM_INT)

1. defect: Bad attempt to compute absolute value of signed random

integer.

- CID 68334 (#1 of 1): RV: Bad use of return value

(FB.RV_ABSOLUTE_VALUE_OF_RANDOM_INT)

1. defect: Bad attempt to compute absolute value of signed random

integer.

- CID 68259 (#1 of 1): Result is not floating-point

(UNINTENDED_INTEGER_DIVISION)

integer_division: Dividing integer expressions 2147483647 and 2, and

then converting the integer quotient to type double. Any remainder,

or fractional part of the quotient, is ignored.

Change-Id: I0f63f7fb75b1ac8ffc1fc1d70e66bb19b788fe8e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/661

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

ASTERIXDB-1322: fix the AQL translator and add tests.

Change-Id: I994bb69ca989c8c9e2ce417d62739a859e5ae94b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/663

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +55
    /asterix-app/data/csv/55.csv
ASTERIXDB-1322: fix variable visitors for SubplanOperator.

- Simplified the implementation of OperatorPropertiesUtil;

- Fixed PushProjectDownRule to not introduce project operator

into a subplan to project outer variables.

Change-Id: I53ee1e27b41c9c80d51a7e1baf058d97338c18a9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/662

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Fix Remove File Operator

The operator didn't get the correct path of the folder to be

deleted. After the fix, the operator gets the correct path

from the IO manager.

Change-Id: I5910409d79e9c1f8e7192a1050b2be0f823e912f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/659

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

Fixes for Coverity Issues

(https://scan.coverity.com/projects/hyracks)

- CID 68257 (#1 of 1): Resource leak (RESOURCE_LEAK)

5. leaked_resource: Variable outputStream going out of scope leaks the

resource it refers to.

- CID 68260 (#1 of 1): Useless call (USELESS_CALL)

side_effect_free: Calling equivalentVars.get(0) is only useful for its

return value, which is ignored.

- CID 68261 (#1 of 1): BC: Bad casts of object references

(FB.BC_EQUALS_METHOD_SHOULD_WORK_FOR_ALL_OBJECTS)

1. defect: Equals method for org.apache.hyracks.algebricks.core.

algebra.properties.LocalOrderProperty assumes the argument is of type

LocalOrderProperty.

- CID 68262 (#1 of 1): BC: Bad casts of object references

(FB.BC_EQUALS_METHOD_SHOULD_WORK_FOR_ALL_OBJECTS)

1. defect: Equals method for org.apache.hyracks.storage.am.rtree.

RTreeCheckTuple assumes the argument is of type RTreeCheckTuple.

Change-Id: I8ba6e578f173b1081a8f55b228704744f08588df

Reviewed-on: https://asterix-gerrit.ics.uci.edu/658

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Remove Unneeded Suppress Warnings

When the classes in this change were created, we were using

deprecated Hadoop library. Since we updated the library,

there is no need to suppress deprecation warnings.

Change-Id: I710c1efbe006ee6de22299f8e6e833a566fb884d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/656

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Michael Blow <michael.blow@couchbase.com>

Exclude "asterix-app/src/test/resources/externallib/" in .gitignore.

Change-Id: Ib1af6dd556009321d0803ffdf9916146e866bb87

Reviewed-on: https://asterix-gerrit.ics.uci.edu/654

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

ASTERIXDB-1205: fix union queries.

Fixed the AqlExpressionToPlanTranslator for union expression;

Eliminate shared operator references in translated logical plans;

Removed IntroduceUnionRule;

Fixed tuple source for SQL++ join clause;

Added regression tests.

Change-Id: Ib79a899ec4b35ece6f8f9f59b80126fed0c48851

Reviewed-on: https://asterix-gerrit.ics.uci.edu/629

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 58 more files in changeset.
Fix ASTERIXDB-1205: cleanup union related rules.

-fixed EliminateSubplanWithInputCardinalityOneRule for general cases;

-fixed the type inference invocation in PushAssignBelowUnionAllRule;

-factored out ITypingContext and IVariableContext so that the operator deep copy

visitor can also be used in the language translator without types;

-added a rule to remove Cartesian product with an ETS input.

Change-Id: I7ad982108a4dc119249222c8ebb9e5897d93783c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/628

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

remove unused member

Change-Id: I8d0660faacc39aa3a0fca50f4813669a0179c408

Reviewed-on: https://asterix-gerrit.ics.uci.edu/653

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Enabled Feed Tests and Added External Library tests

Feed tests had been switched off for a while due to having too many

sporadic failures. Now, we are switching them back on.

In addition, a new set of tests have been added to test that external

library works as expected.

Change-Id: Idd1fccd136fa2645b2707bbf7c04e60991ae8d4a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/625

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +1106
    /asterix-app/data/external-parser/August16-20-long.txt
    • -0
    • +12869
    /asterix-app/data/external-parser/dropbox/jobads1.txt
    • -0
    • +12869
    /asterix-app/data/external-parser/dropbox/jobads2.txt
    • -0
    • +12869
    /asterix-app/data/external-parser/jobads.new
    • -0
    • +1106
    /asterix-app/data/external-parser/jobads.old
    • -0
    • +12869
    /asterix-app/data/external-parser/jobads.txt
    • -0
    • +684032
    /asterix-app/data/external-parser/testdata.txt
  1. … 190 more files in changeset.
Add Test Servers to Test Framework

This change introduces the ability to start and stop

different servers for any purpose, especially for testing

External data sources and different types of feeds.

Change-Id: I3d0434925972770cdae168656e1672cf0f225980

Reviewed-on: https://asterix-gerrit.ics.uci.edu/651

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

Add single-query-wide unique id generator.

Change-Id: I302f1547c5668250cf4801dbe26ae2aa6f455aa6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/647

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

remove unused pagacke org/apache/asterix/om/io

Change-Id: Ib1239b5e6c83b6b93da4f02d83492118bd0e97c2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/652

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

ASTERIXDB-1087: Make CC configurable from Asterix

Change-Id: I32b5e4b4242bdab16a664c9085bd8cd96fcc1142

Reviewed-on: https://asterix-gerrit.ics.uci.edu/648

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Fixed CreatePointDescriptor

- Fixed a typo in CreatePointDescriptor.

Change-Id: Ib2529e8cbdaf6d14b03dd27de9d8c0f8afa451f4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/650

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Fix Potential IllegalArgumentException

In the event multiple URLs were used for jar deployment,

an IllegalArgumentException would have been thrown.

Replaced problematic reflection call in favor of a sub-

class.

Change-Id: I7c9e1ec2c1541e7593dbc09dac52bfdc9e1cbec6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/649

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Remove Trailing Whitespace

This change removes all trailing whitespace in

java files in asterixdb

Change-Id: I32cda384f71dd5237a3c63e83c661f1cf64dc0eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/646

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 312 more files in changeset.
Remove Trailing White Spaces

This change removes all trailing white spaces in

java files in hyracks

Change-Id: I1c2ea4ff1190a965a857c0c3760cbebdf21860b3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/645

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <michael.blow@couchbase.com>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 643 more files in changeset.
patch 7

  1. … 3 more files in changeset.
ASTERIXDB-1304: remove interfaces and consolidate

Change-Id: Ibb8819f51c655fadaf47fdbad0af5f66b261b2b5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/643

Reviewed-by: Michael Blow <michael.blow@couchbase.com>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Merge asterix-experiments to master

Change-Id: I79b91e045bd056e62adb6e48795a6625baee0062

Reviewed-on: https://asterix-gerrit.ics.uci.edu/642

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +165
    /asterix-experiments/pom.xml
    • -0
    • +37
    /asterix-experiments/src/main/assembly/binary-assembly.xml
  1. … 240 more files in changeset.
Merge changes necessary for asterix-experiments pkg

Change-Id: I1256608bbb97d747747325c2050d5a2dc89beeb0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/641

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Asterix NCs Failback Support

- Allow Failed NCs to failback and takeover their partitions.

- New cluster API servlet for cluster state description.

- Remove nodeId from txn logs except remote FLUSH_LOG.

- Add partition id in UPDATE and Entity_COMMIT logs.

- Adapt remote recovery to new logs format.

- Refactor RecoveryManager and split Analysis and Redo phase.

- Spill remote recover logs to temporary file.

- Replicate files to remote replicas for partitions of interest only.

- Introduce NC active/inactive partitions concept.

- Test case for failback.

Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24

Reviewed-on: https://asterix-gerrit.ics.uci.edu/613

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 78 more files in changeset.
Support Sending Messages Alongside Frame Data

This change supports sending messages with records. The tuple Appender

reserves 100 bytes for a message. Before sending the frame, it appends

The message in the last tuple position. The message is read from the

task context as the shared object between different operators in the

pipeline. The first use of this feature will be within feeds to request

acks for at least once semantics.

Change-Id: Iaa23e9f8a909ddcafc1c3ee95181092eb04ee1ad

Reviewed-on: https://asterix-gerrit.ics.uci.edu/605

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

Support Sending Messages Alongside Frame Data

This change support sending messages with records. The tuple Appender

reserves 100 bytes for a message. Before sending the frame, it appends

The message in the last tuple position. The message is read from the

task context as the shared object between different operators in the

pipeline. The first use of this feature will be within feeds to request

acks for at least once semantics.

Change-Id: I56ae8124052c13a52ca42965b8d00e18ecf35a28

Reviewed-on: https://asterix-gerrit.ics.uci.edu/604

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <michael.blow@couchbase.com>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 4 more files in changeset.