Clone Tools
  • last updated 19 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move Hyracks to subfolder

  1. … 4413 more files in changeset.
Support Change Feeds and Ingestion of Records with MetaData

This change allows feeds to perform upserts and deletes

in order to perform replication of an external data source.

The change does so by performing the following:

1. The adapter produces [PK][Record]. (Record == null --> delete)

2. The insert is replaced by an upsert operator.

Change-Id: I3749349e2b9f1b03c8b310eb99d3f44d08be77df

Reviewed-on: https://asterix-gerrit.ics.uci.edu/620

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 24 more files in changeset.
a few formatting issues.

  1. … 3 more files in changeset.
Implemented the memory-bounded HashGroupby and HashJoin for BigObject

It contains both hash grouby and hash join changes.

The main change is

1. update the ExternalGroupby to Hash-based groupby

2. update the Join operators to use the Buffermanager.

The buffer manager part is moved from the Sort package to upper

level so that it can be shared by all the operators.

Change-Id: I248f3a374fdacad7d57e49cf18d8233745e55460

Reviewed-on: https://asterix-gerrit.ics.uci.edu/398

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 146 more files in changeset.
More Coverity Fixes

- CID 68210 (#1 of 1): Check of thread-shared field evades lock

acquisition (LOCK_EVASION)

6. thread1_overwrites_value_in_field: Thread1 sets index to a new

value. Now the two threads have an inconsistent view of index and

updates to fields of index or fields correlated with index may be

lost.

- CID 68209 (#1 of 1): Check of thread-shared field evades lock

acquisition (LOCK_EVASION)

6. thread1_overwrites_value_in_field: Thread1 sets index to a new

value. Now the two threads have an inconsistent view of index and

updates to fields of index or fields correlated with index may be

lost.

- CID 68208 (#1 of 1): Check of thread-shared field evades lock

acquisition (LOCK_EVASION)

6. thread1_overwrites_value_in_field: Thread1 sets index to a new

value. Now the two threads have an inconsistent view of index and

updates to fields of index or fields correlated with index may be

lost.

- CID 68435 (#1 of 1): UR: Uninitialized read of field in constructor

(FB.UR_UNINIT_READ)

1. defect: Uninitialized read of btreeCmp.

- CID 68416 (#1 of 1): UL: Lock not released on all paths

(FB.UL_UNRELEASED_LOCK_EXCEPTION_PATH)

1. defect:

org.apache.hyracks.storage.am.lsm.invertedindex.inmemory.PartitionedIn

MemoryInvertedIndex.updatePartitionIndexes(short) does not release

lock on all exception paths.

- CID 68415 (#1 of 1): UL: Lock not released on all paths

(FB.UL_UNRELEASED_LOCK)

1. defect:

org.apache.hyracks.storage.common.buffercache.BufferCache.sweepAndFlus

h(int, boolean) does not release lock on all paths.

- CID 68414 (#1 of 1): UL: Lock not released on all paths

(FB.UL_UNRELEASED_LOCK)

1. defect:

org.apache.hyracks.storage.common.buffercache.BufferCache.dumpState()

does not release lock on all paths.

- CID 68412 (#1 of 1): UL: Lock not released on all paths

(FB.UL_UNRELEASED_LOCK)

1. defect:

org.apache.hyracks.storage.am.lsm.common.impls.VirtualBufferCache.dele

teFile(int, boolean) does not release lock on all paths.

- CID 68337 (#1 of 1): RV: Bad use of return value

(FB.RV_ABSOLUTE_VALUE_OF_RANDOM_INT)

1. defect: Bad attempt to compute absolute value of signed random

integer.

- CID 68336 (#1 of 1): RV: Bad use of return value

(FB.RV_ABSOLUTE_VALUE_OF_RANDOM_INT)

1. defect: Bad attempt to compute absolute value of signed random

integer.

- CID 68335 (#1 of 1): RV: Bad use of return value

(FB.RV_ABSOLUTE_VALUE_OF_RANDOM_INT)

1. defect: Bad attempt to compute absolute value of signed random

integer.

- CID 68334 (#1 of 1): RV: Bad use of return value

(FB.RV_ABSOLUTE_VALUE_OF_RANDOM_INT)

1. defect: Bad attempt to compute absolute value of signed random

integer.

- CID 68259 (#1 of 1): Result is not floating-point

(UNINTENDED_INTEGER_DIVISION)

integer_division: Dividing integer expressions 2147483647 and 2, and

then converting the integer quotient to type double. Any remainder,

or fractional part of the quotient, is ignored.

Change-Id: I0f63f7fb75b1ac8ffc1fc1d70e66bb19b788fe8e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/661

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 13 more files in changeset.
Remove Trailing White Spaces

This change removes all trailing white spaces in

java files in hyracks

Change-Id: I1c2ea4ff1190a965a857c0c3760cbebdf21860b3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/645

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <michael.blow@couchbase.com>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 643 more files in changeset.
Pass resource partition id to IModificationOperationCallback

- Pass resource partition id to IModificationOperationCallback

to include it in txn logs.

- New cluster event NODE_SHUTTING_DOWN for graceful shutdown.

- Remove unnecessary replication STOP operation.

Change-Id: Iddd566ea97512c1dbd2217befdc2bb8822e77763

Reviewed-on: https://asterix-gerrit.ics.uci.edu/589

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 2 more files in changeset.
remove end-of-line whitespace

Change-Id: If3542874d7f3a536a6fe524aba2fffaf6447a614

Reviewed-on: https://asterix-gerrit.ics.uci.edu/600

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 241 more files in changeset.
Add Support for Upsert Operation

This change adds support for upsert operations. it includes

creating a primary and secondary upsert operators in addition

to adding a new function "before" to the index operation call

back to correctly perform locking for the upsert operation.

Change-Id: I2705f43b6e6d187ee29b9ba5a7946d422990022a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/476

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 52 more files in changeset.
Add flush() to IFrameWriter

This change introduces flush() method in frame writers. It is

intended to be used for long running jobs (Feeds) to flush contents

of frames all the way to storage.

Change-Id: I85424bab7965b71aac709280af066e1655457aa3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/584

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 82 more files in changeset.
Fix for ASTERIXDB-1247

It seems like the root of this is the testing harness closing the index/deleting the file before it has

had a chance to flush all of its pages. There are also some changes to cover potential

corner cases where confiscated pages could be lost, but this doesn't seem to directly

affect the bug.

Change-Id: Ia580242b3f7753fc2f793f879332de3270ee3fee

Reviewed-on: https://asterix-gerrit.ics.uci.edu/575

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 6 more files in changeset.
Make Index Path Relative to NC IO Devices

This change includes the following:

- Store relative index path in persisted local resource as resource name.

- Fetch index absolute path based on FileSplit IO device and index relative path.

This change will enable a node to perform operations on an index that was created

on another node by finding the absolve path based on the node IO deviec and

the index relative path. This is needed in case a node leaves the cluster and

a replica node is assigned to take over and recover the index partitions

that belonged to the failed node.

Change-Id: I09fabd097b8a995ca959fe60286acdb43edfcfa1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/563

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 12 more files in changeset.
Add Method to Get LSN Byte Offset in LSM Disk Components

The LSN byte offset is needed to support updating LSNs in remote replicas.

Change-Id: I2eb407e607157fe9c71a9cf22b04accc832be31b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/560

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 1 more file in changeset.
Changes required to support MessageBroker implementation

This change includes the following:

- API changes to support MessageBroker implementation.

- IResourceIdFactory interface to support application dependent implementation.

Change-Id: Ib9f49234eebe912c48e7f71980433a9b42595741

Reviewed-on: https://asterix-gerrit.ics.uci.edu/485

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 20 more files in changeset.
patch 2

  1. … 12 more files in changeset.
Changed the IFrameWriter Contract

Updated existing operators and added a test case for BTreeSearchOperatorNodePushable.

With this change, calling the open method itself moves it to the open state and

hence, close must be called.

Change-Id: I03da090002f79f4db7b5b31454ce3ac2b9e40c7f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/551

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 37 more files in changeset.
patch 1

  1. … 56 more files in changeset.
Fix ASTERIXDB-1192

Change-Id: I3fcdd584d761d0d06a30aad41ea2a14438bb43eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/507

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

Make LSM bulkload append-only and write-once.

Allows for usage of LSM indexes with underlying storage that is append-only.

This also results in a small improvement for LSM component bulk load speed.

- Tree metadata (filters, etc) now lie at the back of the tree file in

append-only mode.

-- Note that you should *not* ever give the append-only flag on bulk-load,

if the tree is ever to be modified in place.

- Append-only operations bypass the buffer cache for writes, but utilize

the buffer cache for memory allocation and reads.

- Addresses ASTERIXDB-1059

Change-Id: I80fb891b5310252143854a336b591bf3f8cd4ba7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/255

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 83 more files in changeset.
ASTERIXDB-1145: Fix error propagating in operators/connectors: 1. When an AbstractUnarySourceOperator instance runs into an exception, it should call writer.fail() first and then throw the exception. 2. An IFrameWriter.fail() implementation should not throw yet-another exception, instead, it should just propgate the failure to its downstream operators and optionally set a "failed" state so that in the close()/nextFrame() method it can potentially behave differently from usual close()/nextFrame().

Change-Id: Ifb538155423687c4aa01a0485adeaab87f291547

Reviewed-on: https://asterix-gerrit.ics.uci.edu/491

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 4 more files in changeset.
ASTERIXDB-1058: ensure memory availablity before allocation

This change includes the following:

- IResourceMemeoryManager responsible for resources memory budget calculation.

- ResourceHeapBufferAllocator which checks with IResourceManager before allocation.

Change-Id: I846ff45402410835f5ed0afd2f701509abc95222

Reviewed-on: https://asterix-gerrit.ics.uci.edu/480

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 6 more files in changeset.
ASTERIXDB-1058: Lazy LSM memory components allocation

Change-Id: I476e756f8d71260ea614c8c072fc9503053866c9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/405

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 9 more files in changeset.
ASTERIXDB-1102: VarSize Encoding to store length of String and ByteArray

This patch is to change the encoding format that stores the length value of

the variable length type (e.g. String, ByteArray) from fix-size encoding

(2bytes) to variable-size encoding ( 1 to 5bytes)

It will solve the issue 1102 to enable us to store a String that longer

than 64K. Also for the common case of storing the short string ( <=

127), it will save one byte per string.

Some important changes include:

1. Add one hyracks-util package to consolidate all the hyracks

independent utility functions. It will reduce the chances of having

duplicate utils in different packages.

2. Move parts of Asterix string functions down to Hyracks

UTF8StringPointable object, which will benefit the other dependencies,

such as VXQuery.

Change-Id: I7e95df0f06984b784ebac2c84b97e56a50207d27

Reviewed-on: https://asterix-gerrit.ics.uci.edu/449

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

  1. … 119 more files in changeset.
ASTERIXDB-1053: change IndexLifecycleManager API to use resource name

Change-Id: I2e8e974fc2f746959639ce94351f8e419a7f9093

Reviewed-on: https://asterix-gerrit.ics.uci.edu/343

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 9 more files in changeset.
Change license headers

Change-Id: I98b18f24a20dcd8dc75e828e47fb0ab88179a5be

Reviewed-on: https://asterix-gerrit.ics.uci.edu/386

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2055 more files in changeset.
Change folder structure for Java repackage

Change only the folders, not the files, for our package name change.

This will break the build, and needs to be followed by a change to

the package name in all of the source files. However performing

the folder move and file change in two steps lets Git understand

that the files are the same, and lets us track revisions across

those files.

Change-Id: I08aff75e25ac7c6298c32cf3402febbc4a318c2a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/307

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Chris Hillery <ceej@lambda.nu>

  1. … 3879 more files in changeset.
Introducing data replication API to LSM indexes

Change-Id: I80565fc9d74e30440d2df5917911904ba8f33c25

Reviewed-on: https://asterix-gerrit.ics.uci.edu/322

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 36 more files in changeset.
Change Java package from edu.uci.ics to org.apache

Change-Id: I99172d856e88954b00cf7cfb24d33bb400f53994

Reviewed-on: https://asterix-gerrit.ics.uci.edu/308

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 2019 more files in changeset.
Add Apache RAT License Auditor plugin, and fix missing licenses

Change-Id: I39d92ec6654c73b4e6b8ba76dd66770bb60c7b79

Reviewed-on: https://asterix-gerrit.ics.uci.edu/260

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Chris Hillery <ceej@lambda.nu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 32 more files in changeset.
VariableSizeFrame(VSizeFrame) support for Hyracks.

This patch replaced Frame/Accessor/Appender with the new API which

supports BigObject.

The ExternalSorter/TopKSorter/ExternalGroupSorter

have been implemented to support big object.

The Groupby && Join should work with BigObject also. But it will break the

memory budget when it encounter a big object. I will fix the memory

problem later in a separate CR.

The design about the frame allocation is

here:https://docs.google.com/presentation/d/15h9iQf5OYsgGZoQTbGHkj1yS2G9q2fd0s1lDAD1EJq0/edit?usp=sharing

Suggest review order:

Patch 12: It includes all of the sorting operators.

Patch 13: It applys the new IFrame API to all Hyracks codes.

Patch 14: Some bug fixes to pass all Asterix's tests.

Patch 15: Skip it!

Patch 16: Some bug fixes to the Asterix's tests in small frame setting.

Later Patch: address the comments

Change-Id: I2e08692078683f6f2cf17387e39037ad851fc05b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/234

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 215 more files in changeset.