Clone
 

yingyi bu <buyingyi@gmail.com> in asterixdb

Revert "ASTERIXDB-1945 [STO] Cleanup Buffer Cache API"

This reverts commit ae3daf6ef3397e583637360dc460c6391e03dc29.

Change-Id: I5e4e23f43a68e82c38fb8d1d7f4c0d01985c3a10

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1842

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 147 more files in changeset.
Add a test case for Hyracks error reporting.

Change-Id: Ic3ac4c7b0a07eacbc448ce5724085eb8d1e6bc4d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/527

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

ASTERIXDB-1198: make ISerializerDeserializer implementations in AsterixDB stateless.

Change-Id: I808b6583db6b48a0e9000dcf0ac88c03c336c3a5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/522

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

  1. … 50 more files in changeset.
ASTERIXDB-1206: call initialize()/deinitialize() in parallel for OperatorNodePushables in SuperActivityOperatorNodePushable.

Change-Id: I8700d5258d658ebbf711b4233bb0def1e8cf7c39

Reviewed-on: https://asterix-gerrit.ics.uci.edu/526

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

ASTERIXDB-1198: make ISerializerDeserializer implementations in Hyracks stateless.

Change-Id: I1ec86d0a93d8f15d88d68fab24dbe858c5ba8842

Reviewed-on: https://asterix-gerrit.ics.uci.edu/521

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

Fix ASTERIXDB-1194,ASTERIXDB-1195,ASTERIXDB-1196,ASTERIXDB-1197.

Change-Id: I7d167b64bf9ec754182b5b2fe44dfc7e5908c686

Reviewed-on: https://asterix-gerrit.ics.uci.edu/325

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 10 more files in changeset.
Add a constructor for FileSplit: FileSplit(String nodeName, String path, int ioDeviceId).

change-Id: I1c7cc7752dd7b6d5261ff02d591f25d76d442a1c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/324

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Clean up compilation warnings.

Change-Id: Idbfcd9c67f91d373c5f7269125778a5681021227

Reviewed-on: https://asterix-gerrit.ics.uci.edu/505

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 95 more files in changeset.
Clean up asterix-om to eliminate compilation warnings.

Change-Id: Ib1c562d087097d9cc749f1e04f1f06106c7f1625

Reviewed-on: https://asterix-gerrit.ics.uci.edu/503

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 30 more files in changeset.
ASTERIXDB-1159: make ARecordType read-only and separate its mutable state to RuntimeRecordTypeInfo.

Change-Id: Ibbfa73907ae9422a4ad85dc4db5fa568855beb17

Reviewed-on: https://asterix-gerrit.ics.uci.edu/501

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 7 more files in changeset.
Override the reset method in ByteArrayAccessibleOutputStream.

Change-Id: Ib43f0b0fd9291b4a7e08963abb850d368ae2c288

Reviewed-on: https://asterix-gerrit.ics.uci.edu/500

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

SQL++ support in AsterixDB: 1. implemented SQL++ expression to logical plan translator; 2. refactored REST API to be agnostic of query languages; 3. disabled fuzzy join queries for SQL++ runtime tests; 4. fixed several rewriting rules.

Change-Id: I82919c4527b304325059519d819a2c30cf2902a9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/479

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1365 more files in changeset.
SQL++ support in Algebricks: 1. added the OuterUnnestOperator; 2. fixed several rewriting rules.

Change-Id: I7dcf57f75ebc0a741b6ec9597525e226b6014fc0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/314

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 13 more files in changeset.
ASTERIXDB-1145: verify the Hyracks fix doesn't break a duplicate key insert test case.

Change-Id: I8f770320423d76ba0429213b3376033c4b141fe2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/492

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

ASTERIXDB-1145: Fix error propagating in operators/connectors: 1. When an AbstractUnarySourceOperator instance runs into an exception, it should call writer.fail() first and then throw the exception. 2. An IFrameWriter.fail() implementation should not throw yet-another exception, instead, it should just propgate the failure to its downstream operators and optionally set a "failed" state so that in the close()/nextFrame() method it can potentially behave differently from usual close()/nextFrame().

Change-Id: Ifb538155423687c4aa01a0485adeaab87f291547

Reviewed-on: https://asterix-gerrit.ics.uci.edu/491

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

ASTERIXDB-1145: Fix error reporting in AlgebricksMetaOperatorDescriptor to avoid undeterministic behaviors.

Change-Id: I07b5f0ab95f3b8feda14b7f6b0f591c237c8b4d3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/487

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

SQL++ parser: 1. refactored asterix-aql to become asterix-lang-common and asterix-lang-aql, where the former is the common part for different languages; 2. added asterix-lang-sqlpp on top of asterix-lang-common; 3. ported parser tests, optimizer tests and runtime tests in asterix-app to their sql++ version, and added parser tests for all the queries.

Change-Id: Ie5af4e3b692ca017ec047a1ba3b404a51beb3a2e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/466

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 10209 more files in changeset.
Fix AQL doc for SQL fans.

Change-Id: I36b94af0280909ba49eca4ad311db3a633b305b9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/420

Tested-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

ASTERIXDB-221: reduce unneceesary partitioning for hash joins.

For a hash join, start top-down data property optimization from

a partitioning-compatiable child, and hence the other child's

partitioning requirement could be updated.

Change-Id: I835ea712c2f427149d45464fcb3841b8d33f6507

Reviewed-on: https://asterix-gerrit.ics.uci.edu/395

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wenhai Li <lwhaymail@yahoo.com>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Let a run file reader have an option to delete the run file after it is read.

Change-Id: Iabbd7c3e00489e2dbd8b1d4b87c7c9f81e8116b9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/397

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -2
    • +6
    /hyracks/hyracks-dataflow-common/pom.xml
Set the priority of the following threads to be Thread.MAX_PRIORITY: 1. heartbeat thread at NC 2. IPC network thread 3. work queue thread in CC

Change-Id: I4e53a85e21a6bdee48a3ca8d004569700f911fbd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/389

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

This change includes: 1. avoid calling ByteBuffer.get() and ByteBuffer.put() for bloom filter set/get operations. ByteBuffer will check boundaries of the index for get/put. It's a very frequent operation. 2. let the hash table size in BufferCache be 3X of the buffer cache page count. This will even reduce conflicts.

Change-Id: Ifd47c410338ce557a494ab3605660bad1899e786

Reviewed-on: https://asterix-gerrit.ics.uci.edu/345

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Reviewed-by: Ian Maxon <imaxon@apache.org>

improve the buffer cache perf. with 1) a better hash function for fileid-pageid, 2) reduce synchronization in clock page replacement policy.

Change-Id: I296c589a556a9afa7f27c6f560fa07fc4e2c1861

Reviewed-on: https://asterix-gerrit.ics.uci.edu/342

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Performance fix for BufferCache. The dpid of a page of file is calculated by fileid<<32 + pageid. But BufferCache.hash(long dpid) returns the hash value dpid%pageMap.length. In many cases, the asterix configuration results in power-of-2 pageMap.length (buffer-cache-size/page-size), which makes fileid useless. That used to result in serious consequences: different partitions contend for the same cache bucket (which contains a link list of size #partitions) for most of the time and therefore the CPU couldn't be saturated.

Change-Id: I4afc406d612e569e23f65afdedc469459235ce7d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/341

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Reviewed-by: Pouria Pirzadeh <pouria.pirzadeh@gmail.com>

Fix the temporary dataset life time threshold.

Change-Id: I076c085d4decfa23ab7413e85fe60458dd96d058

Reviewed-on: https://asterix-gerrit.ics.uci.edu/339

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

1. Fix the memory bloat issue introduced by https://github.com/apache/incubator-asterixdb/commit/c66d23a5ac65ec5218ee47134aea423fd62a32cc Recycle object pools in ADMDataParser before parsing each top-level ADM record.

2. Avoid using Strings as parameters of object pools because string comparison is slow and several comparisons done for parsing each ADM record.

Change-Id: I31848d8bd42a07f72035aefa2d671fdb2fce037d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/334

Reviewed-by: Pouria Pirzadeh <pouria.pirzadeh@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Preston Carman <prestonc@apache.org>

Add record type and primary key in the response of connector servlet api.

Change-Id: Ieefe79557cfb3786a6b22371a2e64ac4161ff900

Reviewed-on: https://asterix-gerrit.ics.uci.edu/315

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Add a JSON rest-api for external connector that uses existing AsterixDB datasets.

Change-Id: I674110b26262fbbd93030b252113e153ff4580ef

Reviewed-on: https://asterix-gerrit.ics.uci.edu/288

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

AsterixDB changes for fixing issue873.

For example, in the following query plan, the change lets the optimizer recognize that $12 and $20 are equivalent.

Therefore, HASH_PARTITION_EXCHANGE [$$12] can be replaced by ONE_TO_ONE_EXCHANGE.

-- COMMIT |PARTITIONED|

project ([$$12])

-- STREAM_PROJECT |PARTITIONED|

exchange

-- ONE_TO_ONE_EXCHANGE |PARTITIONED|

delete from TinySocial:TweetMessages from %0->$$4 partitioned by [%0->$$12]

-- INSERT_DELETE |PARTITIONED|

exchange

-- ONE_TO_ONE_EXCHANGE |PARTITIONED|

materialize

-- MATERIALIZE |PARTITIONED|

exchange

-- HASH_PARTITION_EXCHANGE [$$12] |PARTITIONED|

assign [$$12] <- [function-call: asterix:field-access-by-index, Args:[%0->$$4, AInt32: {0}]]

-- ASSIGN |PARTITIONED|

project ([$$4])

-- STREAM_PROJECT |PARTITIONED|

assign [$$4] <- [function-call: asterix:open-record-constructor, Args:[AString: {tweetid}, %0->$$14, AString: {user}, function-call: asterix:field-access-by-index, Args:[%0->$$0, AInt32: {1}], AString: {sender-location}, function-call: asterix:field-access-by-index, Args:[%0->$$0, AInt32: {2}], AString: {send-time}, function-call: asterix:field-access-by-index, Args:[%0->$$0, AInt32: {3}], AString: {referred-topics}, function-call: asterix:field-access-by-index, Args:[%0->$$0, AInt32: {4}], AString: {message-text}, function-call: asterix:field-access-by-index, Args:[%0->$$0, AInt32: {5}]]]

-- ASSIGN |PARTITIONED|

exchange

-- ONE_TO_ONE_EXCHANGE |PARTITIONED|

unnest-map [$$14, $$0] <- function-call: asterix:index-search, Args:[AString: {TweetMessages}, AInt32: {0}, AString: {TinySocial}, AString: {TweetMessages}, ABoolean: {false}, ABoolean: {false}, ABoolean: {false}, AInt32: {1}, %0->$$20, AInt32: {1}, %0->$$21, TRUE, TRUE, TRUE]

-- BTREE_SEARCH |PARTITIONED|

exchange

-- ONE_TO_ONE_EXCHANGE |PARTITIONED|

assign [$$20, $$21] <- [AString: {15}, AString: {15}]

-- ASSIGN |PARTITIONED|

empty-tuple-source

-- EMPTY_TUPLE_SOURCE |PARTITIONED|

Change-Id: Ife8c378a62cdbbcd8c19b521de246162f1f3d6ec

Reviewed-on: https://asterix-gerrit.ics.uci.edu/267

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wenhai Li <lwhaymail@yahoo.com>

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>

Algebricks fix for issue 873.

Change-Id: I78a4a30638d6cc5681b5410046fff6345b515291

Reviewed-on: https://asterix-gerrit.ics.uci.edu/266

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wenhai Li <lwhaymail@yahoo.com>

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>