asterixdb

Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
YARN integration for AsterixDB

This is an initial version of YARN integration for AsterixDB.

- Uses static assignment of CC and NC nodes to NM locations

- Stores state locally on each NM, outside of HDFS

- "All or nothing" container allocation. We don't attempt to

move or rellocate containers the RM may kill (yet).

- Retains feature parity with managix.

Change-Id: I49c849179d17fc7faa446b9be57a0695df6836ab

Reviewed-on: https://asterix-gerrit.ics.uci.edu/161

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -0
    • +103
    /asterix-common/src/main/resources/schema/yarn_cluster.xsd
    • -0
    • +236
    /asterix-doc/src/site/markdown/yarn.md
    • binary
    /asterix-doc/src/site/resources/images/asterix_webui.png
    • binary
    /asterix-doc/src/site/resources/images/running_inst.png
    • binary
    /asterix-doc/src/site/resources/images/yarn_clust.png
    • -0
    • +241
    /asterix-yarn/pom.xml
    • -0
    • +101
    /asterix-yarn/src/main/assembly/binary-assembly.xml
  1. … 51 more files in changeset.
Introduces Feeds 2.0

commit c3f577861fc705d848c1641605689cadd6973bae

Merge: ebc4cae fc0c2c0

Author: ramangrover29 <ramangrover29@gmail.com>

Date: Fri Jun 26 13:04:05 2015 -0700

Merge branch 'raman/feeds_2_release' of https://code.google.com/p/asterixdb-sandbox into raman/feeds_2_release

Conflicts:

asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/FeedServlet.java

asterix-external-data/src/main/java/edu/uci/ics/asterix/external/library/java/JObjectAccessors.java

commit ebc4cae21a7302869f953df1ebda601e798d12d2

Author: ramangrover29 <ramangrover29@gmail.com>

Date: Sat Jun 20 17:14:45 2015 -0700

Introduces Feeds 2.0

Some of the prominent chnages introduced are as follows

a) Support for building a cascade network of feeds (via secondary feeds feature)

b) Feed Management Console for tracking active feeds and associated metrics

c) Support for elastic runtime for data ingestion

d) Improved fault-tolerance with support for logging of failed records

Documentation has been added at asterix-doc/src/site/markdown/feeds/

commit fc0c2c0549a6ee8b202e57607d2e110478cd57bb

Author: ramangrover29 <ramangrover29@gmail.com>

Date: Sat Jun 20 17:14:45 2015 -0700

Introduces Feeds 2.0

Some of the prominent chnages introduced are as follows

a) Support for building a cascade network of feeds (via secondary feeds feature)

b) Feed Management Console for tracking active feeds and associated metrics

c) Support for elastic runtime for data ingestion

d) Improved fault-tolerance with support for logging of failed records

Documentation has been added at asterix-doc/src/site/markdown/feeds/

Change-Id: I498f01c591a229aaf51cec43ab20f3e5c4f072f4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/297

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +12
    /asterix-app/data/twitter/obamatweets_duplicate.adm
  1. … 369 more files in changeset.
Changes to support Feeds 2.0 (random partitioning of tuples)

commit 8b5c352d831aa0d7e006457f0b1430ac12b54731

Author: ramangrover29 <ramangrover29@gmail.com>

Date: Sat Jun 20 17:16:11 2015 -0700

Changes to support Feeds 2.0 (random partitioning of tuples)

Change-Id: I712c1f019cbc43f66d50620772c3df03c5944394

Reviewed-on: https://asterix-gerrit.ics.uci.edu/296

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

Make managix scripts properly executable

Most of the scripts managix was using were missing the shebang at the beginning of the script.

This can be problematic if one uses a shell besides bash. This patch fixes that.

Change-Id: Ie7722f832d0d25bec049e123a0ed1570d0aeb650

Reviewed-on: https://asterix-gerrit.ics.uci.edu/286

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Raman Grover <ramangrover29@gmail.com>

  1. … 6 more files in changeset.
Remove erroneous casts of list item offsets

The offset of an item in a list is read and written as an int,

but for some reason we were casting it down to a short when

recording the offsets during list building.

Change-Id: Ic058aa219e016d5efd719ddec0d1bb90a23e0c48

Reviewed-on: https://asterix-gerrit.ics.uci.edu/292

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

Fix for issue 872

Change-Id: If8cf621b8b89302cb28bd34f63e23f9c5f684351

Reviewed-on: https://asterix-gerrit.ics.uci.edu/290

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Add a JSON rest-api for external connector that uses existing AsterixDB datasets.

Change-Id: I674110b26262fbbd93030b252113e153ff4580ef

Reviewed-on: https://asterix-gerrit.ics.uci.edu/288

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

VariableSizeFrame(VSizeFrame) support for Hyracks.

This patch replaced Frame/Accessor/Appender with the new API which

supports BigObject.

The ExternalSorter/TopKSorter/ExternalGroupSorter

have been implemented to support big object.

The Groupby && Join should work with BigObject also. But it will break the

memory budget when it encounter a big object. I will fix the memory

problem later in a separate CR.

The design about the frame allocation is

here:https://docs.google.com/presentation/d/15h9iQf5OYsgGZoQTbGHkj1yS2G9q2fd0s1lDAD1EJq0/edit?usp=sharing

Suggest review order:

Patch 12: It includes all of the sorting operators.

Patch 13: It applys the new IFrame API to all Hyracks codes.

Patch 14: Some bug fixes to pass all Asterix's tests.

Patch 15: Skip it!

Patch 16: Some bug fixes to the Asterix's tests in small frame setting.

Later Patch: address the comments

Change-Id: I2e08692078683f6f2cf17387e39037ad851fc05b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/234

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 205 more files in changeset.
VariableSizeFrame(VSizeFrame) support for Asterix (Runtime Only)

Apply the https://asterix-gerrit.ics.uci.edu/#/c/234/ API changes to

Asterix level.

Change-Id: I5459e877707a1494fc1bebf03d4457a7427e9e0f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/259

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -0
    • +150
    /asterix-app/data/big-object/customer.tbl.big
    • -0
    • +6005
    /asterix-app/data/big-object/lineitem.tbl.big
    • -0
    • +1500
    /asterix-app/data/big-object/order.tbl.big
    • -0
    • +23
    /asterix-app/src/test/resources/runtimets/only.xml
  1. … 29 more files in changeset.
AsterixDB changes for fixing issue873.

For example, in the following query plan, the change lets the optimizer recognize that $12 and $20 are equivalent.

Therefore, HASH_PARTITION_EXCHANGE [$$12] can be replaced by ONE_TO_ONE_EXCHANGE.

-- COMMIT |PARTITIONED|

project ([$$12])

-- STREAM_PROJECT |PARTITIONED|

exchange

-- ONE_TO_ONE_EXCHANGE |PARTITIONED|

delete from TinySocial:TweetMessages from %0->$$4 partitioned by [%0->$$12]

-- INSERT_DELETE |PARTITIONED|

exchange

-- ONE_TO_ONE_EXCHANGE |PARTITIONED|

materialize

-- MATERIALIZE |PARTITIONED|

exchange

-- HASH_PARTITION_EXCHANGE [$$12] |PARTITIONED|

assign [$$12] <- [function-call: asterix:field-access-by-index, Args:[%0->$$4, AInt32: {0}]]

-- ASSIGN |PARTITIONED|

project ([$$4])

-- STREAM_PROJECT |PARTITIONED|

assign [$$4] <- [function-call: asterix:open-record-constructor, Args:[AString: {tweetid}, %0->$$14, AString: {user}, function-call: asterix:field-access-by-index, Args:[%0->$$0, AInt32: {1}], AString: {sender-location}, function-call: asterix:field-access-by-index, Args:[%0->$$0, AInt32: {2}], AString: {send-time}, function-call: asterix:field-access-by-index, Args:[%0->$$0, AInt32: {3}], AString: {referred-topics}, function-call: asterix:field-access-by-index, Args:[%0->$$0, AInt32: {4}], AString: {message-text}, function-call: asterix:field-access-by-index, Args:[%0->$$0, AInt32: {5}]]]

-- ASSIGN |PARTITIONED|

exchange

-- ONE_TO_ONE_EXCHANGE |PARTITIONED|

unnest-map [$$14, $$0] <- function-call: asterix:index-search, Args:[AString: {TweetMessages}, AInt32: {0}, AString: {TinySocial}, AString: {TweetMessages}, ABoolean: {false}, ABoolean: {false}, ABoolean: {false}, AInt32: {1}, %0->$$20, AInt32: {1}, %0->$$21, TRUE, TRUE, TRUE]

-- BTREE_SEARCH |PARTITIONED|

exchange

-- ONE_TO_ONE_EXCHANGE |PARTITIONED|

assign [$$20, $$21] <- [AString: {15}, AString: {15}]

-- ASSIGN |PARTITIONED|

empty-tuple-source

-- EMPTY_TUPLE_SOURCE |PARTITIONED|

Change-Id: Ife8c378a62cdbbcd8c19b521de246162f1f3d6ec

Reviewed-on: https://asterix-gerrit.ics.uci.edu/267

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wenhai Li <lwhaymail@yahoo.com>

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>

Algebricks fix for issue 873.

Change-Id: I78a4a30638d6cc5681b5410046fff6345b515291

Reviewed-on: https://asterix-gerrit.ics.uci.edu/266

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wenhai Li <lwhaymail@yahoo.com>

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>

Fix the HashFunction Bug in OptimizedHybridHashJoinOperatorDescriptor

The following commits from your working branch will be included:

The HashFunction used for InMemoryHashJoin is not update with level when

the OptimizedHybridHashJoin switches to InMemoryHashJoin. As the result,

it becomes the NestedLoopJoin after the 2nd round.

This patch is a fix for it.

Change-Id: Id25c85b7fadbb6bb969d0d94a51c60ac2573938e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/285

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Pouria Pirzadeh <pouria.pirzadeh@gmail.com>

Fix issue 897 (Stale metadata breaks ADM/AQL 101)

This fixes the issue for me, but I think this really needs some careful consideration.

Identical issues could be lurking anywhere the passed object is modified and that info needs to be conveyed to the caller.

I feel like I should not have to read from MetadataNode directly to get this info.

Change-Id: I088d33ad7810d7e1ca5471202cf20fa9a651195c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/283

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>

website fixes (moving to ASF resources)

- users mailing list

- location of website resources

- whitespace

Change-Id: I802dea3793aaa25887b3cc9be16a4d4be3c2c629

Reviewed-on: https://asterix-gerrit.ics.uci.edu/280

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

    • -1
    • +1
    /asterix-doc/src/site/markdown/install.md
    • -1
    • +1
    /asterix-doc/src/site/markdown/overview.md
    • -0
    • +15
    /asterix-doc/src/site/resources/data/fbm.adm
    • -0
    • +10
    /asterix-doc/src/site/resources/data/fbu.adm
    • -0
    • +6005
    /asterix-doc/src/site/resources/data/lineitem.tbl
    • -0
    • +12
    /asterix-doc/src/site/resources/data/twm.adm
    • -0
    • +4
    /asterix-doc/src/site/resources/data/twu.adm
    • binary
    /asterix-doc/src/site/resources/images/AsterixCluster.png
simple site

Change-Id: I02a125773e98cc39d8a6fbf5162176e2bcb5e707

Reviewed-on: https://asterix-gerrit.ics.uci.edu/276

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -45
    • +35
    /asterix-doc/src/site/markdown/index.md
    • -0
    • +45
    /asterix-doc/src/site/markdown/overview.md
    • binary
    /asterix-doc/src/site/resources/images/asf_logo_wide.png
    • binary
    /asterix-doc/src/site/resources/images/egg-logo.png
AsterixDB changes for: 1. fix "writerCount!=0" issue 2. protect deactivate/activate call of an index by its opTracker.

Note that this change includes https://asterix-gerrit.ics.uci.edu/#/c/269/.

Change-Id: I4669d49fd1cf9b058f7e0da233ad45c6c78b5797

Reviewed-on: https://asterix-gerrit.ics.uci.edu/279

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

1. Fix the "writerCount!=0 during component flushing" issue 2. Fix the duplicate LSM disk component file name issue by avoiding duplicate timestamps for different components.

Note that this change includes https://asterix-gerrit.ics.uci.edu/#/c/268/.

Change-Id: I805eab33603f52e19a1b76f1c315f9b75b6e3c03

Reviewed-on: https://asterix-gerrit.ics.uci.edu/278

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 4 more files in changeset.
Added a flag to identify first NC start during creation

Change-Id: If1b76ec83ae19f9c643a223e44712228ed7037f5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/272

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Temp dataset support: 1. DDLs for creating a temporary dataset 2. Garbage collection for temporary dataset 3. Reading, inserting, and deleting data from (to) a temporary dataset is locking-free (except metadata locks) and logging-free (except flush/merge/job-commit logs).

Change-Id: Id0960acd1b4dfbfa7685dba227634572ca50b126

Reviewed-on: https://asterix-gerrit.ics.uci.edu/241

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 238 more files in changeset.
Add a flag for LSM-based indices to indicate whether force pages to disk devices during flush and merge.

Change-Id: I988716c03cffe30b008e144d3a478ee25e367212

Reviewed-on: https://asterix-gerrit.ics.uci.edu/240

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 66 more files in changeset.
fix multi-delete statement issue

Change-Id: I32c55e225fb8cc3f164da7029b4c31c26c6e574d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/271

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

allow counter to reset from a start

Change-Id: Ibcc6fdb66bc758450fd75c70960b481692e06bfe

Reviewed-on: https://asterix-gerrit.ics.uci.edu/270

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Make index selection deterministic when multiple avaialbe indexes are available.

The priority of indexes is as follows:

1. First choice primary B+ tree

2. Second choice some other B+ tree

3. Third choice some R tree

4. Finally, choose based on index/dataset/dataverse name strings.

Change-Id: I5a24d9ab407568c3241cdaf9255643e912874384

Reviewed-on: https://asterix-gerrit.ics.uci.edu/264

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

README.md typo fix.

Change-Id: Id53935df48a1f0627b33b8084fbc52b09b4c4483

Reviewed-on: https://asterix-gerrit.ics.uci.edu/262

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Revise README.md

Change-Id: I778a5a872e2696a165bdf2bbdd9aefae71ec748a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/258

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

Add README markdown

This is mainly for GitHub.

Change-Id: I233c0eebe12954c8d311a19074c41841cdfa01c9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/256

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Range connector update with order by hint.

Change-Id: I1b36f6dea801950c39bf63f27a0ec70d3584c5b7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/254

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

  1. … 21 more files in changeset.
Range connector update with order by hint.

Change-Id: Iec1fbd79f62bfeef2081858bdfab3ff894f63e03

Reviewed-on: https://asterix-gerrit.ics.uci.edu/253

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>

Fixing LSMHarness issue.

Change-Id: I8afc0f189f5a64cc56be4a89903999c7c90a65c1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/252

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Fixing LSMHarness issue.

The following commits from your working branch will be included:

commit 4a2a16f16df99cbf29ac53bf7009e2dc07bdbb26

Author: hubailmor@gmail.com <mhubail@uci.edu>

Date: Fri Apr 24 17:24:02 2015 -0700

Fixing LSMHarness issue

Change-Id: I7c1b3e8283fc3a661c80202cc896a42a1b322416

Reviewed-on: https://asterix-gerrit.ics.uci.edu/251

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>