Clone Tools
  • last updated 12 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Temp dataset support: 1. DDLs for creating a temporary dataset 2. Garbage collection for temporary dataset 3. Reading, inserting, and deleting data from (to) a temporary dataset is locking-free (except metadata locks) and logging-free (except flush/merge/job-commit logs).

Change-Id: Id0960acd1b4dfbfa7685dba227634572ca50b126

Reviewed-on: https://asterix-gerrit.ics.uci.edu/241

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

    • -0
    • +39
    ./asterix/metadata/GarbageCollector.java
    • -0
    • +25
    ./asterix/metadata/IDatasetDetails.java
    • -23
    • +80
    ./asterix/metadata/MetadataCache.java
    • -25
    • +86
    ./asterix/metadata/MetadataManager.java
    • -7
    • +17
    ./asterix/metadata/api/IMetadataManager.java
  1. … 241 more files in changeset.
Make index selection deterministic when multiple avaialbe indexes are available.

The priority of indexes is as follows:

1. First choice primary B+ tree

2. Second choice some other B+ tree

3. Third choice some R tree

4. Finally, choose based on index/dataset/dataverse name strings.

Change-Id: I5a24d9ab407568c3241cdaf9255643e912874384

Reviewed-on: https://asterix-gerrit.ics.uci.edu/264

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -1
    • +39
    ./asterix/metadata/entities/Index.java
  1. … 6 more files in changeset.
Added support of typed indexes over open fields & indexes over nested fields

Open indexes requires user to provide a type along with a indexed field name.

This type would be enforced for all the indexed records, i.e. index cannot be created if in some records a field with provided name has a different type.

Index-specific rewrite rules match provided type with the inferred types of other arguments in join\select statements and trigger index rewrite.

Nested indexes use the same semantics as the regular indexes, with exception that field could be located arbitrarily deep inside nested structure

Change-Id: I53d00aba243ccf7cf79cf7d775dd305813d24f98

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/97

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

    • -17
    • +21
    ./asterix/metadata/MetadataNode.java
    • -1
    • +4
    ./asterix/metadata/api/IMetadataIndex.java
    • -18
    • +39
    ./asterix/metadata/declared/LoadableDataSource.java
    • -14
    • +55
    ./asterix/metadata/entities/Index.java
  1. … 1777 more files in changeset.
This change covers three points: 1. move the generation of UUIDs out of parsers 2. fixes a bug in the metadata lock when performing load operations 3. the external data scan operator is again a leaf operator 4. made sure the startup call doesn't return until the servers are ready to receive requests Merged master into branch for the second time.

The following commits from your working branch will be included:

commit da929c990249fe7f0699417fd7f7ea34908355af

Author: Abdullah Alamoudi <bamousaa@gmail.com>

Date: Fri Jan 16 11:51:14 2015 +0300

Moved generation of UUIDs out of parsers

Change-Id: I32437493cf48c79417f4280e9d8a85cfb1559b62

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/205

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

    • -0
    • +117
    ./asterix/metadata/declared/LoadableDataSource.java
    • -0
    • +180
    ./asterix/metadata/declared/PKGeneratingAdapter.java
    • -4
    • +5
    ./asterix/metadata/feeds/FeedUtil.java
  1. … 50 more files in changeset.
- Fixed Type Casting issue - Reorganized duplicated internal class in the DelimitedDataParser and DelimitedDataParserFactory - Prevented a user from creating an inverted index on a dataset with a variable-length PK - INT64 is now the default type - Issue 852 fixed

Change-Id: I2d71e8a21da4f709c3259a3d3f678c640f9e1160

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/192

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

    • -5
    • +6
    ./asterix/metadata/MetadataManager.java
  1. … 843 more files in changeset.
Add test case and documentation for CSV parsing with headers.

Also fix a minor CSV parsing bug with line counting.

Change-Id: Ib875d60aa2465d4a50ee50c5c0e9356185f35c73

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/228

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 8 more files in changeset.
Introduce ("header"="true") option for CSV parsing

Change-Id: I473bc7b66d0c0d1355b5fb3e392cb9ece151e50e

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/226

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Chris Hillery <ceej@lambda.nu>

  1. … 11 more files in changeset.
Modified locking on DatasetLifeCycleManager

Change-Id: Ia5ab435f53879ba1d08b6dee24eb4969c5ad16e3

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/206

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 13 more files in changeset.
Issue 548: Doc for CSV

Change-Id: Ia343a7a795876835607be024bb7b8ad3fec33933

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/194

Reviewed-by: Till Westmann <westmann@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 6 more files in changeset.
This change integrates a run statement for running Pregelix jobs into the AQL.

Therefore it also provides a new FlushOperator to flush all memory components of a dataset to disc.

Change-Id: I1f97cfdc79943abf035a7342bb777d59af6518e9

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/193

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 23 more files in changeset.
Avoid returning 'null[]' after unhandled exceptions

Some exceptions that bubble to the top do not have messages,

so this stops those exceptions from causing 'null []' to be returned

to the user. Instead the class of the exception and a default error

message are shown when there is no other message to show from

the exception.

Change-Id: I44d856e5c4600f0a1606971ade8fbc0b0bb46a88

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/190

Reviewed-by: Chris Hillery <ceej@lambda.nu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1 more file in changeset.
Issue 548: Initial implementation of CSV output method.

Can be selected via the HTTP interface by setting the Accept: header to

text/csv.

Displays strings, numerics, booleans, and a couple duration types. Detects

situations that cannot be respresented as CSV (list values, nested records)

and throws an exception.

Introduces "outputRecordType" set option to define a fixed RecordType that

all results will be coerced to, to ensure consistent CSV output.

Added test support for CSV output, with one test case for now.

Change-Id: Ib53da6b3c69e38095bdc684b0e8cd53b9f4b1543

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/165

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

  1. … 72 more files in changeset.
Add the Binary data type and corresponding helper functions to Asterix.

The binary data type is implemented as a bytearray. Its storage format

follows the String type which has 2 bytes for length and then store the

bytes contents.

Binary data will take hex("") or base64("") as the constructor method to

passing a hex string or base64 string into Asterix. For output we use

hex("") format.

The parse-[hex|base64](string) function will parse the corresponding hex

or base64 string to binary type. The print-[hex|base64](binary)

functions will print the binary to hex or base64 STRING format.

The sub-binary(binary, offset, [length]) function works the same as

substring(string, offset, [length])

The find-binary(srcbinary, targetbinary, [start-offset]) will find the

position of the targetbinary in the srcbinary.

Change-Id: I5ecf0cc115c44070fb5c1fc5b0ec12a95d4243a4

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/175

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 126 more files in changeset.
Fix issue 786, issue 601, the prefix search issue reported by Abdullah, and the workaround query for issue 785.

Change-Id: I8bc4f18cd34c3d581f99f1509a4531c093aeec31

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/145

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 21 more files in changeset.
Issue 804: Spell "Adapter" consistently

Change-Id: I3141c1ddd2357ba8dd8ad94d6b98abc78151cd72

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/148

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -2
    • +2
    ./asterix/metadata/feeds/FeedUtil.java
    • -2
    • +2
    ./asterix/metadata/feeds/IngestionRuntime.java
  1. … 32 more files in changeset.
Changes to fix issue 802 commit fa829826d717123e415a994cc69e47c7ec1833f0 Author: Young-Seok <kisskys@gmail.com> Date: Thu Sep 25 10:09:20 2014 -0700

Change-Id: I8f6153e482e2079a0654263802ecd4c396291082

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/140

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

  1. … 1 more file in changeset.
Fixed CSV parser to recognize quote and delimiter inside a string

Change-Id: Iaaabc23e86df4f9bbee9f06b7976d7fbdcbb0f3f

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/135

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

  1. … 75 more files in changeset.
Implement TokenizeOperator in addition to the changes made by Zachary Heilbron. Now, the compiler builds the bulk-load plan. For other details, please refer to the corresponding Hyracks changes.

Change-Id: I646539d88ea2bdc6da4fbf2b6e9460a6189125ff

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/79

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

    • -0
    • +122
    ./asterix/metadata/declared/FieldExtractingAdapter.java
    • -0
    • +59
    ./asterix/metadata/declared/FieldExtractingAdapterFactory.java
    • -3
    • +3
    ./asterix/metadata/feeds/FeedUtil.java
  1. … 104 more files in changeset.
apply code conventions to files containing tabs

Change-Id: I429b02253a5008c50540a79075c257d01bc39a99

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/112

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 12 more files in changeset.
Added a new merge policy, called correlated-prefix.

Change-Id: I8f961eb28b4ea460f6326c8b4acaea02b50819e9

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/108

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +4
    ./asterix/metadata/utils/DatasetUtils.java
  1. … 27 more files in changeset.
remove global metadata lock

Change-Id: Id05ff463fee356b3270b53d0b3137c4b1bc3d830

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/84

Reviewed-by: Sattam Alsubaiee <salsubaiee@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +88
    ./asterix/metadata/utils/DatasetLock.java
    • -0
    • +512
    ./asterix/metadata/utils/MetadataLockManager.java
  1. … 9 more files in changeset.
Make the filter field optional in the Dataset dataset of the metadata.

Change-Id: Ibd2ba0edf4c8969ba4974375f5b0e0f6ddfb02ba

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/94

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 13 more files in changeset.
Added filters to AsterixDB's primary and secondary indexes.

Change-Id: If16b2758bb25392c8dd0405d7d59ae503efad3f0

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/82

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -2
    • +81
    ./asterix/metadata/utils/DatasetUtils.java
  1. … 115 more files in changeset.
Add external indexes

This change include the following:

1. additional data parser for external data that parse hdfs records using Hive serdes.

2. allow users to create external data. this includes:

a) changes in metadata external dataset details.

b) addition of a new metadata index to store external file's statuses.

c) the pipeline for building the B-Tree and R-Tree indexes.

d) hyracks operators to fetch records with their RIDs using different formats.

e) hyracks operators to lookup and parse external records.

f) test cases for indexing and index access of different hdfs file formats.

g) exposing the secondary indexes over external data to the compiler.

3. adding a new aql command to refresh external datasets. this includes

a) global recovery on system startup.

b) changes in the aql parser.

c) construction of bulk modify pipelines and additional operators to perform local commit and abort operations (using 2PC protocol).

4. Added copyright header to all new files

5. Added additional test cases to test left outer join on external data

Change-Id: I1065a473299f6027eb073aeeba3a56d137f6f98e

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/70

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -1
    • +4
    ./asterix/metadata/IDatasetDetails.java
    • -0
    • +65
    ./asterix/metadata/MetadataManager.java
    • -18
    • +177
    ./asterix/metadata/MetadataNode.java
    • -0
    • +66
    ./asterix/metadata/api/IMetadataManager.java
    • -0
    • +86
    ./asterix/metadata/api/IMetadataNode.java
  1. … 136 more files in changeset.
fixed issue 731, 740, and more.

commit 0b46141bea8d503896dc06308f102131df2e4f3d

Author: Young-Seok <kisskys@gmail.com>

Date: Tue May 20 12:52:54 2014 -0700

fixed issues of access method rules that try to use incompatible indexes and similarity functions

commit a0ea4e411503de265f1883aa3837a45be4a8747a

Merge: bb8fe91 b5785a9

Author: Young-Seok <kisskys@gmail.com>

Date: Sun May 18 13:00:33 2014 -0700

merged from master branch to kisskys/left-outer-join-issue branch

commit bb8fe91ffd4fec3d495d32442020447693be8548

Author: Young-Seok <kisskys@gmail.com>

Date: Sun May 18 11:33:54 2014 -0700

another fix for picking available index for leftouterjoin plan

commit 60b057ecec6a157e3e11cb316ef7d38601483741

Merge: a743e44 6cb7fd9

Author: Young-Seok <kisskys@gmail.com>

Date: Sun May 11 22:22:42 2014 -0700

merged master to kisskys/left-outer-join-issue branch

commit a743e4493f0f84f7a71e671478592d487e7510e3

Author: Young-Seok <kisskys@gmail.com>

Date: Sun May 11 20:51:50 2014 -0700

changes for left-outer-join to pick available indexes

Change-Id: I0d89d20c6cc076f40d1fbc5687f0b70e49a91eed

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/33

Reviewed-by: Inci Cetindil <icetindil@gmail.com>

Tested-by: Ian Maxon <imaxon@uci.edu>

  1. … 73 more files in changeset.
some small cleanup

    • -2
    • +7
    ./asterix/metadata/feeds/FeedUtil.java
  1. … 4 more files in changeset.
Adjust creation of Rtree to reflect the new LSM Rtree Optimization

  1. … 2 more files in changeset.
fix for issue #732

    • -1
    • +1
    ./asterix/metadata/feeds/FeedUtil.java
  1. … 2 more files in changeset.
addressed review comments

  1. … 1 more file in changeset.
changes to fix issue 727

  1. … 5 more files in changeset.