Clone Tools
  • last updated 10 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2540][STO] Optimize Storage Disk I/O

- user model changes: yes. Add a new storage option:

storage.disk.force.bytes (default 16MB),

- storage format changes: no.

- interface changes: yes.

Introduced IPageWriteCallback to LSM indexes

Details:

- Bypass all queuing (from BufferCache and IOManager) for disk writes.

This queuing is unnecessary but destroys fairness among multiple

writers.

- Introduce IPageWriteCallback to control the behavior of disk page

writes. Currently, this interface is used to perform disk forces

regularly for each writer thread.

Change-Id: I1f618dc7c186623e860239b4d97640fe3528e75b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3285

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 142 more files in changeset.
[NO ISSUE] Make IOManager more configurable

Change-Id: I1c8ad11c2b8b983ef4bf7cf78c2f068accddfff4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3133

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Contrib: Michael Blow <mblow@apache.org>

  1. … 10 more files in changeset.
[NO ISSUE] Bump versions to 0.[39].4.2-SNAPSHOT

Change-Id: If0cf05002c8019c1e1af43a18e75e617c0f7d4c9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3299

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 101 more files in changeset.
Advance version to AsterixDB 0.9.4.2-SNAPSHOT and Hyracks 0.3.4.2-SNAPSHOT

Change-Id:I47827d49e8a589ae885b134bf5c7557fae3b0620

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3209

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Ian Maxon <imaxon@uci.edu>

  1. … 101 more files in changeset.
Release Apache AsterixDB 0.9.4.1 and 0.3.4.1

Change-Id:I0af32fba00277c516b2671cc3567d61a2765f526

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3208

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Ian Maxon <imaxon@uci.edu>

  1. … 101 more files in changeset.
[ASTERIXDB-2422][STO] Introduce compressed storage

- user model changes: yes

- Add new configuration in the with-caluse to enable compression

- Add new nc configuration in the config file

- storage format changes: yes

- Pages of the primary index can be compressed

- Add a companion file (Look Aside File) with the compressed index

- Allow optional values in the LocalResource

- Add compression information in Metadata.Dataset

- interface changes: yes

- ICCApplicationContext:

- Add getCompressionManager()

- IBufferCache:

- Add getCompressedFileWriter(int fileId)

- ICachedPageInternal:

- Add setCompressedPageOffset(long offset)

- Add getCompressedPageOffset()

- Add setCompressedPageSize(int size)

- Add getCompressedPageSize()

Details:

- Add new integration test for this patch

- Fix ASTERIXDB-2464

- Add ddl-with-clause type validator

Additional details in the design document:

https://cwiki.apache.org/confluence/display/ASTERIXDB/Compression+in+AsterixDB

Change-Id: Idde6f37c810c30c7f1a5ee8bcbc1e3e5f4410031

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2857

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 191 more files in changeset.
Merge release-0.9.4 into master

Change-Id:Ied2b685f015e0f35563ebd0510231a5d663494f4

  1. … 38 more files in changeset.
Advance version to AsterixDB 0.9.5-SNAPSHOT and Hyracks 0.3.5-SNAPSHOT

Change-Id:Ied2b685f015e0f35563ebd0510231a5d663494f7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2953

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 99 more files in changeset.
Release Apache AsterixDB 0.9.4 and 0.3.4

Change-Id:Ib015ae6e13f38eb726512a9c5ddeeda428ad1dff

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2952

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 100 more files in changeset.
[NO ISSUE] Report all BufferCache write failures.

- user model changes: no

- storage format changes: no

- interface changes: yes

+ IPageWriteFailureCallback: used to notify async

IO caller when something goes wrong.

Details:

- Before this change, it is possible for failures to

be lost and for bulkload operations to not be

aware of failure to write some pages. This can be

dangerous.

- To avoid this, when sending a page to be written

a PageWriteFailureCallback is associated with the

page to notify the caller that a failure took place.

Change-Id: I97fd3dccff85dab84d644359be6f66b15ee708ef

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2787

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 47 more files in changeset.
[ASTERIXDB-1130][STO]: JSON serialization for persisted objects

- user model changes: no

- storage format changes:

This change replaces the use of Java serialization for persisted objects

such as dataset/index metadata, checkpoints, ect...

This will break backward compatibly with any existing AsterixDB instance.

However, the change is needed to enable future backward compatibility support

for persisted objects.

- interface changes:

IJsonSerializable: contains API to serialize a class as a JsonNode.

IPersistedResourceRegistry: contains a mapping between an IJsonSerializable

class and a unique type id. An IPersistedResourceRegistry is responsible

for generating the class identifier in the JSON output.

The class identifier will always contain the following attributes:

@type: a unique type id that identifies the object type.

@version: the version of the serialized class.

@class: the serialized class full name.

Any registered class with PersistedResourceRegistry must provide

a static fromJson(IPersistedResourceRegistry, JsonNode) method for

deserialization. This is ensured during the class registration process.

Change-Id: I5b103e06eab6627dbfe9d531caae1a3ac4b296da

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2752

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Murtadha Hubail <mhubail@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 128 more files in changeset.
[NO ISSUE][STO] Limit BufferCacheHeaderHelper Pool Size

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Limit the BufferCacheHeaderHelper pool size to the IO

requests queue size.

Change-Id: I7914241110e9fe3e5e21f35cf03ba6ecdf132343

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2727

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 2 more files in changeset.
[NO ISSUE] Correct import of InvokeUtil in BufferCache

- user-model changes: no

- storage format changes: no

- interface changes: no

Details:

- InvokeUtil was moved just when the previous change was submitted.

- That broke the build, this fixes it.

Change-Id: I4c9c1829db05ac691f7b0e6dc459d7646496ab6b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2396

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 1 more file in changeset.
[NO ISSUE][STO] Perform IO reads in uninterruptible threads

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Previously, IO reads are performed on the task threads.

- Task threads can be interrupted which can cause deadlocks

due to a JDK synchronization bug.

- After this change, 2 IO read threads are there per IO device.

- Threads will receive IO read requests and process them.

- Such threads are never interrupted and are killed through

the use of a poison pill.

Change-Id: Id28d57a222f42962284b24296cb9b91658e5dc77

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2387

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 3 more files in changeset.
Merge release 0.9.3 into master

Change-Id: I9ff693c4b3c8101557e555c266f68c91714d1742

  1. … 70 more files in changeset.
[ASTERIXDB-1564][CONF] Consistently Use Log4j2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace java.util.logging by Log4j2.

- Excluded classes due to their tests:

- IoUtil

- Tracer

Change-Id: Ic137571292f45de1f1994c61d328b97185012197

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2226

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 390 more files in changeset.
Advance version to AsterixDB 0.9.4-SNAPSHOT and Hyracks 0.3.4-SNAPSHOT

Change-Id:I5902012492cb51231a5b78c464a6ac0ab9dfdc06

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2171

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

  1. … 107 more files in changeset.
Release Apache AsterixDB 0.9.3 and 0.3.3

Change-Id:I86e454c944db8a4c960aa89bb0e9c6040181e28c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2170

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

  1. … 107 more files in changeset.
Advance version to AsterixDB 0.9.3-SNAPSHOT and Hyracks 0.3.3-SNAPSHOT

Change-Id:Id6cf0f514e357efe66c8da4a3cf10cd88e56fcca

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1925

Reviewed-by: Ian Maxon <imaxon@apache.org>

Integration-Tests: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

  1. … 106 more files in changeset.
Release Apache AsterixDB 0.9.2 and 0.3.2

Change-Id:I670467cf58f6432677237bcb2442ced65cd154ac

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1924

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 107 more files in changeset.
Separate index build from index access

This change separates index build from index access.

All indexes now have a single dataflow helper which

uses the index path to locate the resource on the nc

to read the resource from memory or disk.

Existing resource metadata and dataflow helpers were

combined into resource builders eliminating lots of

duplicated code.

Change-Id: Ie4ea3aaa63dff8d246fa43ca7c7359729bc8cf47

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1728

Integration-Tests: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 571 more files in changeset.
Advance version to AsterixDB 0.9.2-SNAPSHOT and Hyracks 0.3.2-SNAPSHOT

Change-Id:I4f05c7ceb1de1ed5c38d7c30e82c2ab7ee4323ca

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1680

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 106 more files in changeset.
Release Apache AsterixDB 0.9.1 and 0.3.1

Change-Id:I8719263e0550b0da1b715e3052e458fefca1ac8f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1679

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 107 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 70 more files in changeset.
[maven-release-plugin] prepare release apache-hyracks-0.3.0-rc2

  1. … 70 more files in changeset.
Revert "[maven-release-plugin] prepare release apache-hyracks-0.3.0-rc1"

This reverts commit 0c940ef7e2014b125ca54e290a3563f44a6e524e.

  1. … 70 more files in changeset.
Revert "[maven-release-plugin] prepare for next development iteration"

This reverts commit ab6fd5e74e80f56858ab643a5863f8d6299b42bc.

  1. … 70 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 70 more files in changeset.
[maven-release-plugin] prepare release apache-hyracks-0.3.0-rc1

  1. … 70 more files in changeset.
Revert "[maven-release-plugin] prepare release apache-hyracks-0.2.18"

This reverts commit e6af0eee8f27019c7cf2114e66572543bbe84d18.

  1. … 70 more files in changeset.