Clone
 

luochen <cluo8@uci.edu> in asterixdb

[NO ISSUE][DOC] Fix doc of recovery memory budget

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

Fix the doc of recovery memory budget to clarify that

the memory budget is enforced for each job, not globally

Change-Id: I36650591fc6bbf5abae739f9fb1981ba22549be2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3983

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[ASTERIXDB-2541][STO] Introduce GreedyScheduler

- user model changes: yes.

Add new option: storage.io.scheduler (async/greedy)

- storage format changes: no.

- interface changes: yes.

Introduce IIndexCursorStats

Details:

- Introduce GreedyScheduler that always executes the merge

operation with the smallest number of remaining pages to minimize

the number of disk components

- Introduce IIndexCursorStats to collect the statistics of index scans.

This allows GreedyScheduler to know the remaning pages of merge

operations.

- Extend AbstractIoOperation so that GreedyScheduler can pause/resume

merge operations if needed.

Change-Id: I38fe394d1180d4e3f6796064c0e6c6630b6ad303

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3284

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 59 more files in changeset.
[NO ISSUE] Fix comment of concurrent merge policy test

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fix the comment of the test case using-concurrent-merge-policy.

Change-Id: I363f5739ac4bb2a9dd14df7be97372eb1e838dd6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3572

Reviewed-by: Luo Chen <cluo8@uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[ASTERIXDB-2600][STO] Introduce ConcurrentMergePolicy

- user model changes: yes. Add a new merge policy and make it as default

- storage format changes: no.

- interface changes: no.

Details:

- Introduce ConcurrentMergePolicy that performs concurrent merges

without the maximum component size.

- Make this merge policy as the default merge policy in AsterixDB since

the PrefixMergePolicy has made some wrong design decisions.

Change-Id: I2ed79847584b9fe846d62ad56ee094863538a2a2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3463

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 18 more files in changeset.
[ASTERIXDB-2540][STO] Optimize Storage Disk I/O

- user model changes: yes. Add a new storage option:

storage.disk.force.bytes (default 16MB),

- storage format changes: no.

- interface changes: yes.

Introduced IPageWriteCallback to LSM indexes

Details:

- Bypass all queuing (from BufferCache and IOManager) for disk writes.

This queuing is unnecessary but destroys fairness among multiple

writers.

- Introduce IPageWriteCallback to control the behavior of disk page

writes. Currently, this interface is used to perform disk forces

regularly for each writer thread.

Change-Id: I1f618dc7c186623e860239b4d97640fe3528e75b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3285

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 128 more files in changeset.
[NO ISSUE][STO]Ensure Bloom filters are not built for secondary indexes

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Calculate bloom filter fields properly based on the index type

so that we do not build unnecessary bloom filters for secondary indexes.

Change-Id: Ib4272e5278bf76fa78af36ccf10d516da12c4810

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3366

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[ASTERIXDB-2310][STO]Enforce Key Uniquness using PKIndex

- user model changes: no

- storage format changes: yes. Primary key index

now has bloom filters.

- interface changes: no

Details:

- Add bloom filters to primary key index.

- Introduce LSMPrimaryInsertOperator to separate uniqueness check from

the primary index. When the primary key index is available, it will be

used for uniqueness check. This implementation of this operation is

similar to LSMPrimaryUpsertOperator.

Change-Id: I7a52bb75ee5b14521972999df2f45ba62adc5af1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2453

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 38 more files in changeset.
[ASTERIXDB-2522][TX] Skip WAIT record during lock conflicts

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- A small optimization to our deadlock-free locking potocol by skipping

the WAIT record. Once the partial frame has been flushed and previous

records are committed, the locks held by the write thread will

eventually be released by the log flusher thread. There is no need to

force the writer thread to wait for the log flusher thread.

Change-Id: I6ef3979d6393d45a6b7b2eb5f09f147299b5cd9f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3232

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[ASTERIXDB-2468] Extend CountVarToCountOneRule to support no group-by

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Extend CountVarToCountOneRule to support the no group-by case so

that count queries without group-by can be handled as well.

- Also extend this rule to support SQL_COUNT. In this case, count(v)

is replaced with count(1) only v is not nullable.

Change-Id: Ie0808912cf45d3a914ebf7f69ec42f73b33c071f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3024

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 17 more files in changeset.
[ASTERIXDB-2467][STO] Fix locking protocol with bad tuples

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- The current locking protocol is not correct when there are bad tuples,

e.g., duplicates. When locking fails, the write must push partial frames

so that the previous acquired locks are released. However, if there are

bad tuples in the frame, after bad tuples are removed from the frame, the

state of pushed partial frames will be completely lost.

Change-Id: I86baa1f21ef7390777c8e2b0ce226095528e7f18

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3003

Reviewed-by: Xikui Wang <xkkwww@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>