asterix-build-configuration.xml

Clone Tools
  • last updated 10 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE] More multi-CC support, ConfigManager updates

- add ability for OptionTypes to natively parse JsonNodes

- allow all options to be overridden at the NC level, not just NC options (i.e. common, cc)

- accept controller id from the CC, avoid configuring this on NCs

- update all CCs with metadata bootstrap, not just the primary CC

- remove TxnIdFactory static singleton, management by metadata node

- remove unused build-properties style test configs

- cleanup test iodevices

Change-Id: Iff60887bf71ce3f3ed7201afd9499612bfc83485

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2344

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -112
    • +0
    ./asterix-build-configuration.xml
  1. … 67 more files in changeset.
[NO ISSUE][STO] Fix memory leaks in storage

- user model changes: no

- storage format changes: no

- interface changes: yes

- Added javadocs to:

-- IBufferCache

-- IExtraPageBlockHelper

- Moved IBufferCache.setPageDiskId -> ICachedPage.setDiskPageId

- Renamed:

-- IBufferCache.flushDirtyPage -> IBufferCache.flush

-- IBufferCache.getNumPages -> IBufferCache.getPageBudget

- Removed:

-- IBufferCache.adviseWontNeed [not used]

-- IBufferCache.tryPin [not used]

details:

- Previously, when adding a kv pair to the metadata of a memory

component, we add a new Pair item to the ArrayList. After

this change, we only update it if it exists.

- VirtualBufferCache used to leak pages when reclaiming pages

of a file after deletion. This has also been fixed.

- New tests for VirtualBufferCache added:

- Checks for memory budget after end of testDisjointPins

- Concurrent Users pinning pages concurrently

- Test for large pages and ensuring allocated large

pages are accounted for through removal of cached

free pages.

Change-Id: I4ae9736c9b5fdba5795245bdf835c023e3f73b15

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2115

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 39 more files in changeset.
Remove Plan Plotter

Change-Id: Ib6a77070344a40c3ae5bb9ff20b8593506d3c746

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1587

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 15 more files in changeset.
Configuration Revamp

- Ini section of node / cc details now returns ini param names instead of

managix option names

- Normalized command line -vs- ini file configuration parameter names

- Eliminated unused parameters

- Ini validation

- Migrate *DB parameters out of [app] and into nc / cc sections as

appropriate

- Eliminate [app] section. Cluster-wide configuration lives in [common]

- Sort properties alphabetically when returned by HTTP api

Change-Id: I95b7e0bd4538ef42817c8826e76412150074b754

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1487

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 174 more files in changeset.
Let log level be default (WARN) in tests

Change-Id: I880e6bf015986dea9235360a859de2f49bfc6e3a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1433

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 6 more files in changeset.
Automatically set parameters.

The parameters include:

-- storage.buffercache.size

-- storage.memorycomponent.numpages

-- storage.memorycomponent.globalbudget

-- -Xmx for NCDriver

Change-Id: I02c233b86b10a998482b8d44123d45b53ebf14dc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1417

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 8 more files in changeset.
ASTERIXDB-1556, ASTERIXDB-1733: Hash Group By and Hash Join conform to the memory budget

- External Hash Group By and Hash Join now conform to the memory budget (compiler.groupmemory and compiler.joinmemory)

- For Optimzed Hybrid Hash Join, we calculate the expected hash table size when the build phase is done and

try to spill one or more partitions if the freespace can't afford the hash table size.

- For External Hash Group By, the number of hash entries (hash table size) is calculated based on

an estimation of the aggregated tuple size and possible hash values for the given field size in that tuple.

- Garbage Collection feature has been added to SerializableHashTable. For external hash group-by,

whenever we spill a data partition to the disk, we also check the ratio of garbage in the hash table.

If it's greater than the given threshold, we conduct a GC on Hash Table.

Change-Id: I2b323e9a2141b4c1dd1652a360d2d9354d3bc3f5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1056

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 41 more files in changeset.
Supports flexible degree of parallelism.

Detailed list of changes include:

- Add the IClusterInfoCollector interface and feed that to the compiler;

- Add the number of cores into the heartbeat;

- Add partition calculation in APIFramework;

- Fix partitioning property requirements in binary operators such as Join and UnionAll;

- Fix PushSubplanIntoGroupByRule for type propagation;

- Fix InlineLeftNtsInSubplanJoinFlatteningVisitor for the returned top join operator;

- Add unit test for partition calculation in APIFrameworkTest

- Add integration test SqlppExecutionFullParallelismIT, SqlppExecutionLessParallelismIT,

AqlExecutionFullParallelismIT, and AqlExecutionLessParallelismIT for different

parallelism settings;

- Refactor AQLExecutionTest and SqlppExecutionTest;

- Separate one cluster state test out from AqlExecutionTest.

Change-Id: If70faf52bed995c7098cc343f2fabd1b6c8d96e7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1398

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 115 more files in changeset.
ASTERIXDB-1570: "Unable to find free page in buffer cache after 1000 cycles"

Increase test buffer cache size to reliably accomodate 20M values

Change-Id: I620de9ade875f7f7de26aa7495d9fd873824ac83

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1346

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 3 more files in changeset.
Added more explanations for two config parameters

- Added more planataions for storage.memorycomponent.numpages and

storage.memorycomponent.globalbudget

Change-Id: Idea269eec4d89f665772267588ce2b019327de04

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1281

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 6 more files in changeset.
Log ResultStateSweeper As Finer, Test Default Level->INFO

Change-Id: Ie8e9408590734694ce1fa5840783adee8d7ce695

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1176

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 2 more files in changeset.
Introduce MessagingNetworkManager for NC2NC AppMessaging

This change introduces MessagingNetworkManager to NodeControllerService.

The MessagingNetworkManager is used to open channels that are used for NC2NC

application messaging. The read and write interfaces of the messaging channels

are set by a MessagingChannelInterfaceFactory which is set by the application.

Change-Id: I5c0bd7c11c1e78954ebceff49cb274d8073a64bd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/897

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 74 more files in changeset.
ASTERIXDB-1569: the config file now supports unit, comma, and decimal.

- This patch fixes an issue that the config file only supports byte

value for the byte required parameters. Now, we can use unit (KB, MB, GB ...),

comma, and decimal.

Change-Id: I763d1b06e93f75775c0294da0d609c2bf3e102c9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1064

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 13 more files in changeset.
Txn Log Replication Optimizations

- Add configurations for txn log replication.

- Establish handshake for continuous log replication.

- Replicate logs in batches instead of one by one.

- Eliminate HashMap iterator object creation per logs batch.

- Fix synchronization for jobs' ACKs from remote replicas.

- Bug fixes for big objects replication.

- Some SONAR fixes for old code.

Change-Id: I25b5b84eba0cd41ac8e87e71368072879fcf8582

Reviewed-on: https://asterix-gerrit.ics.uci.edu/883

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 16 more files in changeset.
Normalize XML Whitespace

Normalize whitespace in project XML files (e.g. pom.xml), to two-spaces

per indentation level.

Cleanup rewriter error messages for user-defined functions

Change-Id: Ic06e3ff4431508d878e0cc614fbf957887a3c682

Reviewed-on: https://asterix-gerrit.ics.uci.edu/836

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -85
    • +85
    ./asterix-build-configuration.xml
  1. … 167 more files in changeset.