Clone
 

ildar absalyamov <ildar.absalyamov@gmail.com> in asterixdb

[NO ISSUE][STO][IDX] LSM storage cleanup

- user model changes: no

- storage format changes: no

- interface changes: yes

- Replaced component lifecycle-related fabric methods in AbstractLSMIndex

with direct method calls of lsmComponent functions.

- Extracted common lifecycle-related functionality from index-specific

disk/memory lsmComponents to interfaces.

- Introduced composable disk component bulkloader design which assembles the

proper bulkload pipeline from individual elements populating lsmFilters,

bloomFilters, buddyBTrees\deletedKeysBTrees, bTress\rTrees\invIndexes.

- Changed methods to return index-specific versions of objects (accessors,

components, index instances) to avoid nasty downcasting.

Change-Id: I6739d751b990e7a28e03e32a5de6e2b670d37a1e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2014

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 79 more files in changeset.
[ASTERIXDB-2114][MTD] Fixed incorrectly persisted FeedPolicy

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- New FeedPolicy was incorrectly persisting PolicyName & Description fields

Change-Id: I04beb24c861b116525a9da467fafc1742f4a73d4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2035

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

[STO][IDX] Eliminated excess antimatter in LSMBTree

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

A combination of some LSM operations (e.g. insert+delete) inserts a

record into the memory component of LSMBTree and then deletes it right

after leaving an antimatter entry. When no flush happens between two

operations this "tombstone" entry does not have any purpose and could

be eliminated during the flush without changing search semantics.

The fix introduces a new bit in record header which tracks if an entry

was inserted and then updated in-place. For secondary indexes this will

happen only when the record is changed from regular to antimatter.

The patch does not introduce changes in storage format because the bit

exists only for memory components.

In addition the patch refactored *TupleWriters, *TupleWriterFactories,

*TupleReferences, *Frames, *FrameFactories to return

index-specific types.

Change-Id: I12a67eff8431b52d1f9051b793a5a64b15c009e9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1538

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 75 more files in changeset.
Fixed race condition during ncMap lookup

Change-Id: I1bfbe712c100f48011a516c373ac8994028dc3dd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1792

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Fix for numeric overflow in integer constructors

Change-Id: I0cb3411bf9a808ee87f4938c60804a8d267c36d0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1509

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Fix a bug in secondary index entry matter/antimatter reconciliation during upsert

Change-Id: I5d5c4c44f49308757e05bed3a20475abdd3d38e7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1488

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Fix for ASTERIXDB-1738: Change feed fails to delete record with meta PK

Change-Id: I5ce8d9c69f96593ee305cfad4b44e486c9f1d6cc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1351

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

- Reapplied fix for ASTERIXDB-1109 on merged master - Cleaned up runtime testsuite, made it validate against XSD

Change-Id: I8fcf41c009f9faf51bc8bccc0c3c7217b7769121

Reviewed-on: https://asterix-gerrit.ics.uci.edu/712

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 12 more files in changeset.
ASTERIXDB-1229: - Fixed RemoveRedundantListifyRule to consider general expressions

Change-Id: I5e7b6f5ca4ed51e91de371b9d0b4e4dabdd2f2df

Reviewed-on: https://asterix-gerrit.ics.uci.edu/555

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Preston Carman <prestonc@apache.org>

ASTERIX-1211: Malformed metadata record for enforced open index - Changed type on the open olist field in index metadata to [ ANY ] - Fixed heterogeneous list deserialization

Change-Id: Ib587a138f59f37c369ba01e627a1ef9dd5a4989e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/532

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

ASTERIXDB-1109: Fixed deletion of records from open secondary index - Avoided creating calling record constructor in delete pipeline - Fixed the case when multiple open indexes enforce the type of the same field & covered it with tests

Change-Id: I41bde91401f67918365de7df19dd2f0de20c73d2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/461

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 24 more files in changeset.
ASTERIXDB-1109: Fixed deletion of records from open secondary index - Changed printers for idx\dataset insert\delete operators - Fixed visiting policy for idx insert\delete operator

Change-Id: Ib2036d2eac4b0a0c6ac2c2e7e1bac383b11106bd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/462

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Fixed m2e lifecycle mapping pom issues

Change-Id: Ifc8927a7fe90577d787c3d30c371ea95dd4aa24e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/453

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Fixed m2e lifecycle mapping pom issues

Change-Id: I51d3d173045e3f5edbca3c5f3835c6fdfa4afeac

Reviewed-on: https://asterix-gerrit.ics.uci.edu/454

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

ASTERIXDB-221: reduce unneceesary partitioning for hash joins.

Fixed broken optimizer tests

Change-Id: I0c8b4208c095d95e06357af6d2e9e465c74dec19

Reviewed-on: https://asterix-gerrit.ics.uci.edu/445

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

ASTERIXDB-1002: Fix exception handling in EmptyTupleSourceRuntimeFactory

Revisiting the previous fix by calling fail() on pipeline

Change-Id: I19f8c8485e483e4d4efeff939e6bd82c7a04a101

Reviewed-on: https://asterix-gerrit.ics.uci.edu/443

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Changed metadata storage format for nullable field types. Moved field name generation to the client out of metadata node code. Changed naming scheme for autogenerated types. Moved GroupName, CompactionPolicy & CompactionPolicyProperties fields up from External\InternalDetails to Dataset record type definition

Change-Id: I223aded8aaf80f0688358899c0e8b0d6988fac93

Reviewed-on: https://asterix-gerrit.ics.uci.edu/323

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 192 more files in changeset.
Added support of typed indexes over open fields & indexes over nested fields

Open indexes requires user to provide a type along with a indexed field name.

This type would be enforced for all the indexed records, i.e. index cannot be created if in some records a field with provided name has a different type.

Index-specific rewrite rules match provided type with the inferred types of other arguments in join\select statements and trigger index rewrite.

Nested indexes use the same semantics as the regular indexes, with exception that field could be located arbitrarily deep inside nested structure

Change-Id: I53d00aba243ccf7cf79cf7d775dd305813d24f98

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/97

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

  1. … 1777 more files in changeset.
Added abstract class for internal\external dataset scans.

Change-Id: I9c222ab5403a897aa3c8e6b48b6b4ad01891afbb

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/153

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

Changed test framework bag parsing and prepared transition to java8

Change-Id: Ia9e75e0d3f607f6a033f10fc894c34e9f6bf87d8

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/220

Reviewed-by: Chris Hillery <ceej@lambda.nu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -3
    • +3
    /asterix-doc/src/site/markdown/install.md
Fix for issue 771 - removed FieldLengthIgnoring comparators and added field length check in Pointable comparator

Change-Id: Icac725fb54db21f1aa37ae0db545fbdebb651b14

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/200

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Fix for issue 813 - enabling prefix btree search with non-matching predicates

Change-Id: I59eb43afa7dbcabd7f43269566d75c9361c55386

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/171

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 77 more files in changeset.
Fix to avoid incorrect serialization of query error report, returned to REST client

Change-Id: Ia86459d56c59858ab59ce49c154f8d95e7b4d146

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/160

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Fix for issue 715 - changed null value serialization policy in lists Fix for issue 760/756 - changed nullable fields type inference in listify Fix for issue 761 - changed type inference for records with multiple lists of non-primitive types

Change-Id: I700f0af27d8b9c2d8bf0c0a2906d2ffe4d0ee900

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/156

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Keren-Audrey Ouaknine <kereno@gmail.com>

Consider opaqueParameters in AbstractFunctionCallExpression equals&hashcode methods to make casting operators with the same argument distinguishable

Change-Id: I7b8bf5979732eb22dff824e48a57fdd1b1da4dda

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/100

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

Reviewed-by: Inci Cetindil <icetindil@gmail.com>