hyracks-test-support

Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2688][HYR] Fix use of a Hyracks task across join stages

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

In hash join, a task from the build stage is being used in the probe stage.

This is a problem since such tasks have already finished and notified the CC they are done.

One observed issue is related to issuing a warning in the probe phase where some warnings are

not reported because they are issued to tasks that have finished (the way this happened is that

a comparator was created in the build phase using the build-phase task. Then, this comparator

was used in the probe phase and issued a warning).

- make IHyracksJobletContext extend IHyracksCommonContext so that it is also a frame manager context

- make activites of join operators use the joblet context instead of the task context for acquiring buffers

- create the probe-to-build comparator in the probe phase so that the right task is used in the comparator

Change-Id: I38a4a779b9620494f15606162f0f1e9487fd0984

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4563

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Michael Blow <mblow@apache.org>

  1. … 10 more files in changeset.
[NO ISSUE][STO] Persist Bloom Filter Existence in Index Metadata

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- To clearly identify whether an index has a bloom filter or

not for BTree indexes, persist this information in the index's

metadata stored on each NC.

- For backward compatibility, when reading an index's metadata that

was created before adding the hasBloomFilter field, default its

value based on whether or not the index is a primary key index.

- Remove unused special readObject from LSMBTreeLocalResource.

Change-Id: Icec570d490987de401c036790ee9567238a60301

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3804

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 20 more files in changeset.
[ASTERIXDB-2541][STO] Introduce GreedyScheduler

- user model changes: yes.

Add new option: storage.io.scheduler (async/greedy)

- storage format changes: no.

- interface changes: yes.

Introduce IIndexCursorStats

Details:

- Introduce GreedyScheduler that always executes the merge

operation with the smallest number of remaining pages to minimize

the number of disk components

- Introduce IIndexCursorStats to collect the statistics of index scans.

This allows GreedyScheduler to know the remaning pages of merge

operations.

- Extend AbstractIoOperation so that GreedyScheduler can pause/resume

merge operations if needed.

Change-Id: I38fe394d1180d4e3f6796064c0e6c6630b6ad303

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3284

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 73 more files in changeset.
[ASTERIXDB-2540][STO] Optimize Storage Disk I/O

- user model changes: yes. Add a new storage option:

storage.disk.force.bytes (default 16MB),

- storage format changes: no.

- interface changes: yes.

Introduced IPageWriteCallback to LSM indexes

Details:

- Bypass all queuing (from BufferCache and IOManager) for disk writes.

This queuing is unnecessary but destroys fairness among multiple

writers.

- Introduce IPageWriteCallback to control the behavior of disk page

writes. Currently, this interface is used to perform disk forces

regularly for each writer thread.

Change-Id: I1f618dc7c186623e860239b4d97640fe3528e75b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3285

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 140 more files in changeset.
[ASTERIXDB-2532][RT] per-operator profiling

Enables profiling in queries at the operator-level when the analyze

variable is set in a query.

Change-Id: Ie16f3901ae5b32920d8552d5fd1ec8bb6e2ec8ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3226

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 60 more files in changeset.
[ASTERIXDB-2631][COMP][RT] Limit number of reported warnings

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Limit number of reported warnings.

- added a new compiler property "COMPILER.RUNTIME.WARNINGS"

with default value set to 0.

- updated Stat class to have a field for the count of runtime

warnings.

- updated few classes to propagate the warnings limit number

to Hyracks Task class.

- added new Option to parse unsigned long values.

- TaskProfile has a field for the count.

- updated TestExecutor to allow specifying parameters in

the body of the POST method.

Change-Id: Ie8756f1b9dcb7bbce92bd77692490fbada9a2482

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3535

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 57 more files in changeset.
[NO ISSUE][RT] Add Thread-Based Stats Collector

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add infra to allow collecting thread-based stats during

runtime for any thread that belongs to a task.

- Collect number of pinned pages per thread and report it

in the TaskProfile.

- Aggregate pinned pages counters from all job tasks and

report it as diskIoCount in the metrics field in the json

response. The plan is to move this stats to the profile

field when it is introduced.

- Collecting pinned pages stats is currently enabled by

default for any job with IndexSearchOperatorNodePushable.

The plan is to allow enabling/disabling as part of the

job profiling change.

- Add test case for diskIoCount metric.

- Remove unused IndexSearchOperatorNodePushable constructor.

Change-Id: I44dfcedcadb3d0f48815b521e7d495e473b02e3d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3555

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 27 more files in changeset.
[NO ISSUE][COMP] Support runtime warnings during constant folding

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add IWarningCollector accessor to IEvaluatorContext

- Provide compiler's warning collector to function evaluators

during constant folding, so they can emit warnings

Change-Id: I61dd53adfbcad2f8479333579185258876f074e9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3484

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 8 more files in changeset.
[ASTERIXDB-2597] Load UDFs via HTTP

- POST existing UDF format to /admin/udf/$DATAVERSE/$LIBNAME

- DELETE against that URL to remove UDFs

Change-Id: I6be9fef54c010bdb32f5c78af9b973f9843f442f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3386

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 37 more files in changeset.
[ASTERIXDB-2598][RT] Add Support For Runtime Warnings

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Add the ability to add runtime warnings per task

and return the generated warnings as part of the

task profile on task completion.

- On successful job completion, aggregate warnings

from all task profiles of a job.

- Return the generated warnings in the query service

response as an array of "warnings" each with a code,

which is currently hard-coded to 1, and a message.

- Fix propagating source location to scalar aggregate

functions.

- Add a flag in test cases definition to indicate whether

or not to check for expected warnings and default it to

false.

- Generate warnings when min/max functions encounter

incomparable type or unsupported input.

- Add support in test framework to extract warnings

along with extracting the result field.

- Add support in test framework to validate generated

and expected warnings.

- Add test cases for min/max generated warnings.

Change-Id: I52fa5b807799487d62e67a8861068e1547aa629a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3451

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 33 more files in changeset.
[ASTERIXDB-2554][HYR] Add UTF8 and byte array comparator factories

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Add comparator factories for UTF8StringPointable, UTF8StringLowercasePointable,

UTF8StringLowercaseTokenPointable and ByteArrayPointable instead of using

PointableBinaryComparatorFactory, a wrapping factory that will create a factory

each time (which also creates a comparator each time).

Change-Id: Ied6a29210a3dc1ba9fd553fb0a67ff4340e4571f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3355

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 47 more files in changeset.
[NO ISSUE] Make IOManager more configurable

Change-Id: I1c8ad11c2b8b983ef4bf7cf78c2f068accddfff4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3133

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Contrib: Michael Blow <mblow@apache.org>

  1. … 9 more files in changeset.
[ASTERIXDB-2516][RT] Move primitive comparators to Hyracks and make singleton

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- moved 2 comparators, boolean and long comparators from asterix to hyracks.

- added byte, short, integer, float and double comparator

factories to Hyracks to replace PointableBinaryComparatorFactory.

- removed checking lengths of 0 from PointableBinaryComparatorFactory.

- changed tests to use the primitive factories.

Change-Id: If15dc4e0dd0db942a4cadb15abbe56cbfe617b48

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3294

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 51 more files in changeset.
[NO ISSUE] Bump versions to 0.[39].4.2-SNAPSHOT

Change-Id: If0cf05002c8019c1e1af43a18e75e617c0f7d4c9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3299

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 101 more files in changeset.
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 625 more files in changeset.
[NO ISSUE] Handle Accept-Charset in QueryResultApiServlet

- exercise non-UTF8 Accept-Charset in TestExecutor

- remove double-buffering on http responses

- minor refactoring / cleanup

Change-Id: I8f37eb684bf2457e5ff451bf5c8fbca742d531f2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3191

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

    • -0
    • +65
    ./src/main/java/org/apache/hyracks/test/string/EncodingUtils.java
  1. … 24 more files in changeset.
Advance version to AsterixDB 0.9.4.2-SNAPSHOT and Hyracks 0.3.4.2-SNAPSHOT

Change-Id:I47827d49e8a589ae885b134bf5c7557fae3b0620

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3209

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Ian Maxon <imaxon@uci.edu>

  1. … 101 more files in changeset.
Release Apache AsterixDB 0.9.4.1 and 0.3.4.1

Change-Id:I0af32fba00277c516b2671cc3567d61a2765f526

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3208

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Ian Maxon <imaxon@uci.edu>

  1. … 101 more files in changeset.
Merge commit '627b31f8141c9789ca44a5dc384c8c6ac30ecefc' from 'release-0.9.4-pre-rc' into 'stabilization-f69489'

Change-Id: I6a1257029f8aa12ec3fb4e3ce86ee3d42e40a131

  1. … 3 more files in changeset.
Merge release-0.9.4 into master

Change-Id:Ied2b685f015e0f35563ebd0510231a5d663494f4

  1. … 38 more files in changeset.
Advance version to AsterixDB 0.9.5-SNAPSHOT and Hyracks 0.3.5-SNAPSHOT

Change-Id:Ied2b685f015e0f35563ebd0510231a5d663494f7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2953

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 99 more files in changeset.
Release Apache AsterixDB 0.9.4 and 0.3.4

Change-Id:Ib015ae6e13f38eb726512a9c5ddeeda428ad1dff

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2952

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 100 more files in changeset.
[NO ISSUE] Put asterix-server components in folder

Change-Id: I6281454bcd6b3f4a6c0f14d0c5b1d082efdd0955

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2880

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3198

Tested-by: Michael Blow <mblow@apache.org>

  1. … 28 more files in changeset.
[NO ISSUE] Put asterix-server components in folder

Change-Id: I6281454bcd6b3f4a6c0f14d0c5b1d082efdd0955

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2880

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 28 more files in changeset.
[NO ISSUE] Misc Fixes for Point MBR RTree

- user-model changes: no

- storage format changes: no

- interface changes: no

Details:

1. Fix the MBR calculation of Point MBR RTrees when finalizing the

bulkload process.

2. Fix the nullFlagBytes of Point MBR Rtrees.

3. Add unit test cases to cover Point MBR RTrees.

Change-Id: Ice24112152a2d93c7d2316b7506e6d6e81f2df44

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2911

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 4 more files in changeset.
[NO ISSUE][OTH] Remove Unused Imports

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: Iafff39073d0fedaff74a26ef7e3260008a79ff0c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2893

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 67 more files in changeset.
[NO ISSUE][OTH] Cleanup Unnecessary Semicolons

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: I73141e6a5d7950c074a80ea664fa6109ff94cdc4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2888

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 56 more files in changeset.
[NO ISSUE][OTH] Cleanup Redundant Type Cast

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: I9c93ce90ad4aa789ae46284431e172ae4777bf55

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2887

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 29 more files in changeset.
[ASTERIXDB-2414][STO] Remove deleted component files from buffer cache

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- When activating an LSM index, we remove files of components

that were merged into a bigger component but not cleaned up yet.

- However, we sometimes leave a file reference mapped in the buffer

cache even when the file is removed from disk.

- This change ensures that all files are removed from the buffer

cache as well.

Change-Id: If0f11bc222662e4b50c1b47b1dfa6b30d1463b2e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2822

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 7 more files in changeset.
[ASTERIXDB-2314][HYR] Dataset in class names in Hyracks

- user model changes: no

- storage format changes: no

- interface changes: yes

Change-Id: I260add3a2dfb4a722440af5008a1a75f9446579e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2478

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 120 more files in changeset.