hyracks-fullstack

Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][API] Add ability to identify secure servlet requests

Change-Id: Ia693e29a47b513a63fdce80383da90ba165c28d6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3603

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 1 more file in changeset.
Merge commit 'e8e59d0e6e' from 'stabilization-f69489'

Change-Id: I73ad57791bb7264a670bb36f4eb6f50640b4a86f

  1. … 1 more file in changeset.
[NO ISSUE] Update Guava, Jackson dependencies

Change-Id: I1a415367b0e403237deab782dd99214b927138db

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3576

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 1 more file in changeset.
[ASTERIXDB-2541][STO] Introduce GreedyScheduler

- user model changes: yes.

Add new option: storage.io.scheduler (async/greedy)

- storage format changes: no.

- interface changes: yes.

Introduce IIndexCursorStats

Details:

- Introduce GreedyScheduler that always executes the merge

operation with the smallest number of remaining pages to minimize

the number of disk components

- Introduce IIndexCursorStats to collect the statistics of index scans.

This allows GreedyScheduler to know the remaning pages of merge

operations.

- Extend AbstractIoOperation so that GreedyScheduler can pause/resume

merge operations if needed.

Change-Id: I38fe394d1180d4e3f6796064c0e6c6630b6ad303

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3284

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 59 more files in changeset.
[ASTERIXDB-2647][COMP] Random partitioner shouldn't preserve input's properties

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Random partitioner operator currently always propagates

the local properties of its input. If the data of the

input operator is partitioned, the random partitioner

should not propagate the local properties of the input

since the random partitioner destroys them.

Change-Id: I0378597451b3e6b25e8b45295159efec262abae2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3575

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[ASTERIXDB-2635][*DB] Fix JSON plan pretty printing

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Follow up change to fix printing "expressions" field

Change-Id: Ia994c0e0001cee2eca31f0bdf51ddb52320ae752

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3574

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[ASTERIXDB-2635][*DB] Fix JSON plan pretty printing

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Use Jackson to pretty print the logical plan as JSON.

- fixed JsonLogicalPlanTest to validate JSON plans

produced by optimizer tests.

Change-Id: Ic5ada2f31afebc3f219b0584b5ae527ddf9e326e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3567

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 7 more files in changeset.
[ASTERIXDB-2600][STO] Introduce ConcurrentMergePolicy

- user model changes: yes. Add a new merge policy and make it as default

- storage format changes: no.

- interface changes: no.

Details:

- Introduce ConcurrentMergePolicy that performs concurrent merges

without the maximum component size.

- Make this merge policy as the default merge policy in AsterixDB since

the PrefixMergePolicy has made some wrong design decisions.

Change-Id: I2ed79847584b9fe846d62ad56ee094863538a2a2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3463

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 30 more files in changeset.
[ASTERIXDB-2540][STO] Optimize Storage Disk I/O

- user model changes: yes. Add a new storage option:

storage.disk.force.bytes (default 16MB),

- storage format changes: no.

- interface changes: yes.

Introduced IPageWriteCallback to LSM indexes

Details:

- Bypass all queuing (from BufferCache and IOManager) for disk writes.

This queuing is unnecessary but destroys fairness among multiple

writers.

- Introduce IPageWriteCallback to control the behavior of disk page

writes. Currently, this interface is used to perform disk forces

regularly for each writer thread.

Change-Id: I1f618dc7c186623e860239b4d97640fe3528e75b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3285

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 128 more files in changeset.
[ASTERIXDB-2631][COMP][RT] Report count of all warnings

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Currently, only runtime warnings are counted. Include

parser & compile-time warnings, as well.

- removed the warn limit as a compiler option and made it

a request parameter.

- the warning collector of the parser collects all warnings

but allow pulling warnings up to a maximum desired.

- the warning collector of the compiler/executor collects up

to the request max-warnings and allow pulling up to a max desired.

- servlets collect from the parser and query compiler &

executor up to the request max-warnings.

Change-Id: Ia7e559313d32a8ba6f450f254a3a7b4b1e662a50

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3564

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 80 more files in changeset.
[ASTERIXDB-2532][RT] per-operator profiling

Enables profiling in queries at the operator-level when the analyze

variable is set in a query.

Change-Id: Ie16f3901ae5b32920d8552d5fd1ec8bb6e2ec8ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3226

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 46 more files in changeset.
Merge commit '8930f0d' from stabilization-f69489

Change-Id: I64cf862bc5ebe19a24acbccf821f68556699222e

  1. … 22 more files in changeset.
[ASTERIXDB-2631][COMP][RT] Limit number of reported warnings

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Limit number of reported warnings.

- added a new compiler property "COMPILER.RUNTIME.WARNINGS"

with default value set to 0.

- updated Stat class to have a field for the count of runtime

warnings.

- updated few classes to propagate the warnings limit number

to Hyracks Task class.

- added new Option to parse unsigned long values.

- TaskProfile has a field for the count.

- updated TestExecutor to allow specifying parameters in

the body of the POST method.

Change-Id: Ie8756f1b9dcb7bbce92bd77692490fbada9a2482

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3535

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 43 more files in changeset.
[NO ISSUE][RT] Add Thread-Based Stats Collector

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add infra to allow collecting thread-based stats during

runtime for any thread that belongs to a task.

- Collect number of pinned pages per thread and report it

in the TaskProfile.

- Aggregate pinned pages counters from all job tasks and

report it as diskIoCount in the metrics field in the json

response. The plan is to move this stats to the profile

field when it is introduced.

- Collecting pinned pages stats is currently enabled by

default for any job with IndexSearchOperatorNodePushable.

The plan is to allow enabling/disabling as part of the

job profiling change.

- Add test case for diskIoCount metric.

- Remove unused IndexSearchOperatorNodePushable constructor.

Change-Id: I44dfcedcadb3d0f48815b521e7d495e473b02e3d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3555

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 15 more files in changeset.
[NO ISSUE][CLUS] Attempt To Get Node IP From Existing Nodes

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- When we fail to resolve a node's host to an IP on node

failure/removal, attempt to find its IP from the list

of existing nodes. This is done to prevent failing the

node removal operation when the node's host address

cannot be resolved anymore.

Change-Id: I8f12a3dbb84a4cb731f1379efd65ca50606a8b07

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3544

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[ASTERIXDB-2625][OTH] Proper error message for invalid uuid

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Returns a proper message for invalid uuid. (incorrect uuid

length)

- Add a setter to allow setting the SourceLocation to an

already created exception.

Change-Id: I2cd836afcb35955e23e6cba7da5e17917d0705b0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3537

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 13 more files in changeset.
[NO ISSUE][COMP] Temporarily disable some property matching

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- ORDERED_PARTITIONED can only satisfy UNORDERED_PARTITIONED

if both use the same partitioning function. Currently there is no

mechanism for reasoning about partitioning functions so this case

has to be disabled until it is in place.

Change-Id: I7f280505f4fa02283fc14a0398760162e1150dc9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3532

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 7 more files in changeset.
[NO ISSUE] Always build with jdk target 8

Change-Id: I2f31adc13ce24c1a631620612259b774ab88fa0b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3531

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 1 more file in changeset.
[NO ISSUE][NET] Ensure Recycling Buffer and Notifying Sender is Atomic

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- To avoid synchronization issues that might occur

due to JVM reordering, ensure that both recycling

read buffers and notifying the sender of their

availability are done atomically before the next

buffer is received from the sender.

Change-Id: Ia3b1920f33bf7d4e7efbd2ea3405cbc4310a78c7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3520

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

(cherry picked from commit 32eed5f384c5851eae1c613fcb3b9532744ed595)

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3525

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][COMP] Introduce connector for partial broadcasts

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduce MToNPartialBroadcastConnectorDescriptor

- Refactor FieldRangePartitionComputerFactory to

use RangeMapSupplier

Change-Id: I4a6f8f17d1709862300db7ab386161b4dfbfee5a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3524

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

[ASTERIXDB-2611][*DB] Fix GroupAll with empty input

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This patch is to fix the issue with group by all where

some partitions are empty. The patch in general handles

propagating outer scope variables through group by all by using

subplans instead of using decor variables when translating

the AST.

- changed IsomorphismVariableMappingVisitor to stop re-visiting

the same nested tuple operator pair again

- moved getFdColumns() up in hierarchy to be shared by others

- added a check to prevent group-all with decoration fields

Change-Id: I56916fc60c696f6c702717210f108f2247b2a397

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3521

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 29 more files in changeset.
[NO ISSUE][NET] Ensure Handshake Connections List is Thread-Safe

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure the list of connections that completed their asynchronous

SSL handshake is accessed in thread-safe manner.

Change-Id: Ibfd4fc569b59f39d730ef04f5d69b23d91713ed7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3523

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[NO ISSUE][NET] Ensure Recycling Buffer and Notifying Sender is Atomic

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- To avoid synchronization issues that might occur

due to JVM reordering, ensure that both recycling

read buffers and notifying the sender of their

availability are done atomically before the next

buffer is received from the sender.

Change-Id: Ia3b1920f33bf7d4e7efbd2ea3405cbc4310a78c7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3520

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

[NO ISSUE][COMP] Improve warnings for SQL++ hints

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Issue warnings for all hints that were not processed

  by the SQL++ parser. A hint is not processed because

  it is either unknown or unexpected at given location

Change-Id: Iec46335a8db4360e5431b145cd9e98c7075c7fda

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3517

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 19 more files in changeset.
[NO ISSUE][COMP] Static range map in full parallel sort

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Support static range map in full parallel sort

- Add static range map option to ORDERED_PARTITIONED

structural property

- Add parseExpression() methods to SQL++ and AQL parsers

- Simplify RangeMapBuilder.parseHint()

Change-Id: I7eab6e6ede8c2dbb714a27801a76ad64bd9be1b8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3513

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 17 more files in changeset.
[ASTERIXDB-2577][RT] One frame per spilled partition in optimized hybrid hash join

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Makes sure there is one frame for each spilled partition during the build close.

Checks the length of large record before flushing a record to the disk blindly

as being large.

Change-Id: I82d4da57e9a9835cab61dc5cc43c45c728e0c2b6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3485

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

[NO ISSUE][COMP] Emit warning for cross product

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Emit warning for a cross product join

Change-Id: Icd2b27f23517842fc532db7741ee60e8da737082

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3512

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 4 more files in changeset.
Merge commit '8c99391' from stabilization-f69489

Change-Id: I139c265ba998c32ec049c8a8bbd7a5a213895d2a

  1. … 7 more files in changeset.
[ASTERIXDB-2599][STO] Cleanup compression LAFs

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Cleanup compression LAFs after merge.

- Delete LAF files when merged components contain

previous components.

- Make sure the recovery after non-graceful shutdown

(in the middle of cleanup) deletes the component and its LAF.

Change-Id: I17adb6145f7bf77470fd82f04321faf7a4007bf7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3498

Reviewed-by: Wail Alkowaileet <wael.y.k@gmail.com>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][NET] Do Not Flush Partial Frames on Network Errors

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not flush partially read frames when a network error

is reported to avoid processing invalid partial frames

before the network failure is detected by the channel

reader.

Change-Id: Ia35b58412cc293426c18a7cd66d8d0a5962db4e2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3503

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>