sorabh hamirwasia <> in drill

DRILL-7046: Support for loading and parsing new RM config file closes #1652

  1. … 49 more files in changeset.
DRILL-6934: Update the option documentation for planner.enable_unnest_lateral closes #1587

DRILL-6788: Intermittent unit test failure TestDrillbitResilience.failsWhenParsing: Query state should be FAILED (and not COMPLETED) closes #1499

DRILL-6785: DataClient is using RootAllocator in the bootstrap instead of dataPool

closes #1502

DRILL-6746: Query can hang when PartitionSender task thread sees a connection failure while sending data batches to remote fragment

closes #1470

DRILL-6729: Enable Unnest/Lateral Join feature by default Remove redundant option key in ExecConstants

closes #1456

DRILL-6687: Updated with review comments

DRILL-6687: Improve RemovingRecordBatch to do transfer when all records needs to be copied Add optimization in SelectionVector2 to enable RemovingRecordBatch to transfer ValueVectors from incoming to output container when all records needs to be copied. Modified FilterRecordBatch and LimitRecordBatch to play by this optimization

  1. … 5 more files in changeset.
DRILL-6674: Minor fixes to avoid auto boxing cost in logging in LateralJoinBatch

DRILL-6652: PartitionLimit changes for Lateral and Unnest Removing Ignore from TestE2EUnnestAndLateral

DRILL-6654: Data verification failure with lateral unnest query having filter in and order by

closes #1418

DRILL-6635: Update UnnesRecordBatch to handle kill differently with respect to PartitionLimitBatch in the subquery Fix in MockLateralJoinBatch for unnest kill tests

DRILL-6635: PartitionLimit for Lateral/Unnest PartitionLimitBatch initial implementation Add unit tests for PartitionLimitBatch

DRILL-6635: PartitionLimit for Lateral/Unnest Protobuf changes to add new operator PartitionLimit

DRILL-6616: Batch Processing for Lateral/Unnest

DRILL-6619: Lateral changes for implicit column Lateral changes to support batch processing from right side. It also fixes memory leak issue with LimitRecordBatch Added debug logging for Lateral and fixed a memory related issue in Limit

DRILL-6542 : IndexOutOfBoundsException for multilevel lateral queries with schema changed partitioned complex data

closes #1374

DRILL-6516: Fix memory leak issue with Sort and StreamingAgg together

DRILL-6561: Lateral excluding the columns from output container provided by projection push into rules

This closes #1356

DRILL-6548: IllegalStateException: Unexpected EMIT outcome received in buildSchema phase

closes #1352

DRILL-6530: JVM crash with a query involving multiple json files with one file having a schema change of one column from string to list

This closes #1343

DRILL-6535: ClassCastException in Lateral Unnest queries when dealing with schema changed json data Note: The issue was happening because for a left incoming all right batches were filtered and hence outputIndex was still 0 when new left incoming came with OK_NEW_SCHEMA. The OK_NEW_SCHEMA change was consumed without updating output container schema.

This closes #1339

DRILL-6445: Fix existing test cases in and add new test case for DRILLBIT_CONTEXT variable

This closes #1289

DRILL-6419: E2E Integration test for Lateral&Unnest

DRILL-6418: Handle Schema change in Unnest And Lateral for unnest field / non-unnest field

Note: Changed Lateral to handle non-empty right batch with OK_NEW_SCHEMA

closes #1271

DRILL-6446: Support for EMIT outcome in TopN - Added comments for TopNBatch and PriorityQueueTemplate - Adding support for SchemaChange across next() call with HyperVector in incoming container. This is achieved by adding a new method in HyperVectorWrapper which just updates the vector[] array holding multiple vectors with provided input ValueVector array. And also modifying RemovingRecordBatch GenericSV4Copier to hold reference to VectorWrapper instead of ValueVector[] for each column in incoming batch - Handling empty batches. Two cases like empty batches in the begining with EMIT outcome and empty batches between consecutive EMIT outcome but after receiving some batches with data and EMIT outcome. Note: In first case of empty batch it was only returning EMIT outcome without properly creating the output container and SV4 vector. Because of that there could be a case where let's say first batch with EMIT outcome is empty then TopN will return an empty batch with SV mode NONE and if later batch comes with some records and EMIT outcome, that will generate output batch with OK_NEW_SCHEMA (since TopN always generate first output batch with records with OK_NEW_SCHEMA as it returns output with SV4 mode). Also let's consider both batch with EMIT outcome were produced after processing first 2 rows of an input batch. This is a problem as this is simulating schema change across rows of same incoming batch which will never be the case.

Note: In second case of empty batches priority queue will not be null and will be uninitialized. Also optimize to send EMIT outcome with output batch which has all the data to return for current iteration

rather than sending it with OK followed by empty batch with EMIT outcome.

closes #1293

DRILL-6255: Drillbit while sending control message to itself creates a connection instead of submitting locally

closes #1253

  1. … 4 more files in changeset.
DRILL-6420: Add Lateral and Unnest Keyword for highlighting on WebUI

closes #1261

DRILL-6323: Lateral Join - Review feedback changes with License header fixes

This closes #1212

DRILL-6311: No logging information in drillbit.log / drillbit.out

closes #1202