Ensure HFile.Reader objects are closed Performance workloads are seeing mxosrvrs with a large accumulation of open sockets, and a resultant decrease in performance. The cause was HFile.Reader objects used in estimating a table's row count not being closed. This fix executes a close() on the Reader in a finally clause, such that it will always be executed even if an exception is thrown.
Even with this fix, it was noted that early phases of workloads showed a significant increase in compile times when row count estimation was enabled. To address this, the estimated counts were saved so that they would not be repeated as long as the NATable remained in cache. Also, estimation was avoided completely for metadata tables. With these changes, my own tests showed a very low overhead for estimation, on the order of 60-70ms per user table in the workload, regardless of query complexity or how many queries referenced the table. However, the results of performance benchmarks executed on clusters did not improve, and so this change also includes turning off the cqd that enables row count estimation for the optimizer (a separate cqd controls use of the feature for Update Statistics, and it is still on by default). Future work will attempt to address the performance impact of turning on the cqd.
Changes to support OSS poc. This checkin contains multiple changes that were added to support OSS poc. These changes are enabled through a special cqd mode_special_4 and not yet externalized for general use. A separate spec contains details of these changes. These changes have been contributed and pre-reviewed by Suresh, Jim C, Ravisha, Mike H, Selva and Khaled. All dev regressions have been run and passed.