Enabling Bulk load and Hive Scan error logging/skip feature Also Fixed the hanging issue with Hive scan (ExHdfsScan operator) when there is an error in data conversion.
ExHbaseAccessBulkLoadPrepSQTcb was not releasing all the resources when there is an error or when the last buffer had some rows.
Error logging/skip feature can be enabled in hive scan using CQDs and in bulk load using the command line options.
For Hive Scan
CQD TRAF_LOAD_CONTINUE_ON_ERROR ‘ON’ to skip errors CQD TRAF_LOAD_LOG_ERROR_ROWS ‘ON’ to log the error rows in Hdfs files.
For Bulk load
LOAD WITH CONTINUE ON ERROR [TO <location>] – to skip error rows LOAD WITH LOG ERROR ROWS – to log the error rows in hdfs files.
The default parent error logging directory in hdfs is /bulkload/logs. The error rows are logged in subdirectory ERR_<date>_<time>. A separate hdfs file is created for every process/operator involved in the bulk load in this directory.
Error rows in hive scan are logged in <sourceHiveTableName>_hive_scan_err_<inst_id> Error rows in bulk upsert are logged in <destTrafTableName>_traf_upsert_err_<inst_id>
Bulk load can also aborted after a certain number of error rows are seen using
LOAD WITH LOG ERROR ROWS, STOP AFTER <n> ERROR ROWS option
Column-level privileges Support for column-level privileges will be in multiple deliveries.
This delivery add the following portions:
1. Creation of the metadata table COLUMN_PRIVILEGE. This table is created when the INITIALIZE AUTHORIZATION command is run. Existing privileges are preserved, but warnings are issued referring to existing metadata tables. An UPDATE option will be added later.
2. Granting of column-level privileges Full support is present for granting column-level privileges. Privileges can be added and updated for one or more columns on a table or view. Support for WITH GRANT OPTION is coded, though not enabled until WITH GRANT OPTION is enabled at the object level.
3. SHOWDDL The SHOWDDL command displays column-level privileges. Regardless of the order the privileges were granted, SHOWDDL displays them in column order, and within each column, in the order they appear in the bitmap (SELECT, INSERT, UPDATE, REFERENCES).
4. Revoking of column-level privileges Only partially implemented. The basic operation of revoking granted column-level privileges and grant option for is implemented. All relevant security checks are performed. GRANTED BY is not implemented. RESTRICT and CASCADE options are not supported. Hence, any dependent objects remain when column-level privileges are revoked.
In addition to column-level revoke only be partially implemented, here are other items not present in this delivery:
1. Privileges can be granted to roles and revoked from roles, but REVOKE ROLE does not consider column-level privileges when determining if an object depends on a role's granted privileges. 2. Similarly, revoke at the object level does not consider column-level privileges that may allow an object to remain after an object-level privilege is revoked. 3. CREATE VIEW does not consider column-level privileges when determining if the user has authority on the referenced tables and views. 4. Run-time DML operations do not considered column-level when determining if the user has authority to perform the query.