ExpPCodeOptsNativeExpr.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move core into subdir to combine repos

    • -12392
    • +0
    ./ExpPCodeOptsNativeExpr.cpp
  1. … 10768 more files in changeset.
Move core into subdir to combine repos

    • -12387
    • +0
    ./ExpPCodeOptsNativeExpr.cpp
  1. … 10622 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

    • -12392
    • +0
    ./ExpPCodeOptsNativeExpr.cpp
  1. … 10837 more files in changeset.
Fix bug 1323826 - SELECT with long IN predicate causes core file

Actually, this check-in does not completely fix the problem, but

it does allow IN predicates (and NOT IN predicates) to have a list

with as many as 3100 items in the list.

NOTE: There are many places in the SQL Compiler code that use recursion.

The changes in this check-in address the issue for long IN lists

and, to some extent, for INSERT statements that attempt to insert

many rows with a single INSERT statement. However, it is still possible

for someone to try a list that is too long. As you make the lists

longer, you find more recursive routines that have the same type of

problem(s) that are being fixed for certain routines by this check-in.

This check-in also fixes a couple of minor problems in the logic used to

debug Native Expressions code. These problems were in

.../sql/generator/Generator.cpp and

.../sql/exp/ExpPCodeOptsNativeExpr.cpp

There were 3 different techniques used to reduce the stack space usage of

various recursive routines that get invoked as a result of long IN lists

or NOT IN lists:

1) Move variables from the stack to heap.

2) Recode the recursive routine to pull out sections of code (not needed

during the recursion) and put those in their own routine. This cuts

the stack space usage because it enables the C++ compiler to generate

code for the recursive routine that needs significantly less stack

space.

3) Declare variables of type ARRAY on the stack (where the ARRAY

overhead is allocated from stack, but the contents come from heap)

to hold certain pieces of data where each recursive level of calling

needs its own value for the variable AND then change the code to use a

'while' loop to process the nodes in the node tree in the same order

that the original recursive routine would have processed the nodes.

Files changed for reducing stack space usage:

sql/optimizer/ItemCache.cpp - use method 2 on ItemExpr::generateCacheKey()

sql/optimizer/NormItemExpr.cpp - use method 2 on ItemExpr::normalizeNode()

and method 1 on BiLogic::predicateEliminatesNullAugmentedRows()

sql/generator/GenPreCode.cpp - use method 2 on

ItemExpr::replaceVEGExpressions()

sql/optimizer/ItemExpr.cpp - use method 2 on ItemExpr::unparsed()

AND ItemExpr::ItemExpr::synthTypeAndValueId()

sql/optimizer/OptRange.cpp - use method 3 on OptRangeSpec::buildRange()

sql/optimizer/BindItemExpr.cpp - use method 3 on

ItemExpr::convertToValueIdSet()

sql/optimizer/NormRelExpr.cpp - use method 3 on

Scan::applyAssociativityAndCommutativity()

sql/optimizer/ItemExpr.h - declare new methods that were created

sql/optimizer/ItemLog.h - declare new methods that were created

Finally, this check-in changes the default value for a CQD named

PCODE_MAX_OPT_BRANCH_CNT from 19000 to 12000. This was to fix a problem

where we used too much *heap* space when we tried to optimize a PCODE

Expression that had too many separate blocks of PCODE instructions (such

as results from a very long NOT IN list.) With this change, we will

choose to run with unoptimized PCODE if trying to optimize the PCODE

would result in overflowing the heap space available.

Change-Id: Ie8ddbab07de2a40095a80adac7873db8c5cb74ac

  1. … 11 more files in changeset.
Fix LP 1328250 - SQL Compiler hits "assert(FALSE)" in getIntConstValue

The SQL Compiler was abending and producing a core file in

getIntConstValue() when the Type of the integer constant was

PCIT::MBIGS.

This is code that is invoked only by the Native Expressions feature.

Currently, if the Native Expressions code encounters any PCODE

instruction that is designed for MBIGS or MBIGU type data, no native

expression is generated. Instead, we continue running with the PCODE

instructions as they are.

However, if PCODE optimization converts an MBIG[SU] type instruction

into a simpler instruction such as a 64-bit move instruction, then

the Native Expressions code goes ahead and tries to produce a native

expression. The problem is that, although PCODE optimization changes

to a simpler PCODE instruction, it does not change the data type of

any integer constant referenced by the instruction. The constant is

left as being an MBIG[SU] type even though the instruction's operands

say that the operand is a standard integer type.

The fix is to use the operand's idea of the data type rather than

the constant's idea of the data type whenever the constant is of

MBIGS or MBIGU type. This technique is already used by the code

when a Floating-Point type constant is referenced by a PCODE

instruction that says the operand is a simpler integer type.

Change-Id: I0e1aa28a51b8f5311ac51d0da5f385dc93bcd4eb

Add new PCode Expression Cache feature.

This new cache is maintained by the SQL Compiler. The purpose of this

cache is to avoid the fairly expensive logic involved in transforming

unoptimized PCode to optimized PCode and, where applicable, to also

avoid

the logic involved in transforming optimized PCode to a Native

Expression. This cache is accessed ONLY by the SQL Compiler code.

NOTES:

* This is second attempt to check in this code. First attempt had to

be abandoned as other developers made changes which prevented

automatic merging.

* This code has been pre-reviewed by Justin, Qifan, Selva, Mike,

Ravisha, Suresh, and Dave B. Many thanks to them for various

suggestions. Most of those suggestions have been incorporated into

this delivery. A few are left for future improvements.

* There is one instance of this new cache per CmpContext.

* There are 5 new CQDs used to control this cache. To be effective for

all instances of the cache, these need to be set in the system

defaults table. The CQD command given to sqlci will affect only the

instance of the cache for the current CmpContext.

The 5 CQDs are:

PCODE_EXPR_CACHE_ENABLED - set to 0 to disable the cache. Default is 1

PCODE_EXPR_CACHE_SIZE - max size in bytes. Default is 2,000,000.

PCODE_EXPR_CACHE_CMP_ONLY - Compare Only mode - useful to QA and

Development only.

PCODE_EXPR_CACHE_DEBUG - set to 1 to enable debug mode. Default is 0

PCODE_DEBUG_LOGDIR - pathname of existing directory where debug log

files will be placed -- one log file per cache

instance. Log files are designed to be easily

imported into an Excel Spreadsheet. No default.

* Also included are a small number of changes to the Native Expressions

feature to (a) Use the new PCODE_DEBUG_LOGDIR cqd to specify where to

put the Native Expressions debug log files, (b) measure cpu-time

rather than wall-clock time for measuring how long it took to produce

a Native Expression, and (c) add a CQD named PCODE_NE_ENABLED so we

can easily disable the Native Expressions feature [though there is

currently no known reason for doing so.]

Change-Id: I58f833f63099743ff6c1107acdff94fe8aef4b70

  1. … 14 more files in changeset.
SHOWPLAN related changes

Reworked the fix for bug 1392522 - mxosrvr core dumped doing showplan

(with N.E. enabled), plus other related code changes.

1) The native code (also known as native expression), if generated, is

stored in the expression's constant area. SHOWPLAN will dump the native

code in the assembly format. The display can be disabled by CQD

PCODE_NE_IN_SHOWPLAN to "OFF". It is "ON" by default. This part had been

reviewed by Jim Capps and Mike Hanlon.

2) Add several SHOWPLAN statements in core/TEST019 without logging the

output. This is to ensure no core generated when getting executor

operator (TDBs) info via SHOWPLAN.

3) A temporary fix to ComTdbHbaseAccess::displayRowId(). The current way

of retrieving begin or end row IDs from the HbaseScanRows for SHOWPLAN

does not match with the way those row IDs are generated (see

HbaseAccess::genListsOfRows), causing core dumps in some cases.

Change-Id: Id33e8c854452c03e15c4bd21f668f123180d9471

  1. … 9 more files in changeset.
Fix #1336979, #1342954, #1340400, #1342945, and #1315194

There are 6 things done by this change set:

* Uses a mutex to prevent use of the LLVM library code by

more than 1 thread at a time.

* Links libjsig.so into sqlci and tdm_arkcmp to use signal

chaining so that both Java and LLVM can have signal handlers

for SIGSEGV (and several other signals).

* Ensures that LLVM's signal handlers are in effect when LLVM is

being used and not in effect when LLVM is not in use. (One would

think that LLVM would ensure this, but it currently does not.)

* Enables the Native Expressions feature by default.

* #ifdef's out the code in SQLMXConnection.cpp which was disabling

the Native Expressions feature for the multithreading T2 driver.

* Changes from default JIT code generation (generating for the specific

x86 chip the SQL Compiler is running on) to generating code for

a generic x86-64 chip. This should prevent the 'invalid' instructions

which were seen on some of the Jenkins slave** machines.

Libjsig.so is part of the Java package that we are using.

The changes being made to the makefiles are to get libjsig.so

linked into the executables before Java and before LLVM.

A short description of libjsig is at:

http://docs.oracle.com/javase/7/docs/technotes/

guides/vm/signal-chaining.html

The way that LLVM currently handles various signals is a little

strange: The first time that you call the LLVM library, it saves off the

signal handlers for 11 different signals and establishes its own handler

for these signals. [Specifically, it does this when our code calls

llvm::verifyFunction().] The LLVM library code does not restore the

signal handlers before returning! (It will restore them only if its

signal handler ever gets called. Basically, LLVM believes that if any

of those 11 signals occurs while in LLVM code, then LLVM has basically

"crashed" and it wants to possibly delete some intermediate files and

essentially shut things down.)

Java, on the other hand, uses these signals for its own purposes -

particularly SIGSEGV. We do not want Java threads calling LLVM's

signal hander. So it is important to not leave LLVM's signal hander

established as the handler for these signals when the LLVM library is

not currently in use.

This has been pre-reviewed by Justin, Qifan, and Selva, except for:

* the 13 lines of changes to sql/exp/ExpPCodeOptsNativeExpr.cpp for

getting the JIT compiler to generate code for a generic x86-64 chip,

and

* the changes to conn/jdbc_type2/native/SQLMXConnection.cpp to re-enable

Native Expressions for the multi-threading T2 driver.

All dev regressions were run several times on a workstation and

several times via EC with no core files being produced.

Phoenix tests and multi-threaded T2 tests were also run several times.

Also, the Jenkins "check tests" (including Phoenix tests) were run

several times -- until the 'core' regression suite ran successfully

on slave07 (which is one of the machine where the 'invalid' machine

instruction was produced previously.)

Change-Id: I07931ccfac6bfa7cfd286a2a8a12140415cc91ac

  1. … 5 more files in changeset.
Fix bug 1336547 -arkesp aborting in Native Expr.

There was a place in the Native Expression code where we were picking up

either a 16-bit or a 32-bit value, assuming it was a positive number,

and using it as an offset into a buffer (in this case a string buffer).

The change is to use the 16-bit or 32-bit value as an unsigned offset

rather than signed.

Change-Id: I3eaac82c2580fe91179b7e50d4ada80bc1615cc0

modified: sql/exp/ExpPCodeOptsNativeExpr.cpp

Closes-Bug: 1328248 (INSERT ... SELECT query returned ERROR[8421] )

Error 8421 is reported when there is an attempt to store a NULL in

a NOT NULL column. In this case, the user's query had properly

qualified the SELECT part of the query so that NULLs should not

have been passed to the INSERT part of the query ... and yet the

error was being reported.

This problem occurred only when the Native Expression feature was

enabled. The fix is in the SQL Compiler code that generates native

code for an expression involving testing for NULLs.

The details of the fix and how/why it works are too complicated to put

into this commit message. Reviewers who would like the details

should send email to the author of this fix and I will send an MS Word

document, complete with color coding, that will explain what I found

in the debugging process and how/why the fix solves the problem.

Change-Id: Ib5b89bbb4362487807c782c6614eef22b542aa43

Initial code drop of Trafodion

    • -0
    • +12240
    ./ExpPCodeOptsNativeExpr.cpp
  1. … 4886 more files in changeset.