OracleAppsDBAKK1: Histograms and Bind Variables, But Why?

anuary 29, 2011 In a recent OTN thread a person asked a couple of good questions about why histograms and the use of bind variables sometimes cause problems. The questions did not ask whether or not one should use histograms on the table columns where those columns often appear in WHERE clauses, with the columns compared to bind variables. Instead, the original poster (OP) asked the all important question WHY. In a previous article I provided my response to an OTN thread where the OP of that thread wanted to use histograms to fix bind peeking problems. The specific questions asked in the recent OTN thread include: When a SQL is using bind variables how histograms affect the excution plan? Why histograms can’t work well with bind variables? I remember a document mentioned that “do not use histograms when using bind variables”. But why? The answers to these questions have been answered many times in articles written by a number of authors, for example: Bind Variable Peeking – Drives Me Nuts! Why do I have hundreds of child cursors when cursor_sharing set to similar in 10g Why are there more cursors in 11g for my query containing bind variables? Philosophy – 1 Adaptive Cursor Sharing Stabilize Oracle 10G’s Bind Peeking Behaviour by Cutting Histograms Rather than point the OP to one of the above articles, I decided instead to create a test case to demonstrate what could happen on Oracle Database 10.2.0.4 (simulated) and 11.2.0.2 when columns that are compared to bind variables in the WHERE clause also have histograms. Below is my response, slightly reworded: —- Histograms can work with bind variables, but the end result is typically not the desired outcome. Bind variables are used to reduce the number of different execution plans. Histograms are used to help the optimizer find what is supposed to be the best execution plan for the supplied predicates, and in the case of bind variables, those are the peeked values of the bind variables. So, if you have a histogram on a column, and for the initial hard parse of the SQL statement the most common value in that column is submitted in the bind variable – the generated execution plan is considered by the optimizer to be the “best” execution plan for the supplied bind variable values. Now assume that instead, the least popular value in the column is specified – the optimizer could produce a very different execution plan for the same SQL statement, one that is optimized for the least popular value (this might be an index range scan, rather than a full table scan). Assume that the execution plan cannot change when the bind variable values change during future executions – if the table column contains a single popular value and many unpopular values, if the initial hard parse is performed with the single popular value, you could find that all future executions of that SQL statement perform full table scans, even when only a couple of rows from the table are selected. Here is a quick test case on Oracle Database 11.2.0.2 to demonstrate: CREATE TABLE T1 ( C1 NUMBER, C2 NUMBER, C3 VARCHAR2(300)); INSERT INTO T1 SELECT * FROM (SELECT ROWNUM C1, DECODE(MOD(ROWNUM,100),99,99,1) C2, RPAD('A',300,'A') C3 FROM DUAL CONNECT BY LEVEL <= 1000000) ORDER BY C2; CREATE INDEX IND_T1_C2 ON T1(C2); EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T1',CASCADE=>TRUE,METHOD_OPT=>'FOR ALL INDEXED COLUMNS SIZE 254') The above created a table with 1,000,000 rows where 99% of the rows have a value of 1 in column C2 and 1% have a value of 99, and the rows are inserted with a perfect clustering factor due to the ORDER BY clause. A histogram was created on the indexed column. Let’s try a test, we will pick an unpopular value of 2 for the bind variable when the query is initially hard parsed: VARIABLE N1 NUMBER EXEC :N1:=2 SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1; no rows selected SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); SQL_ID c7su63uw7nch6, child number 0 ------------------------------------- SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1 Plan hash value: 236868917 ------------------------------------------------------------------------------------------------------------ | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 3 | 1 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 5957 | 0 |00:00:00.01 | 3 | 1 | |* 2 | INDEX RANGE SCAN | IND_T1_C2 | 1 | 5957 | 0 |00:00:00.01 | 3 | 1 | ------------------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C2"=:N1) So, there were no rows selected, the optimizer predicted that 5,957 rows would be returned, and an index access path was selected for data retrieval. Would this index access path also be appropriate for a bind variable value of 1? Let’s continue the test, this time picking the value 99 for the bind variable: EXEC :N1:=99 SET TIMING ON SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1; ... 10000 rows selected. Elapsed: 00:00:05.35 SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); SQL_ID c7su63uw7nch6, child number 0 ------------------------------------- SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1 Plan hash value: 236868917 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 10000 |00:00:00.02 | 1783 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 5957 | 10000 |00:00:00.02 | 1783 | |* 2 | INDEX RANGE SCAN | IND_T1_C2 | 1 | 5957 | 10000 |00:00:00.01 | 690 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C2"=:N1) Once again, the execution plan shows that the optimizer predicted 5,957 rows would be retrieved even though 10,000 rows were actually retrieved. Notice also that the child number is still shown as 0, indicating that a hard parse was not performed. Let’s continue the test, this time with a bind variable value of 1: EXEC :N1:=1 SET AUTOTRACE TRACEONLY STATISTICS SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1; 990000 rows selected. Elapsed: 00:00:18.78 Statistics --------------------------------------------------- 1 recursive calls 1 db block gets 108571 consistent gets 0 physical reads 96 redo size 21958348 bytes sent via SQL*Net to client 726508 bytes received via SQL*Net from client 66001 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 990000 rows processed SET AUTOTRACE OFF Because I used AUTOTRACE to prevent the 990,000 rows from scrolling on screen, I have to specify the SQL_ID and CHILD_NUMBER to retrieve the execution plan: SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR('c7su63uw7nch6',0,'ALLSTATS LAST')); SQL_ID c7su63uw7nch6, child number 0 ------------------------------------- SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1 Plan hash value: 236868917 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 10000 |00:00:00.02 | 1783 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 5957 | 10000 |00:00:00.02 | 1783 | |* 2 | INDEX RANGE SCAN | IND_T1_C2 | 1 | 5957 | 10000 |00:00:00.01 | 690 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C2"=:N1) That cannot be the execution plan that was used because it still shows that 10,000 rows were retrieved during the last execution, where the AUTOTRACE statistics showed that 990,000 rows were actually retrieved. Let’s try again, this time retrieving the execution plan for CHILD_NUMBER 1: SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR('c7su63uw7nch6',1,'ALLSTATS LAST')); SQL_ID c7su63uw7nch6, child number 1 ------------------------------------- SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1 Plan hash value: 3617692013 ------------------------------------------------------------------------------------ | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | ------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 1 | | 990K|00:00:00.83 | 108K| |* 1 | TABLE ACCESS FULL| T1 | 1 | 988K| 990K|00:00:00.83 | 108K| ------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter("C2"=:N1) The above shows the actual execution plan that was used (sse the article Explain Plan Lies, Autotrace Lies, TKPROF Lies, What is the Plan? to see why we cannot use AUTOTRACE or EXPLAIN PLAN to see the actual execution plan). Adaptive cursor sharing (first available with Oracle Database 11.1) stepped in and forced the re-evaluation of the execution plan to prevent a very slow retrieval through the index – that re-evaluation will not happen prior to Oracle Database 11.1 (CURSOR_SHARING=’SIMILAR’ might have the same effect in older Oracle Database releases when literal values are used in the SQL statement). Just to demonstrate: ALTER SESSION SET OPTIMIZER_FEATURES_ENABLE='10.2.0.4'; VARIABLE N1 NUMBER EXEC :N1:=2 SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1; no rows selected Elapsed: 00:00:00.00 SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); SQL_ID c7su63uw7nch6, child number 2 ------------------------------------- SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1 Plan hash value: 236868917 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 3 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 5957 | 0 |00:00:00.01 | 3 | |* 2 | INDEX RANGE SCAN | IND_T1_C2 | 1 | 5957 | 0 |00:00:00.01 | 3 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C2"=:N1) Note in the above that the CHILD_NUMBER is now 2 because we changed the optimizer’s execution environment (see the articles How to Determine which First Rows OPTIMIZER_MODE was Specified, SELECT Statement is Fast, INSERT INTO Using the SELECT Statement is Brutally Slow 3, Reviewing Session-Level Parameters to better understand what might trigger a change in the optimizer’s execution environment). Continuing: EXEC :N1:=99 SET TIMING ON SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1; 10000 rows selected. Elapsed: 00:00:05.31 SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); SQL_ID c7su63uw7nch6, child number 2 ------------------------------------- SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1 Plan hash value: 236868917 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 10000 |00:00:00.02 | 1783 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 5957 | 10000 |00:00:00.02 | 1783 | |* 2 | INDEX RANGE SCAN | IND_T1_C2 | 1 | 5957 | 10000 |00:00:00.01 | 690 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C2"=:N1) The CHILD_NUMBER is still 2, so there was no hard parse. Continuing: EXEC :N1:=1 SET AUTOTRACE TRACEONLY STATISTICS SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1; 990000 rows selected. Elapsed: 00:00:16.91 Statistics --------------------------------------------------- 0 recursive calls 0 db block gets 175927 consistent gets 0 physical reads 0 redo size 21958348 bytes sent via SQL*Net to client 726508 bytes received via SQL*Net from client 66001 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 990000 rows processed SET AUTOTRACE OFF SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR('c7su63uw7nch6',2,'ALLSTATS LAST')); SQL_ID c7su63uw7nch6, child number 2 ------------------------------------- SELECT /*+ GATHER_PLAN_STATISTICS */ C1, C2 FROM T1 WHERE C2 = :N1 Plan hash value: 236868917 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 990K|00:00:01.63 | 175K| | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 5957 | 990K|00:00:01.63 | 175K| |* 2 | INDEX RANGE SCAN | IND_T1_C2 | 1 | 5957 | 990K|00:00:00.68 | 67932 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C2"=:N1) The above is the execution plan for CHILD_NUMBER 2 – notice that this time it is reporting 990,000 rows retrieved, so this IS the execution plan that was used for the bind variable value that exists in 99% of the table rows. Adaptive cursor sharing did not take effect and force the re-evaluation of the execution plan – the execution plan was NOT changed to a full table scan. That is the risk that you take if you allow histograms to exist on columns that have unequal distributions of values, bind variables are used in the WHERE clause that references the column, and bind variable peeking is enabled (enabled by default in Oracle Database 9i and above, bind variable peeking is controlled by the hidden parameter _OPTIM_PEEK_USER_BINDS, which defaults to TRUE). Comments : 6 Comments » Categories : Bind Variable, Histogram 10046 Extended SQL Trace Interpretation 3 6 09 2010 September 6, 2010 (Back to the Previous Post in the Series) In previous blog articles in this series we looked at various methods for starting a 10046 trace file and also the basics of walking through a trace file. Today’s blog article continues the investigation of 10046 trace files. To begin, we need a test script: CREATE TABLE T3_1 AS SELECT ROWNUM C1, LPAD('A',100,'A') C2 FROM DUAL CONNECT BY LEVEL<=10000; CREATE TABLE T4_1 AS SELECT ROWNUM C1, LPAD('A',100,'A') C2 FROM DUAL CONNECT BY LEVEL<=10000; CREATE INDEX IND_T4_1 ON T4_1(C1); EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T3_1',CASCADE=>TRUE) EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T4_1',CASCADE=>TRUE) ALTER SYSTEM FLUSH BUFFER_CACHE; ALTER SYSTEM FLUSH BUFFER_CACHE; SET ARRAYSIZE 15 SET AUTOTRACE TRACEONLY STATISTICS VARIABLE N1 NUMBER VARIABLE N2 NUMBER EXEC :N1 := 1 EXEC :N2 := 2 ALTER SESSION SET TRACEFILE_IDENTIFIER = '10046TraceTest'; EXEC DBMS_SESSION.SESSION_TRACE_ENABLE(WAITS=>TRUE, BINDS=>TRUE, PLAN_STAT=>'ALL_EXECUTIONS') SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1; EXEC :N2 := 10000 SET ARRAYSIZE 500 SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1; SELECT SYSDATE FROM DUAL; EXEC DBMS_SESSION.SESSION_TRACE_DISABLE; EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T3_1',CASCADE=>TRUE) EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T4_1',CASCADE=>TRUE) ALTER SYSTEM FLUSH BUFFER_CACHE; ALTER SYSTEM FLUSH BUFFER_CACHE; SET ARRAYSIZE 15 SET AUTOTRACE TRACEONLY STATISTICS VARIABLE N1 NUMBER VARIABLE N2 NUMBER EXEC :N1 := 1 EXEC :N2 := 2 ALTER SESSION SET TRACEFILE_IDENTIFIER = '10046TraceTest2'; EXEC DBMS_SESSION.SESSION_TRACE_ENABLE(WAITS=>TRUE, BINDS=>TRUE, PLAN_STAT=>'ALL_EXECUTIONS') SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1; EXEC :N2 := 10000 SET ARRAYSIZE 500 SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1; SELECT SYSDATE FROM DUAL; EXEC DBMS_SESSION.SESSION_TRACE_DISABLE; The above test script: Creates two tables. Creates an index is created on the second table. Table and index statistics are collected (the index statistics would already exist on Oracle Database 10.1 and above, as the statistics are calculated automatically during index creation). The buffer cache is flushed twice to force physical reads when the test SQL statements are performed. Two bind variables are defined. The trace file to be created is named. 10046 tracing at level 12 is enabled for the session, specifying that the execution plan should be written after each execution of the SQL statement, rather when that cursor is closed (the PLAN_STAT parameter is only valid in Oracle Database 11.1 and above). A SQL statement using the bind variables is executed with the array fetch size set at the SQL*Plus default of 15. The value of the second bind variable is increased from 2 (a very small number) to 10,000 (a very large number for the tables involved). The array fetch size is increased from the default value of 15 to a value of 500. The SQL statement is executed again with the new bind variable values. A simple SQL statement is executed to force the closing of the previous SQL statement and the printing of the execution plan to the trace file (this is needed prior to Oracle Database 11.1, or when the PLAN_STAT parameter is not set to the default value of ALL_EXECUTIONS – note that the SQL statement being added to the session cursor cache after three executions may prevent the execution plan from being written to the trace file unless the PLAN_STAT parameter is set to ALL_EXECUTIONS). The test is repeated a second time with a different name specified for the trace file. The above script was executed on Oracle Database 11.2.0.1, on a platform that only supports direct, asynchronous I/O (equivalent to setting the FILESYSTEMIO_OPTIONS initialization parameter to SETALL). The direct, asynchronous I/O configuration prevents an operating system level file cache from speeding up the access to the physical data files (the OS would be able to completely avoid accessing the disks if an O.S. level file cache were used). Let’s start by looking at the statistics that were output by AUTOTRACE: N1 = 1, N2 = 2, first execution: Statistics -------------------------------------------- 1 recursive calls 0 db block gets 163 consistent gets 164 physical reads 0 redo size 840 bytes sent via SQL*Net to client 520 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 2 rows processed Interesting, the number of physical blocks read from disk (164) exceeds the number of consistent gets (163) plus the current mode gets (db block gets = 0). No redo was generated (redo generated on a SELECT might be caused by delayed block cleanout). 840 bytes were sent back to the client, and the server received 520 bytes from the client. Two round trips were performed in order to retrieve two rows from the database (even though the array fetch size was set to 15). There was also a single recursive call, probably during the hard parse of the SQL statement. — N1 = 1, N2 = 10,000, first execution: Statistics ---------------------------------------------- 0 recursive calls 0 db block gets 374 consistent gets 188 physical reads 0 redo size 94102 bytes sent via SQL*Net to client 729 bytes received via SQL*Net from client 21 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 10000 rows processed The number of recursive calls dropped from one to zero (probably because a hard parse was not required). The number of consistent gets (blocks) is now roughly double the number of physical reads (blocks). 94,102 bytes were sent back to the client, and the server received 729 bytes from the client. Twenty-one round trips were performed to retrieve the 10,000 rows, and there are still no sorts in memory or on disk. — N1 = 1, N2 = 2, second execution: Statistics -------------------------------------------- 1 recursive calls 0 db block gets 163 consistent gets 171 physical reads 0 redo size 840 bytes sent via SQL*Net to client 520 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 2 rows processed On the second execution with the N2 bind variable reset to a value of 2 we see another recursive call (maybe another hard parse?). Again, the number of physical blocks read from disk (171 – seven more than earlier) exceeds the number of consistent gets (163 – the same as earlier) plus the number of current mode gets (db block gets = 0). — N1 = 1, N2 = 10,000, second execution: Statistics ---------------------------------------------- 1 recursive calls 0 db block gets 336 consistent gets 150 physical reads 0 redo size 94102 bytes sent via SQL*Net to client 729 bytes received via SQL*Net from client 21 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 10000 rows processed On the second execution with the N2 bind variable reset to a value of 10,000 we see another recursive call (maybe another hard parse?). The number of consistent gets (336 – thirty-eight less than earlier) is a bit more than twice the number of physical block reads (150 – again thirty-eight less than earlier, and now less than what was required when N2 was set to a value of 2). What happened, is Oracle Database 11.2.0.1 toying with our performance tuning ability? Maybe we should look at the TKPROF output (note that this is output from 11.2.0.1 TKPROF): tkprof or112_ora_384_10046TraceTest.trc or112_ora_384_10046TraceTest.txt tkprof or112_ora_384_10046TraceTest2.trc or112_ora_384_10046TraceTest2.txt TKPROF N1 = 1, N2 = 2 and N2 = 10,000, first execution: SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 2 0.00 0.00 0 0 0 0 Execute 2 0.00 0.00 0 0 0 0 Fetch 23 0.01 0.20 352 537 0 10002 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 27 0.01 0.20 352 537 0 10002 Misses in library cache during parse: 1 Misses in library cache during execute: 1 Optimizer mode: ALL_ROWS Parsing user id: 90 Rows Row Source Operation ------- --------------------------------------------------- 2 FILTER (cr=163 pr=164 pw=0 time=0 us) 2 HASH JOIN (cr=163 pr=164 pw=0 time=0 us cost=51 size=109 card=1) 2 TABLE ACCESS FULL T3_1 (cr=158 pr=155 pw=0 time=0 us cost=47 size=8 card=2) 2 TABLE ACCESS BY INDEX ROWID T4_1 (cr=5 pr=9 pw=0 time=0 us cost=3 size=210 card=2) 2 INDEX RANGE SCAN IND_T4_1 (cr=3 pr=8 pw=0 time=4 us cost=2 size=0 card=2)(object id 83176) Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ SQL*Net message to client 23 0.00 0.00 db file sequential read 2 0.00 0.00 db file scattered read 42 0.02 0.18 SQL*Net message from client 23 0.03 0.05 Assuming that TKPROF from the Oracle 11.1.0.6 client was not used, the above indicates that the same execution plan was used when C2 was set to 2 and also when it was set to 10,000. Because the execution plan was written after each execution, the execution plan provided by TKPROF is a little confusing when trying to cross-reference the statistics to see exactly what happened. The statistics show that 10,002 rows were retrieved in 0.20 seconds with 537 consistent gets and 352 blocks read from disk, while the execution plan shows that 2 rows were retrieved in 0.000000 seconds with 163 consistent gets and 164 blocks read from disk. The summarized wait events in this case cannot be correlated to the lines in the execution plan because TKPROF only printed the first execution plan found in the trace file for the query. TKPROF N1 = 1, N2 = 2 and N2 = 10,000, second execution: SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.00 0.00 171 163 0 2 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.00 0.00 171 163 0 2 Misses in library cache during parse: 0 Misses in library cache during execute: 1 Optimizer mode: ALL_ROWS Parsing user id: 90 Rows Row Source Operation ------- --------------------------------------------------- 2 FILTER (cr=163 pr=171 pw=0 time=0 us) 2 HASH JOIN (cr=163 pr=171 pw=0 time=0 us cost=51 size=109 card=1) 2 TABLE ACCESS FULL T3_1 (cr=158 pr=155 pw=0 time=1 us cost=47 size=8 card=2) 2 TABLE ACCESS BY INDEX ROWID T4_1 (cr=5 pr=16 pw=0 time=0 us cost=3 size=210 card=2) 2 INDEX RANGE SCAN IND_T4_1 (cr=3 pr=8 pw=0 time=4 us cost=2 size=0 card=2)(object id 83176) Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ SQL*Net message to client 2 0.00 0.00 db file sequential read 1 0.00 0.00 db file scattered read 19 0.00 0.00 SQL*Net message from client 2 0.00 0.00 ******************************************************************************** SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 21 0.03 0.01 150 336 0 10000 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 23 0.03 0.01 150 336 0 10000 Misses in library cache during parse: 0 Misses in library cache during execute: 1 Optimizer mode: ALL_ROWS Parsing user id: 90 Rows Row Source Operation ------- --------------------------------------------------- 10000 FILTER (cr=336 pr=150 pw=0 time=4999 us) 10000 HASH JOIN (cr=336 pr=150 pw=0 time=4486 us cost=95 size=1090000 card=10000) 10000 TABLE ACCESS FULL T3_1 (cr=158 pr=0 pw=0 time=1265 us cost=47 size=40000 card=10000) 10000 TABLE ACCESS FULL T4_1 (cr=178 pr=150 pw=0 time=1410 us cost=47 size=1050000 card=10000) Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ SQL*Net message to client 21 0.00 0.00 db file sequential read 1 0.00 0.00 SQL*Net message from client 21 0.00 0.01 db file scattered read 16 0.00 0.00 Because we used the 11.2.0.1 client, TKPROF clearly shows that the execution plan changed during the second execution when the value of N2 changed from 2 to 10,000. With the two execution plans and a single execution with each value of N2 we are able to cross-reference the various statistics. With N2 at a value of 2, we see that: 163 blocks were obtained from memory using consist gets, 3 of which belonged to the IND_T4_1 index, 2 of which belonged to the T4_1 table, and 158 of which belonged to the T3_1 table. 171 blocks were read from disk, 8 of which belonged to the IND_T4_1 index, 8 of which belonged to the T4_1 table, and 155 of which belonged to the T3_1 table. There was only 1 single block read, with 19 of the 20 physical read events performed using multi-block reads with an average of 8.95 blocks read per multi-block read [(171 – 1)/19]. The total physical read time per the wait events is 0.00 seconds. The elapsed time and CPU time per the statistics summary is 0.00 seconds. With N2 at a value of 10,000, we see that: 336 blocks were obtained from memory using consist gets, 0 of which belonged to the IND_T4_1 index, 178 of which belonged to the T4_1 table, and 158 of which belonged to the T3_1 table. 150 blocks were read from disk, 0 of which belonged to the IND_T4_1 index, 150 of which belonged to the T4_1 table, and 0 of which belonged to the T3_1 table. There was only 1 single block read, with 16 of the 17 physical read events performed using multi-block reads with an average of 9.31 blocks read per multi-block read [(150 – 1) / 16]. The total physical read time per the wait events is 0.00 seconds. The elapsed time and CPU time per the statistics summary is 0.01 seconds and 0.03 seconds, respectively. The TKPROF summaries are nice, but there is still a missing element – detail. Before we dive into the raw 10046 trace, let’s find the OBJECT_ID and DATA_OBJECT_ID for the various objects in our test: SELECT OBJECT_NAME, OBJECT_ID, DATA_OBJECT_ID FROM DBA_OBJECTS WHERE OBJECT_NAME IN ('T3_1','T4_1','IND_T4_1'); OBJECT_NAME OBJECT_ID DATA_OBJECT_ID ----------- ---------- -------------- T3_1 83174 83174 T4_1 83175 83175 IND_T4_1 83176 83176 As we saw in the execution plan earlier, 83176 represents the IND_T4_1 index. 83174 represents table T3_1, and 83175 represents table T4_1. The objects were created in an ASSM autoallocate tablespace. Now the raw 10046 trace files. First execution 10046 trace file, with comments: ===================== PARSING IN CURSOR #15 len=101 dep=0 uid=90 oct=3 lid=90 tim=774686889 hv=3458052021 ad='36ce14150' sqlid='4122a6m71vbxp' SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 END OF STMT PARSE #15:c=0,e=122,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=0,tim=774686889 BINDS #15: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=48 off=0 kxsbbbfp=1e8e0aa8 bln=22 avl=02 flg=05 value=1 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24 kxsbbbfp=1e8e0ac0 bln=22 avl=02 flg=01 value=2 From the above, the parse call required 0.000122 seconds to perform a hard parse using an optimizer goal of 1 (ALL_ROWS), and two numeric bind variables were provided. The SQL_ID and HASH_VALUE are also displayed (these may be used to retrieve a DBMS_XPLAN formatted version of the execution plan). Continuing with the 10046 trace file: EXEC #15:c=0,e=705,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=39991016,tim=774687655 WAIT #15: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=0 tim=774687683 WAIT #15: nam='db file sequential read' ela= 93 file#=7 block#=1819466 blocks=1 obj#=83174 tim=774687866 WAIT #15: nam='db file scattered read' ela= 25845 file#=7 block#=1819467 blocks=5 obj#=83174 tim=774713774 WAIT #15: nam='db file scattered read' ela= 1443 file#=7 block#=1819472 blocks=8 obj#=83174 tim=774715430 WAIT #15: nam='db file scattered read' ela= 13640 file#=7 block#=1819481 blocks=7 obj#=83174 tim=774729168 WAIT #15: nam='db file scattered read' ela= 1788 file#=7 block#=1819488 blocks=8 obj#=83174 tim=774731067 WAIT #15: nam='db file scattered read' ela= 1372 file#=7 block#=1819497 blocks=7 obj#=83174 tim=774732504 WAIT #15: nam='db file scattered read' ela= 926 file#=7 block#=1819504 blocks=8 obj#=83174 tim=774733534 WAIT #15: nam='db file scattered read' ela= 685 file#=7 block#=1819513 blocks=7 obj#=83174 tim=774734310 WAIT #15: nam='db file scattered read' ela= 25768 file#=8 block#=214144 blocks=8 obj#=83174 tim=774760171 WAIT #15: nam='db file scattered read' ela= 1079 file#=8 block#=214153 blocks=7 obj#=83174 tim=774761358 WAIT #15: nam='db file scattered read' ela= 1044 file#=8 block#=214160 blocks=8 obj#=83174 tim=774762502 WAIT #15: nam='db file scattered read' ela= 896 file#=8 block#=214169 blocks=7 obj#=83174 tim=774763509 WAIT #15: nam='db file scattered read' ela= 1184 file#=8 block#=214176 blocks=8 obj#=83174 tim=774764754 WAIT #15: nam='db file scattered read' ela= 1816 file#=8 block#=214185 blocks=7 obj#=83174 tim=774766671 WAIT #15: nam='db file scattered read' ela= 770 file#=8 block#=214192 blocks=8 obj#=83174 tim=774767541 WAIT #15: nam='db file scattered read' ela= 872 file#=8 block#=214201 blocks=7 obj#=83174 tim=774768531 WAIT #15: nam='db file scattered read' ela= 899 file#=8 block#=214208 blocks=8 obj#=83174 tim=774769526 WAIT #15: nam='db file scattered read' ela= 2080 file#=7 block#=1819522 blocks=36 obj#=83174 tim=774771767 WAIT #15: nam='db file scattered read' ela= 24065 file#=7 block#=1819720 blocks=8 obj#=83176 tim=774796086 WAIT #15: nam='db file sequential read' ela= 303 file#=8 block#=214219 blocks=1 obj#=83175 tim=774796473 FETCH #15:c=0,e=108849,p=164,cr=161,cu=0,mis=0,r=1,dep=0,og=1,plh=39991016,tim=774796549 From the above, we see that the first fetch returned only a single row, and that fetch required 0.108849 seconds, reading 164 blocks from disk, with 161 consistent gets. The one single block read at the start was for object 83174 (table T3_1), while the one single block read just before the fetch was for object 83175 (table T4_1). Object 83176 (index IND_T4_1) was read using a multi-block read – not all index accesses are necessarily single block reads, although a fast full index scan will usually use multi-block reads, the execution plan shows a range scan. What was the purpose of the two single block reads of the tables? SELECT SEGMENT_NAME, HEADER_FILE, HEADER_BLOCK FROM DBA_SEGMENTS WHERE SEGMENT_NAME IN ('T3_1','T4_1'); SEGMENT_NAME HEADER_FILE HEADER_BLOCK ------------ ----------- ------------ T3_1 7 1819466 T4_1 8 214218 Comparing those numbers with the single block reads, we see that the segment header block of table T3_1 was read, and the block immediately after the segment header block of table T4_1 was read. Notice that there is quite a bit of variation in the block read access times, ranging from 0.000093 seconds to 0.000303 seconds for a single block read and 0.000685 seconds (0.7ms) to 0.025845 seconds (25.8ms) for the multi-block reads. While these physical read times were achieved using typical hard drives without the benefit of an operating system cache or RAID controller cache, or even a SAN cache, for the most part the times are atypically low for “spinning rust” (for example, a 15,000 RPM hard drive requires 4ms for a single rotation, the average seek time should be at least 1/2 of the single rotation time) – cache memory built into the physical drives is likely the cause of the atypically low average read times. We see that for the most part, the multi-block reads are either 7 or 8 blocks, with a single 36 block multi-block read (the autoallocate extent size for the final extent in table T3_1 increased from the previous extent’s 64KB to 1MB, allowing for a multi-block read larger than 64KB). The PLAN_HASH_VALUE 39991016 (visible in V$SQL) was also written to the trace file. Continuing with the 10046 trace file: WAIT #15: nam='SQL*Net message from client' ela= 36739 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774833320 WAIT #15: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774833369 FETCH #15:c=0,e=48,p=0,cr=2,cu=0,mis=0,r=1,dep=0,og=1,plh=39991016,tim=774833406 STAT #15 id=1 cnt=2 pid=0 pos=1 obj=0 op='FILTER (cr=163 pr=164 pw=0 time=0 us)' STAT #15 id=2 cnt=2 pid=1 pos=1 obj=0 op='HASH JOIN (cr=163 pr=164 pw=0 time=0 us cost=51 size=109 card=1)' STAT #15 id=3 cnt=2 pid=2 pos=1 obj=83174 op='TABLE ACCESS FULL T3_1 (cr=158 pr=155 pw=0 time=0 us cost=47 size=8 card=2)' STAT #15 id=4 cnt=2 pid=2 pos=2 obj=83175 op='TABLE ACCESS BY INDEX ROWID T4_1 (cr=5 pr=9 pw=0 time=0 us cost=3 size=210 card=2)' STAT #15 id=5 cnt=2 pid=4 pos=1 obj=83176 op='INDEX RANGE SCAN IND_T4_1 (cr=3 pr=8 pw=0 time=4 us cost=2 size=0 card=2)' WAIT #15: nam='SQL*Net message from client' ela= 455 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774833972 *** SESSION ID:(1232.2) 2010-09-06 10:17:12.191 CLOSE #15:c=0,e=7,dep=0,type=0,tim=774861965 ===================== From the above, we see that SQL*Plus required roughly 0.036739 seconds to process the first row that it received from the database and to request the next 15 rows. The second fetch call, which also returned a single row, required just 0.000048 seconds and required 2 consistent gets. If we subtract the tim= value found on the first FETCH call from the tim= value found on the second fetch call, we see that 0.036857 seconds actually elapsed, so 0.000118-0.000048 seconds = 0.000070 seconds of excution time that essentially disappeared – part of the unaccounted for time for the trace file. Continuing with the 10046 trace file: PARSING IN CURSOR #16 len=26 dep=0 uid=90 oct=47 lid=90 tim=774862238 hv=3000858490 ad='36ce13ac0' sqlid='56ndayftduxvu' BEGIN :N2 := 10000; END; END OF STMT PARSE #16:c=0,e=223,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=0,tim=774862237 BINDS #16: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=24 off=0 kxsbbbfp=1ddb9628 bln=22 avl=02 flg=05 value=2 WAIT #16: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774862876 EXEC #16:c=0,e=611,p=0,cr=0,cu=0,mis=1,r=1,dep=0,og=1,plh=0,tim=774862890 WAIT #16: nam='SQL*Net message from client' ela= 5694 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774868616 *** SESSION ID:(1232.2) 2010-09-06 10:17:12.206 CLOSE #16:c=0,e=11,dep=0,type=0,tim=774869382 ===================== In the above, SQL*Plus changed the bind variable value from 2 to 10,000 (interesting that the old value seems to be passed in as a bind variable). Continuing with the 10046 trace file: PARSING IN CURSOR #17 len=101 dep=0 uid=90 oct=3 lid=90 tim=774869440 hv=3458052021 ad='36ce14150' sqlid='4122a6m71vbxp' SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 END OF STMT PARSE #17:c=0,e=27,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=39991016,tim=774869439 BINDS #17: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=48 off=0 kxsbbbfp=1e8e0aa8 bln=22 avl=02 flg=05 value=1 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24 kxsbbbfp=1e8e0ac0 bln=22 avl=02 flg=01 value=10000 EXEC #17:c=0,e=74,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=39991016,tim=774869572 WAIT #17: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774869596 FETCH #17:c=0,e=3430,p=0,cr=161,cu=0,mis=0,r=1,dep=0,og=1,plh=39991016,tim=774873041 WAIT #17: nam='SQL*Net message from client' ela= 133 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774873205 WAIT #17: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774873238 WAIT #17: nam='db file scattered read' ela= 427 file#=8 block#=214220 blocks=4 obj#=83175 tim=774873768 WAIT #17: nam='db file scattered read' ela= 1251 file#=8 block#=214224 blocks=8 obj#=83175 tim=774875328 FETCH #17:c=0,e=2314,p=12,cr=10,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774875542 Remember here that we did not flush the buffer cache in between the execution with N2 set to 2, and when it was set to 10,000. Notice that the first fetch of the first row this time completed in 0.003430 seconds (a decrease from 0.036739 seconds) and did not require any physical reads. The second fetch call, which retrieved 500 rows, performed a multi-block read of 4 blocks, and then a multi-block read of 8 blocks. Why 4 blocks and not 8 (or 128 to permit a 1MB multi-block read) for the first read request? Block 214219 was already in the buffer cache (see the trace file section from the previous output), and since Oracle will not perform a physical read of a block for the buffer cache that is already in the buffer cache, the multi-block read of the 64KB extent could only read 3 or 4 blocks (214216 through 214218 or 214220 through 214223) before hitting an extent boundry. The first fetch call performed 161 consistent gets with no physical reads, while the second fetch call performed 10 consistent gets with 12 physical block reads (of table T4_1) – I still wonder why the number of consistent gets is less than the number of physical reads in this one fetch call (likely a difference in the array fetch size and the number of rows returned by the multi-block read size). Continuing with the 10046 trace file: WAIT #17: nam='SQL*Net message from client' ela= 679 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774876242 WAIT #17: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774876273 WAIT #17: nam='db file scattered read' ela= 735 file#=8 block#=214232 blocks=8 obj#=83175 tim=774877386 FETCH #17:c=0,e=1319,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774877583 WAIT #17: nam='SQL*Net message from client' ela= 665 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774878269 WAIT #17: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774878299 WAIT #17: nam='db file scattered read' ela= 847 file#=8 block#=214240 blocks=8 obj#=83175 tim=774879444 FETCH #17:c=0,e=1387,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774879677 WAIT #17: nam='SQL*Net message from client' ela= 660 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774880358 WAIT #17: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774880388 WAIT #17: nam='db file scattered read' ela= 727 file#=8 block#=214248 blocks=8 obj#=83175 tim=774881432 WAIT #17: nam='db file scattered read' ela= 1424 file#=7 block#=1819728 blocks=8 obj#=83176 tim=774883006 FETCH #17:c=0,e=2768,p=16,cr=10,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774883147 WAIT #17: nam='SQL*Net message from client' ela= 709 driver id=1413697536 #bytes=1 p3=0 obj#=83176 tim=774883879 WAIT #17: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83176 tim=774883917 WAIT #17: nam='db file scattered read' ela= 15955 file#=8 block#=214256 blocks=8 obj#=83175 tim=774900156 FETCH #17:c=0,e=16555,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774900460 WAIT #17: nam='SQL*Net message from client' ela= 942 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774901423 WAIT #17: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774901479 WAIT #17: nam='db file scattered read' ela= 1119 file#=8 block#=214264 blocks=8 obj#=83175 tim=774902934 FETCH #17:c=0,e=1743,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774903205 WAIT #17: nam='SQL*Net message from client' ela= 751 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774903977 WAIT #17: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774904012 WAIT #17: nam='db file scattered read' ela= 14243 file#=7 block#=1819648 blocks=8 obj#=83175 tim=774918513 FETCH #17:c=0,e=14844,p=8,cr=10,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774918846 WAIT #17: nam='SQL*Net message from client' ela= 948 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774919817 WAIT #17: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774919873 WAIT #17: nam='db file scattered read' ela= 854 file#=7 block#=1819656 blocks=8 obj#=83175 tim=774921056 FETCH #17:c=0,e=1500,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774921356 WAIT #17: nam='SQL*Net message from client' ela= 930 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774922307 WAIT #17: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774922364 WAIT #17: nam='db file scattered read' ela= 1288 file#=7 block#=1819664 blocks=8 obj#=83175 tim=774923937 FETCH #17:c=0,e=1944,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774924291 WAIT #17: nam='SQL*Net message from client' ela= 895 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774925206 WAIT #17: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774925262 WAIT #17: nam='db file scattered read' ela= 929 file#=7 block#=1819672 blocks=8 obj#=83175 tim=774926493 FETCH #17:c=0,e=1564,p=8,cr=10,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774926810 WAIT #17: nam='SQL*Net message from client' ela= 877 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774927707 WAIT #17: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774927762 WAIT #17: nam='db file scattered read' ela= 946 file#=7 block#=1819680 blocks=8 obj#=83175 tim=774928969 FETCH #17:c=15600,e=1639,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774929385 WAIT #17: nam='SQL*Net message from client' ela= 888 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774930316 WAIT #17: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774930371 WAIT #17: nam='db file scattered read' ela= 3361 file#=7 block#=1819688 blocks=8 obj#=83175 tim=774934020 WAIT #17: nam='db file scattered read' ela= 1173 file#=7 block#=1819736 blocks=8 obj#=83176 tim=774935320 FETCH #17:c=0,e=5280,p=16,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774935634 WAIT #17: nam='SQL*Net message from client' ela= 665 driver id=1413697536 #bytes=1 p3=0 obj#=83176 tim=774936320 WAIT #17: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83176 tim=774936355 WAIT #17: nam='db file scattered read' ela= 979 file#=7 block#=1819696 blocks=8 obj#=83175 tim=774937529 FETCH #17:c=0,e=1538,p=8,cr=10,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774937883 WAIT #17: nam='SQL*Net message from client' ela= 665 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774938570 WAIT #17: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774938605 WAIT #17: nam='db file scattered read' ela= 742 file#=7 block#=1819704 blocks=8 obj#=83175 tim=774939560 FETCH #17:c=0,e=1289,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774939884 WAIT #17: nam='SQL*Net message from client' ela= 663 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774940568 WAIT #17: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774940602 WAIT #17: nam='db file scattered read' ela= 2105 file#=7 block#=1819712 blocks=8 obj#=83175 tim=774942878 FETCH #17:c=0,e=2643,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774943235 WAIT #17: nam='SQL*Net message from client' ela= 878 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774944134 WAIT #17: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774944189 WAIT #17: nam='db file scattered read' ela= 950 file#=8 block#=214272 blocks=8 obj#=83175 tim=774945372 FETCH #17:c=0,e=1581,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774945754 WAIT #17: nam='SQL*Net message from client' ela= 876 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774946651 WAIT #17: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774946707 WAIT #17: nam='db file scattered read' ela= 4210 file#=8 block#=214280 blocks=8 obj#=83175 tim=774951052 FETCH #17:c=0,e=4854,p=8,cr=10,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774951544 WAIT #17: nam='SQL*Net message from client' ela= 662 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774952228 WAIT #17: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774952262 WAIT #17: nam='db file scattered read' ela= 2160 file#=8 block#=214288 blocks=8 obj#=83175 tim=774954531 FETCH #17:c=0,e=2727,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774954979 WAIT #17: nam='SQL*Net message from client' ela= 870 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774955869 WAIT #17: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774955924 WAIT #17: nam='db file scattered read' ela= 1331 file#=8 block#=214296 blocks=8 obj#=83175 tim=774957425 WAIT #17: nam='db file scattered read' ela= 924 file#=7 block#=1819744 blocks=8 obj#=83176 tim=774958429 FETCH #17:c=0,e=2950,p=16,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=774958857 WAIT #17: nam='SQL*Net message from client' ela= 870 driver id=1413697536 #bytes=1 p3=0 obj#=83176 tim=774959748 WAIT #17: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83176 tim=774959803 WAIT #17: nam='db file scattered read' ela= 17894 file#=8 block#=214304 blocks=8 obj#=83175 tim=774977886 FETCH #17:c=0,e=18537,p=8,cr=10,cu=0,mis=0,r=499,dep=0,og=1,plh=39991016,tim=774978324 STAT #17 id=1 cnt=10000 pid=0 pos=1 obj=0 op='FILTER (cr=374 pr=188 pw=0 time=9742 us)' STAT #17 id=2 cnt=10000 pid=1 pos=1 obj=0 op='HASH JOIN (cr=374 pr=188 pw=0 time=8717 us cost=51 size=109 card=1)' STAT #17 id=3 cnt=10000 pid=2 pos=1 obj=83174 op='TABLE ACCESS FULL T3_1 (cr=158 pr=0 pw=0 time=885 us cost=47 size=8 card=2)' STAT #17 id=4 cnt=10000 pid=2 pos=2 obj=83175 op='TABLE ACCESS BY INDEX ROWID T4_1 (cr=216 pr=188 pw=0 time=5255 us cost=3 size=210 card=2)' STAT #17 id=5 cnt=10000 pid=4 pos=1 obj=83176 op='INDEX RANGE SCAN IND_T4_1 (cr=42 pr=24 pw=0 time=2531 us cost=2 size=0 card=2)' WAIT #17: nam='SQL*Net message from client' ela= 2221 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=774980966 *** SESSION ID:(1232.2) 2010-09-06 10:17:12.316 CLOSE #17:c=0,e=8,dep=0,type=0,tim=774989253 ===================== From the above, we see that the execution plan, indicated by the STAT lines, was the same as when N2 was set to a value of 2. However, unlike with TKPROF, we are able to see how long the second execution required just from looking at the STAT lines in the raw 10046 trace file. ——————————————————— Second execution 10046 trace file, without embedded comments: ===================== PARSING IN CURSOR #7 len=101 dep=0 uid=90 oct=3 lid=90 tim=868879999 hv=3458052021 ad='36ce14150' sqlid='4122a6m71vbxp' SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 END OF STMT PARSE #7:c=0,e=22,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=39991016,tim=868879999 BINDS #7: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=48 off=0 kxsbbbfp=1d7ba010 bln=22 avl=02 flg=05 value=1 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24 kxsbbbfp=1d7ba028 bln=22 avl=02 flg=01 value=2 EXEC #7:c=0,e=989,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=39991016,tim=868881046 WAIT #7: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=0 tim=868881075 WAIT #7: nam='db file sequential read' ela= 33 file#=7 block#=1819466 blocks=1 obj#=83174 tim=868881219 WAIT #7: nam='db file scattered read' ela= 28 file#=7 block#=1819467 blocks=5 obj#=83174 tim=868881311 WAIT #7: nam='db file scattered read' ela= 33 file#=7 block#=1819472 blocks=8 obj#=83174 tim=868881491 WAIT #7: nam='db file scattered read' ela= 29 file#=7 block#=1819481 blocks=7 obj#=83174 tim=868881590 WAIT #7: nam='db file scattered read' ela= 30 file#=7 block#=1819488 blocks=8 obj#=83174 tim=868881684 WAIT #7: nam='db file scattered read' ela= 28 file#=7 block#=1819497 blocks=7 obj#=83174 tim=868881783 WAIT #7: nam='db file scattered read' ela= 30 file#=7 block#=1819504 blocks=8 obj#=83174 tim=868881885 WAIT #7: nam='db file scattered read' ela= 28 file#=7 block#=1819513 blocks=7 obj#=83174 tim=868881986 WAIT #7: nam='db file scattered read' ela= 32 file#=8 block#=214144 blocks=8 obj#=83174 tim=868882088 WAIT #7: nam='db file scattered read' ela= 28 file#=8 block#=214153 blocks=7 obj#=83174 tim=868882190 WAIT #7: nam='db file scattered read' ela= 30 file#=8 block#=214160 blocks=8 obj#=83174 tim=868882288 WAIT #7: nam='db file scattered read' ela= 30 file#=8 block#=214169 blocks=7 obj#=83174 tim=868882393 WAIT #7: nam='db file scattered read' ela= 30 file#=8 block#=214176 blocks=8 obj#=83174 tim=868882493 WAIT #7: nam='db file scattered read' ela= 29 file#=8 block#=214185 blocks=7 obj#=83174 tim=868882595 WAIT #7: nam='db file scattered read' ela= 31 file#=8 block#=214192 blocks=8 obj#=83174 tim=868882696 WAIT #7: nam='db file scattered read' ela= 29 file#=8 block#=214201 blocks=7 obj#=83174 tim=868882799 WAIT #7: nam='db file scattered read' ela= 31 file#=8 block#=214208 blocks=8 obj#=83174 tim=868882930 WAIT #7: nam='db file scattered read' ela= 98 file#=7 block#=1819522 blocks=36 obj#=83174 tim=868883136 WAIT #7: nam='db file scattered read' ela= 61 file#=7 block#=1819720 blocks=8 obj#=83176 tim=868883431 WAIT #7: nam='db file scattered read' ela= 48 file#=8 block#=214216 blocks=8 obj#=83175 tim=868883532 FETCH #7:c=0,e=2481,p=171,cr=161,cu=0,mis=0,r=1,dep=0,og=1,plh=39991016,tim=868883573 WAIT #7: nam='SQL*Net message from client' ela= 175 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868883778 WAIT #7: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868883813 FETCH #7:c=0,e=46,p=0,cr=2,cu=0,mis=0,r=1,dep=0,og=1,plh=39991016,tim=868883848 STAT #7 id=1 cnt=2 pid=0 pos=1 obj=0 op='FILTER (cr=163 pr=171 pw=0 time=0 us)' STAT #7 id=2 cnt=2 pid=1 pos=1 obj=0 op='HASH JOIN (cr=163 pr=171 pw=0 time=0 us cost=51 size=109 card=1)' STAT #7 id=3 cnt=2 pid=2 pos=1 obj=83174 op='TABLE ACCESS FULL T3_1 (cr=158 pr=155 pw=0 time=1 us cost=47 size=8 card=2)' STAT #7 id=4 cnt=2 pid=2 pos=2 obj=83175 op='TABLE ACCESS BY INDEX ROWID T4_1 (cr=5 pr=16 pw=0 time=0 us cost=3 size=210 card=2)' STAT #7 id=5 cnt=2 pid=4 pos=1 obj=83176 op='INDEX RANGE SCAN IND_T4_1 (cr=3 pr=8 pw=0 time=4 us cost=2 size=0 card=2)' WAIT #7: nam='SQL*Net message from client' ela= 217 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868884157 *** SESSION ID:(1232.2) 2010-09-06 10:18:46.228 CLOSE #7:c=0,e=8,dep=0,type=0,tim=868890954 ===================== PARSING IN CURSOR #16 len=26 dep=0 uid=90 oct=47 lid=90 tim=868891061 hv=3000858490 ad='36ce13ac0' sqlid='56ndayftduxvu' BEGIN :N2 := 10000; END; END OF STMT PARSE #16:c=0,e=52,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=0,tim=868891060 BINDS #16: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=24 off=0 kxsbbbfp=1d7ba028 bln=22 avl=02 flg=05 value=2 WAIT #16: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868891170 EXEC #16:c=0,e=84,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,plh=0,tim=868891184 WAIT #16: nam='SQL*Net message from client' ela= 8292 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868899504 *** SESSION ID:(1232.2) 2010-09-06 10:18:46.228 CLOSE #16:c=0,e=11,dep=0,type=0,tim=868900212 ===================== PARSING IN CURSOR #12 len=101 dep=0 uid=90 oct=3 lid=90 tim=868900272 hv=3458052021 ad='36ce14150' sqlid='4122a6m71vbxp' SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 END OF STMT PARSE #12:c=0,e=30,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=39991016,tim=868900272 BINDS #12: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=48 off=0 kxsbbbfp=1d7c5920 bln=22 avl=02 flg=05 value=1 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24 kxsbbbfp=1d7c5938 bln=22 avl=02 flg=01 value=10000 EXEC #12:c=0,e=941,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=1389598788,tim=868901272 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868901301 WAIT #12: nam='db file sequential read' ela= 48 file#=8 block#=214218 blocks=1 obj#=83175 tim=868904157 FETCH #12:c=15600,e=3235,p=1,cr=162,cu=0,mis=0,r=1,dep=0,og=1,plh=1389598788,tim=868904553 WAIT #12: nam='SQL*Net message from client' ela= 155 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868904737 WAIT #12: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868904769 WAIT #12: nam='db file scattered read' ela= 39 file#=8 block#=214224 blocks=8 obj#=83175 tim=868905030 FETCH #12:c=0,e=409,p=8,cr=8,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868905169 WAIT #12: nam='SQL*Net message from client' ela= 599 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868905788 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868905817 WAIT #12: nam='db file scattered read' ela= 32 file#=8 block#=214233 blocks=7 obj#=83175 tim=868906092 FETCH #12:c=0,e=409,p=7,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868906219 WAIT #12: nam='SQL*Net message from client' ela= 599 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868906839 WAIT #12: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868906868 WAIT #12: nam='db file scattered read' ela= 34 file#=8 block#=214240 blocks=8 obj#=83175 tim=868907088 FETCH #12:c=0,e=382,p=8,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868907242 WAIT #12: nam='SQL*Net message from client' ela= 620 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868907883 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868907912 WAIT #12: nam='db file scattered read' ela= 34 file#=8 block#=214249 blocks=7 obj#=83175 tim=868908142 FETCH #12:c=0,e=380,p=7,cr=8,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868908284 WAIT #12: nam='SQL*Net message from client' ela= 602 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868908906 WAIT #12: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868908935 WAIT #12: nam='db file scattered read' ela= 34 file#=8 block#=214256 blocks=8 obj#=83175 tim=868909143 FETCH #12:c=0,e=382,p=8,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868909309 WAIT #12: nam='SQL*Net message from client' ela= 600 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868909929 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868909957 WAIT #12: nam='db file scattered read' ela= 32 file#=8 block#=214265 blocks=7 obj#=83175 tim=868910171 FETCH #12:c=0,e=375,p=7,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868910325 WAIT #12: nam='SQL*Net message from client' ela= 597 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868910942 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868910970 WAIT #12: nam='db file scattered read' ela= 34 file#=7 block#=1819648 blocks=8 obj#=83175 tim=868911164 FETCH #12:c=0,e=380,p=8,cr=8,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868911343 WAIT #12: nam='SQL*Net message from client' ela= 598 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868911962 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868911990 WAIT #12: nam='db file scattered read' ela= 32 file#=7 block#=1819657 blocks=7 obj#=83175 tim=868912193 FETCH #12:c=0,e=387,p=7,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868912370 WAIT #12: nam='SQL*Net message from client' ela= 597 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868912987 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868913016 WAIT #12: nam='db file scattered read' ela= 35 file#=7 block#=1819664 blocks=8 obj#=83175 tim=868913198 FETCH #12:c=0,e=384,p=8,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868913392 WAIT #12: nam='SQL*Net message from client' ela= 599 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868914011 WAIT #12: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868914040 WAIT #12: nam='db file scattered read' ela= 31 file#=7 block#=1819673 blocks=7 obj#=83175 tim=868914230 FETCH #12:c=0,e=378,p=7,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868914410 WAIT #12: nam='SQL*Net message from client' ela= 597 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868915027 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868915056 WAIT #12: nam='db file scattered read' ela= 34 file#=7 block#=1819680 blocks=8 obj#=83175 tim=868915227 FETCH #12:c=0,e=385,p=8,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868915434 WAIT #12: nam='SQL*Net message from client' ela= 598 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868916052 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868916083 WAIT #12: nam='db file scattered read' ela= 32 file#=7 block#=1819689 blocks=7 obj#=83175 tim=868916259 FETCH #12:c=0,e=382,p=7,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868916457 WAIT #12: nam='SQL*Net message from client' ela= 596 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868917073 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868917102 WAIT #12: nam='db file scattered read' ela= 33 file#=7 block#=1819696 blocks=8 obj#=83175 tim=868917258 FETCH #12:c=0,e=391,p=8,cr=8,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868917485 WAIT #12: nam='SQL*Net message from client' ela= 598 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868918104 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868918159 WAIT #12: nam='db file scattered read' ela= 35 file#=7 block#=1819705 blocks=7 obj#=83175 tim=868918347 FETCH #12:c=15600,e=437,p=7,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868918562 WAIT #12: nam='SQL*Net message from client' ela= 604 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868919187 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868919216 WAIT #12: nam='db file scattered read' ela= 34 file#=7 block#=1819712 blocks=8 obj#=83175 tim=868919362 FETCH #12:c=0,e=392,p=8,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868919600 WAIT #12: nam='SQL*Net message from client' ela= 598 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868920218 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868920247 WAIT #12: nam='db file scattered read' ela= 102 file#=8 block#=214274 blocks=36 obj#=83175 tim=868920500 FETCH #12:c=0,e=585,p=36,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868920824 WAIT #12: nam='SQL*Net message from client' ela= 600 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868921445 WAIT #12: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868921475 FETCH #12:c=0,e=290,p=0,cr=8,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868921756 WAIT #12: nam='SQL*Net message from client' ela= 593 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868922368 WAIT #12: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868922397 FETCH #12:c=0,e=286,p=0,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868922675 WAIT #12: nam='SQL*Net message from client' ela= 579 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868923272 WAIT #12: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868923301 FETCH #12:c=0,e=287,p=0,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=868923580 WAIT #12: nam='SQL*Net message from client' ela= 601 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868924201 WAIT #12: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868924230 FETCH #12:c=0,e=314,p=0,cr=8,cu=0,mis=0,r=499,dep=0,og=1,plh=1389598788,tim=868924536 STAT #12 id=1 cnt=10000 pid=0 pos=1 obj=0 op='FILTER (cr=336 pr=150 pw=0 time=4999 us)' STAT #12 id=2 cnt=10000 pid=1 pos=1 obj=0 op='HASH JOIN (cr=336 pr=150 pw=0 time=4486 us cost=95 size=1090000 card=10000)' STAT #12 id=3 cnt=10000 pid=2 pos=1 obj=83174 op='TABLE ACCESS FULL T3_1 (cr=158 pr=0 pw=0 time=1265 us cost=47 size=40000 card=10000)' STAT #12 id=4 cnt=10000 pid=2 pos=2 obj=83175 op='TABLE ACCESS FULL T4_1 (cr=178 pr=150 pw=0 time=1410 us cost=47 size=1050000 card=10000)' WAIT #12: nam='SQL*Net message from client' ela= 2182 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868927063 *** SESSION ID:(1232.2) 2010-09-06 10:18:46.275 CLOSE #12:c=0,e=9,dep=0,type=0,tim=868935422 ===================== PARSING IN CURSOR #17 len=24 dep=0 uid=90 oct=3 lid=90 tim=868935520 hv=124468195 ad='37c5bc8a0' sqlid='c749bc43qqfz3' SELECT SYSDATE FROM DUAL END OF STMT PARSE #17:c=0,e=45,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1388734953,tim=868935519 EXEC #17:c=0,e=12,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1388734953,tim=868935599 WAIT #17: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868935633 FETCH #17:c=0,e=7,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,plh=1388734953,tim=868935656 STAT #17 id=1 cnt=1 pid=0 pos=1 obj=0 op='FAST DUAL (cr=0 pr=0 pw=0 time=0 us cost=2 size=0 card=1)' WAIT #17: nam='SQL*Net message from client' ela= 100 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868935790 FETCH #17:c=0,e=1,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=1388734953,tim=868935812 WAIT #17: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868935824 WAIT #17: nam='SQL*Net message from client' ela= 147 driver id=1413697536 #bytes=1 p3=0 obj#=83175 tim=868935981 How does the 10046 trace file for the second half of the script compare with the 10046 trace file for the first half of the script? What has changed? What is the same? And just to keep you wondering (what has changed and why), here is output from another box that is also running Oracle Database 11.2.0.1 using direct, asynchronous I/O and ASSM autoallocate tablespaces: SELECT OBJECT_NAME, OBJECT_ID, DATA_OBJECT_ID FROM DBA_OBJECTS WHERE OBJECT_NAME IN ('T3_1','T4_1','IND_T4_1'); OBJECT_NAME OBJECT_ID DATA_OBJECT_ID ------------ --------- -------------- T3_1 77260 77260 T4_1 77261 77261 IND_T4_1 77262 77262 SELECT SEGMENT_NAME, HEADER_FILE, HEADER_BLOCK FROM DBA_SEGMENTS WHERE SEGMENT_NAME IN ('T3_1','T4_1','IND_T4_1'); SEGMENT_NAME HEADER_FILE HEADER_BLOCK ------------ ----------- ------------ IND_T4_1 8 13410 T3_1 8 13154 T4_1 7 2556514 The raw trace file from the first execution on the second server: ===================== PARSING IN CURSOR #14 len=101 dep=0 uid=185 oct=3 lid=185 tim=480383990 hv=3458052021 ad='46953aca8' sqlid='4122a6m71vbxp' SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 END OF STMT PARSE #14:c=0,e=129,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=0,tim=480383989 BINDS #14: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=48 off=0 kxsbbbfp=1fc575b8 bln=22 avl=02 flg=05 value=1 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24 kxsbbbfp=1fc575d0 bln=22 avl=02 flg=01 value=2 EXEC #14:c=0,e=667,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=39991016,tim=480384712 WAIT #14: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=0 tim=480384738 WAIT #14: nam='db file sequential read' ela= 455 file#=8 block#=13154 blocks=1 obj#=77260 tim=480385283 WAIT #14: nam='db file scattered read' ela= 528 file#=8 block#=13155 blocks=5 obj#=77260 tim=480385892 WAIT #14: nam='db file scattered read' ela= 70 file#=8 block#=13160 blocks=8 obj#=77260 tim=480386198 WAIT #14: nam='db file scattered read' ela= 418 file#=8 block#=13169 blocks=7 obj#=77260 tim=480386706 WAIT #14: nam='db file scattered read' ela= 58 file#=8 block#=13176 blocks=8 obj#=77260 tim=480386845 WAIT #14: nam='db file scattered read' ela= 348 file#=7 block#=2556417 blocks=7 obj#=77260 tim=480387265 WAIT #14: nam='db file scattered read' ela= 49 file#=7 block#=2556424 blocks=8 obj#=77260 tim=480387409 WAIT #14: nam='db file scattered read' ela= 360 file#=7 block#=2556433 blocks=7 obj#=77260 tim=480387840 WAIT #14: nam='db file scattered read' ela= 47 file#=7 block#=2556440 blocks=8 obj#=77260 tim=480387975 WAIT #14: nam='db file scattered read' ela= 345 file#=7 block#=2556449 blocks=7 obj#=77260 tim=480388392 WAIT #14: nam='db file scattered read' ela= 49 file#=7 block#=2556456 blocks=8 obj#=77260 tim=480388545 WAIT #14: nam='db file scattered read' ela= 358 file#=7 block#=2556465 blocks=7 obj#=77260 tim=480388980 WAIT #14: nam='db file scattered read' ela= 51 file#=7 block#=2556472 blocks=8 obj#=77260 tim=480389123 WAIT #14: nam='db file scattered read' ela= 383 file#=7 block#=2556481 blocks=7 obj#=77260 tim=480389577 WAIT #14: nam='db file scattered read' ela= 47 file#=7 block#=2556488 blocks=8 obj#=77260 tim=480389694 WAIT #14: nam='db file scattered read' ela= 402 file#=7 block#=2556497 blocks=7 obj#=77260 tim=480390166 WAIT #14: nam='db file scattered read' ela= 118 file#=7 block#=2556504 blocks=8 obj#=77260 tim=480390371 WAIT #14: nam='db file scattered read' ela= 772 file#=8 block#=13186 blocks=36 obj#=77260 tim=480391278 WAIT #14: nam='db file sequential read' ela= 343 file#=8 block#=13411 blocks=1 obj#=77262 tim=480391856 WAIT #14: nam='db file scattered read' ela= 378 file#=8 block#=13412 blocks=4 obj#=77262 tim=480392334 WAIT #14: nam='db file sequential read' ela= 42 file#=7 block#=2556515 blocks=1 obj#=77261 tim=480392421 FETCH #14:c=0,e=7721,p=161,cr=161,cu=0,mis=0,r=1,dep=0,og=1,plh=39991016,tim=480392475 WAIT #14: nam='SQL*Net message from client' ela= 820 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480393374 WAIT #14: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480393452 FETCH #14:c=0,e=53,p=0,cr=2,cu=0,mis=0,r=1,dep=0,og=1,plh=39991016,tim=480393492 STAT #14 id=1 cnt=2 pid=0 pos=1 obj=0 op='FILTER (cr=163 pr=161 pw=0 time=0 us)' STAT #14 id=2 cnt=2 pid=1 pos=1 obj=0 op='HASH JOIN (cr=163 pr=161 pw=0 time=0 us cost=51 size=109 card=1)' STAT #14 id=3 cnt=2 pid=2 pos=1 obj=77260 op='TABLE ACCESS FULL T3_1 (cr=158 pr=155 pw=0 time=1 us cost=47 size=8 card=2)' STAT #14 id=4 cnt=2 pid=2 pos=2 obj=77261 op='TABLE ACCESS BY INDEX ROWID T4_1 (cr=5 pr=6 pw=0 time=0 us cost=3 size=210 card=2)' STAT #14 id=5 cnt=2 pid=4 pos=1 obj=77262 op='INDEX RANGE SCAN IND_T4_1 (cr=3 pr=5 pw=0 time=5 us cost=2 size=0 card=2)' WAIT #14: nam='SQL*Net message from client' ela= 928 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480394549 *** 2010-09-06 15:33:24.557 *** SESSION ID:(4.7) 2010-09-06 15:33:24.557 CLOSE #14:c=0,e=12,dep=0,type=0,tim=480410306 ===================== PARSING IN CURSOR #16 len=26 dep=0 uid=185 oct=47 lid=185 tim=480410638 hv=3000858490 ad='467bc4398' sqlid='56ndayftduxvu' BEGIN :N2 := 10000; END; END OF STMT PARSE #16:c=0,e=257,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=0,tim=480410637 BINDS #16: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=24 off=0 kxsbbbfp=1fc575d0 bln=22 avl=02 flg=05 value=2 WAIT #16: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480411380 EXEC #16:c=0,e=707,p=0,cr=0,cu=0,mis=1,r=1,dep=0,og=1,plh=0,tim=480411398 WAIT #16: nam='SQL*Net message from client' ela= 13898 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480425340 *** SESSION ID:(4.7) 2010-09-06 15:33:24.573 CLOSE #16:c=0,e=15,dep=0,type=0,tim=480427017 ===================== PARSING IN CURSOR #2 len=101 dep=0 uid=185 oct=3 lid=185 tim=480427079 hv=3458052021 ad='46953aca8' sqlid='4122a6m71vbxp' SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 END OF STMT PARSE #2:c=0,e=30,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=39991016,tim=480427079 BINDS #2: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=48 off=0 kxsbbbfp=2157a380 bln=22 avl=02 flg=05 value=1 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24 kxsbbbfp=2157a398 bln=22 avl=02 flg=01 value=10000 EXEC #2:c=0,e=67,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=39991016,tim=480427204 WAIT #2: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480427226 FETCH #2:c=0,e=3440,p=0,cr=161,cu=0,mis=0,r=1,dep=0,og=1,plh=39991016,tim=480430680 WAIT #2: nam='SQL*Net message from client' ela= 639 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480431346 WAIT #2: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480431379 WAIT #2: nam='db file sequential read' ela= 88 file#=7 block#=2556516 blocks=1 obj#=77261 tim=480431556 WAIT #2: nam='db file sequential read' ela= 41 file#=7 block#=2556517 blocks=1 obj#=77261 tim=480431766 WAIT #2: nam='db file sequential read' ela= 42 file#=7 block#=2556518 blocks=1 obj#=77261 tim=480431973 WAIT #2: nam='db file sequential read' ela= 41 file#=7 block#=2556519 blocks=1 obj#=77261 tim=480432178 WAIT #2: nam='db file sequential read' ela= 37 file#=7 block#=2556520 blocks=1 obj#=77261 tim=480432392 WAIT #2: nam='db file sequential read' ela= 41 file#=7 block#=2556521 blocks=1 obj#=77261 tim=480432597 WAIT #2: nam='db file sequential read' ela= 38 file#=7 block#=2556522 blocks=1 obj#=77261 tim=480432797 FETCH #2:c=0,e=1577,p=7,cr=10,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480432945 WAIT #2: nam='SQL*Net message from client' ela= 1377 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480434342 WAIT #2: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480434374 WAIT #2: nam='db file sequential read' ela= 28 file#=7 block#=2556523 blocks=1 obj#=77261 tim=480434445 WAIT #2: nam='db file sequential read' ela= 41 file#=7 block#=2556524 blocks=1 obj#=77261 tim=480434653 WAIT #2: nam='db file sequential read' ela= 44 file#=7 block#=2556525 blocks=1 obj#=77261 tim=480434863 WAIT #2: nam='db file sequential read' ela= 43 file#=7 block#=2556526 blocks=1 obj#=77261 tim=480435065 WAIT #2: nam='db file sequential read' ela= 41 file#=7 block#=2556527 blocks=1 obj#=77261 tim=480435266 WAIT #2: nam='db file sequential read' ela= 374 file#=7 block#=2556529 blocks=1 obj#=77261 tim=480435806 WAIT #2: nam='db file sequential read' ela= 269 file#=7 block#=2556530 blocks=1 obj#=77261 tim=480436279 WAIT #2: nam='db file sequential read' ela= 54 file#=7 block#=2556531 blocks=1 obj#=77261 tim=480436490 FETCH #2:c=0,e=2201,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480436565 WAIT #2: nam='SQL*Net message from client' ela= 1375 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480437961 WAIT #2: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480437994 WAIT #2: nam='db file sequential read' ela= 316 file#=7 block#=2556532 blocks=1 obj#=77261 tim=480438368 WAIT #2: nam='db file sequential read' ela= 36 file#=7 block#=2556533 blocks=1 obj#=77261 tim=480438619 WAIT #2: nam='db file sequential read' ela= 41 file#=7 block#=2556534 blocks=1 obj#=77261 tim=480438817 WAIT #2: nam='db file sequential read' ela= 40 file#=7 block#=2556535 blocks=1 obj#=77261 tim=480439020 WAIT #2: nam='db file sequential read' ela= 38 file#=7 block#=2556536 blocks=1 obj#=77261 tim=480439222 WAIT #2: nam='db file sequential read' ela= 62 file#=7 block#=2556537 blocks=1 obj#=77261 tim=480439479 WAIT #2: nam='db file sequential read' ela= 42 file#=7 block#=2556538 blocks=1 obj#=77261 tim=480439693 WAIT #2: nam='db file sequential read' ela= 41 file#=7 block#=2556539 blocks=1 obj#=77261 tim=480439901 FETCH #2:c=15600,e=1968,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480439952 WAIT #2: nam='SQL*Net message from client' ela= 1413 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480441410 WAIT #2: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480441442 WAIT #2: nam='db file sequential read' ela= 39 file#=7 block#=2556540 blocks=1 obj#=77261 tim=480441558 WAIT #2: nam='db file sequential read' ela= 39 file#=7 block#=2556541 blocks=1 obj#=77261 tim=480441758 WAIT #2: nam='db file sequential read' ela= 39 file#=7 block#=2556542 blocks=1 obj#=77261 tim=480441959 WAIT #2: nam='db file sequential read' ela= 40 file#=7 block#=2556543 blocks=1 obj#=77261 tim=480442160 WAIT #2: nam='db file scattered read' ela= 494 file#=8 block#=13312 blocks=8 obj#=77261 tim=480442829 WAIT #2: nam='db file scattered read' ela= 78 file#=8 block#=13416 blocks=8 obj#=77262 tim=480443114 FETCH #2:c=0,e=1843,p=20,cr=10,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480443275 WAIT #2: nam='SQL*Net message from client' ela= 1372 driver id=1413697536 #bytes=1 p3=0 obj#=77262 tim=480444666 WAIT #2: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77262 tim=480444699 WAIT #2: nam='db file scattered read' ela= 90 file#=8 block#=13320 blocks=8 obj#=77261 tim=480445061 FETCH #2:c=0,e=662,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480445350 WAIT #2: nam='SQL*Net message from client' ela= 1364 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480446736 WAIT #2: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480446768 WAIT #2: nam='db file scattered read' ela= 505 file#=8 block#=13328 blocks=8 obj#=77261 tim=480447559 FETCH #2:c=0,e=1060,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480447818 WAIT #2: nam='SQL*Net message from client' ela= 1347 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480449184 WAIT #2: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480449217 WAIT #2: nam='db file scattered read' ela= 71 file#=8 block#=13336 blocks=8 obj#=77261 tim=480449537 FETCH #2:c=0,e=650,p=8,cr=10,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480449857 WAIT #2: nam='SQL*Net message from client' ela= 1375 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480451251 WAIT #2: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480451283 WAIT #2: nam='db file scattered read' ela= 487 file#=8 block#=13344 blocks=8 obj#=77261 tim=480452037 FETCH #2:c=0,e=1045,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480452318 WAIT #2: nam='SQL*Net message from client' ela= 1502 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480453839 WAIT #2: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480453872 WAIT #2: nam='db file scattered read' ela= 75 file#=8 block#=13352 blocks=8 obj#=77261 tim=480454179 FETCH #2:c=0,e=662,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480454524 WAIT #2: nam='SQL*Net message from client' ela= 1392 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480455961 WAIT #2: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480456011 WAIT #2: nam='db file scattered read' ela= 509 file#=8 block#=13360 blocks=8 obj#=77261 tim=480456764 FETCH #2:c=0,e=1095,p=8,cr=10,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480457096 WAIT #2: nam='SQL*Net message from client' ela= 1287 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480458402 WAIT #2: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480458434 WAIT #2: nam='db file scattered read' ela= 74 file#=8 block#=13368 blocks=8 obj#=77261 tim=480458717 FETCH #2:c=0,e=646,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480459070 WAIT #2: nam='SQL*Net message from client' ela= 1282 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480460372 WAIT #2: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480460405 WAIT #2: nam='db file scattered read' ela= 486 file#=8 block#=13376 blocks=8 obj#=77261 tim=480461118 WAIT #2: nam='db file scattered read' ela= 506 file#=8 block#=13424 blocks=8 obj#=77262 tim=480461750 FETCH #2:c=0,e=1662,p=16,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480462056 WAIT #2: nam='SQL*Net message from client' ela= 1298 driver id=1413697536 #bytes=1 p3=0 obj#=77262 tim=480463374 WAIT #2: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77262 tim=480463408 WAIT #2: nam='db file scattered read' ela= 166 file#=8 block#=13384 blocks=8 obj#=77261 tim=480463763 FETCH #2:c=0,e=743,p=8,cr=10,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480464140 WAIT #2: nam='SQL*Net message from client' ela= 1285 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480465444 WAIT #2: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480465477 WAIT #2: nam='db file scattered read' ela= 482 file#=8 block#=13392 blocks=8 obj#=77261 tim=480466169 FETCH #2:c=0,e=1043,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480466510 WAIT #2: nam='SQL*Net message from client' ela= 1326 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480467855 WAIT #2: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480467887 WAIT #2: nam='db file scattered read' ela= 72 file#=8 block#=13400 blocks=8 obj#=77261 tim=480468126 FETCH #2:c=0,e=661,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480468538 WAIT #2: nam='SQL*Net message from client' ela= 1310 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480469868 WAIT #2: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480469901 WAIT #2: nam='db file sequential read' ela= 336 file#=7 block#=2556546 blocks=1 obj#=77261 tim=480470411 WAIT #2: nam='db file sequential read' ela= 44 file#=7 block#=2556547 blocks=1 obj#=77261 tim=480470701 WAIT #2: nam='db file sequential read' ela= 396 file#=7 block#=2556548 blocks=1 obj#=77261 tim=480471336 WAIT #2: nam='db file sequential read' ela= 42 file#=7 block#=2556549 blocks=1 obj#=77261 tim=480471575 WAIT #2: nam='db file sequential read' ela= 288 file#=7 block#=2556550 blocks=1 obj#=77261 tim=480472025 WAIT #2: nam='db file sequential read' ela= 38 file#=7 block#=2556551 blocks=1 obj#=77261 tim=480472256 FETCH #2:c=15600,e=2414,p=6,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480472305 WAIT #2: nam='SQL*Net message from client' ela= 1307 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480473659 WAIT #2: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480473692 WAIT #2: nam='db file sequential read' ela= 266 file#=7 block#=2556552 blocks=1 obj#=77261 tim=480474032 WAIT #2: nam='db file sequential read' ela= 42 file#=7 block#=2556553 blocks=1 obj#=77261 tim=480474264 WAIT #2: nam='db file sequential read' ela= 278 file#=7 block#=2556554 blocks=1 obj#=77261 tim=480474708 WAIT #2: nam='db file sequential read' ela= 23 file#=7 block#=2556555 blocks=1 obj#=77261 tim=480474939 WAIT #2: nam='db file sequential read' ela= 287 file#=7 block#=2556556 blocks=1 obj#=77261 tim=480475305 WAIT #2: nam='db file sequential read' ela= 39 file#=7 block#=2556557 blocks=1 obj#=77261 tim=480475546 WAIT #2: nam='db file sequential read' ela= 281 file#=7 block#=2556558 blocks=1 obj#=77261 tim=480475989 FETCH #2:c=0,e=2484,p=7,cr=10,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480476165 WAIT #2: nam='SQL*Net message from client' ela= 1301 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480477486 WAIT #2: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480477519 WAIT #2: nam='db file sequential read' ela= 110 file#=7 block#=2556559 blocks=1 obj#=77261 tim=480477664 WAIT #2: nam='db file sequential read' ela= 53 file#=7 block#=2556560 blocks=1 obj#=77261 tim=480477956 WAIT #2: nam='db file sequential read' ela= 39 file#=7 block#=2556561 blocks=1 obj#=77261 tim=480478164 WAIT #2: nam='db file sequential read' ela= 39 file#=7 block#=2556562 blocks=1 obj#=77261 tim=480478364 WAIT #2: nam='db file sequential read' ela= 41 file#=7 block#=2556563 blocks=1 obj#=77261 tim=480478562 WAIT #2: nam='db file sequential read' ela= 70 file#=7 block#=2556564 blocks=1 obj#=77261 tim=480478822 WAIT #2: nam='db file sequential read' ela= 38 file#=7 block#=2556565 blocks=1 obj#=77261 tim=480479055 WAIT #2: nam='db file sequential read' ela= 42 file#=7 block#=2556566 blocks=1 obj#=77261 tim=480479261 FETCH #2:c=0,e=1869,p=8,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480479377 WAIT #2: nam='SQL*Net message from client' ela= 1289 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480480704 WAIT #2: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480480736 WAIT #2: nam='db file sequential read' ela= 86 file#=7 block#=2556567 blocks=1 obj#=77261 tim=480480874 WAIT #2: nam='db file sequential read' ela= 39 file#=7 block#=2556568 blocks=1 obj#=77261 tim=480481096 WAIT #2: nam='db file scattered read' ela= 549 file#=8 block#=13432 blocks=8 obj#=77262 tim=480481724 WAIT #2: nam='db file sequential read' ela= 38 file#=7 block#=2556569 blocks=1 obj#=77261 tim=480481922 WAIT #2: nam='db file sequential read' ela= 38 file#=7 block#=2556570 blocks=1 obj#=77261 tim=480482122 WAIT #2: nam='db file sequential read' ela= 38 file#=7 block#=2556571 blocks=1 obj#=77261 tim=480482325 WAIT #2: nam='db file sequential read' ela= 38 file#=7 block#=2556572 blocks=1 obj#=77261 tim=480482537 WAIT #2: nam='db file sequential read' ela= 37 file#=7 block#=2556573 blocks=1 obj#=77261 tim=480482738 WAIT #2: nam='db file sequential read' ela= 36 file#=7 block#=2556574 blocks=1 obj#=77261 tim=480482939 FETCH #2:c=0,e=2276,p=16,cr=11,cu=0,mis=0,r=500,dep=0,og=1,plh=39991016,tim=480483002 WAIT #2: nam='SQL*Net message from client' ela= 1305 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480484352 WAIT #2: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480484384 WAIT #2: nam='db file sequential read' ela= 49 file#=7 block#=2556575 blocks=1 obj#=77261 tim=480484502 WAIT #2: nam='db file sequential read' ela= 39 file#=7 block#=2556576 blocks=1 obj#=77261 tim=480484710 WAIT #2: nam='db file sequential read' ela= 110 file#=7 block#=2556577 blocks=1 obj#=77261 tim=480484979 WAIT #2: nam='db file sequential read' ela= 105 file#=7 block#=2556578 blocks=1 obj#=77261 tim=480485257 WAIT #2: nam='db file sequential read' ela= 108 file#=7 block#=2556579 blocks=1 obj#=77261 tim=480485531 WAIT #2: nam='db file sequential read' ela= 108 file#=7 block#=2556580 blocks=1 obj#=77261 tim=480485827 WAIT #2: nam='db file sequential read' ela= 42 file#=7 block#=2556581 blocks=1 obj#=77261 tim=480486045 FETCH #2:c=15600,e=1841,p=7,cr=10,cu=0,mis=0,r=499,dep=0,og=1,plh=39991016,tim=480486215 STAT #2 id=1 cnt=10000 pid=0 pos=1 obj=0 op='FILTER (cr=374 pr=183 pw=0 time=53328 us)' STAT #2 id=2 cnt=10000 pid=1 pos=1 obj=0 op='HASH JOIN (cr=374 pr=183 pw=0 time=51020 us cost=51 size=109 card=1)' STAT #2 id=3 cnt=10000 pid=2 pos=1 obj=77260 op='TABLE ACCESS FULL T3_1 (cr=158 pr=0 pw=0 time=1645 us cost=47 size=8 card=2)' STAT #2 id=4 cnt=10000 pid=2 pos=2 obj=77261 op='TABLE ACCESS BY INDEX ROWID T4_1 (cr=216 pr=183 pw=0 time=46277 us cost=3 size=210 card=2)' STAT #2 id=5 cnt=10000 pid=4 pos=1 obj=77262 op='INDEX RANGE SCAN IND_T4_1 (cr=42 pr=24 pw=0 time=2404 us cost=2 size=0 card=2)' WAIT #2: nam='SQL*Net message from client' ela= 3978 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480490557 *** SESSION ID:(4.7) 2010-09-06 15:33:24.651 CLOSE #2:c=0,e=9,dep=0,type=0,tim=480512517 ===================== The raw trace file from the second execution on the second server: ===================== PARSING IN CURSOR #14 len=101 dep=0 uid=185 oct=3 lid=185 tim=480800887 hv=3458052021 ad='46953aca8' sqlid='4122a6m71vbxp' SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 END OF STMT PARSE #14:c=0,e=20,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=39991016,tim=480800887 BINDS #14: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=48 off=0 kxsbbbfp=1fc575b8 bln=22 avl=02 flg=05 value=1 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24 kxsbbbfp=1fc575d0 bln=22 avl=02 flg=01 value=2 EXEC #14:c=0,e=898,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=39991016,tim=480801842 WAIT #14: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=0 tim=480801869 WAIT #14: nam='db file sequential read' ela= 57 file#=8 block#=13154 blocks=1 obj#=77260 tim=480802004 WAIT #14: nam='db file scattered read' ela= 83 file#=8 block#=13155 blocks=5 obj#=77260 tim=480802180 WAIT #14: nam='db file scattered read' ela= 54 file#=8 block#=13160 blocks=8 obj#=77260 tim=480802462 WAIT #14: nam='db file scattered read' ela= 52 file#=8 block#=13169 blocks=7 obj#=77260 tim=480802654 WAIT #14: nam='db file scattered read' ela= 59 file#=8 block#=13176 blocks=8 obj#=77260 tim=480802843 WAIT #14: nam='db file scattered read' ela= 48 file#=7 block#=2556417 blocks=7 obj#=77260 tim=480803026 WAIT #14: nam='db file scattered read' ela= 54 file#=7 block#=2556424 blocks=8 obj#=77260 tim=480803212 WAIT #14: nam='db file scattered read' ela= 47 file#=7 block#=2556433 blocks=7 obj#=77260 tim=480803398 WAIT #14: nam='db file scattered read' ela= 56 file#=7 block#=2556440 blocks=8 obj#=77260 tim=480803591 WAIT #14: nam='db file scattered read' ela= 56 file#=7 block#=2556449 blocks=7 obj#=77260 tim=480803780 WAIT #14: nam='db file scattered read' ela= 48 file#=7 block#=2556456 blocks=8 obj#=77260 tim=480803957 WAIT #14: nam='db file scattered read' ela= 53 file#=7 block#=2556465 blocks=7 obj#=77260 tim=480804152 WAIT #14: nam='db file scattered read' ela= 64 file#=7 block#=2556472 blocks=8 obj#=77260 tim=480804342 WAIT #14: nam='db file scattered read' ela= 52 file#=7 block#=2556481 blocks=7 obj#=77260 tim=480804534 WAIT #14: nam='db file scattered read' ela= 65 file#=7 block#=2556488 blocks=8 obj#=77260 tim=480804726 WAIT #14: nam='db file scattered read' ela= 120 file#=7 block#=2556497 blocks=7 obj#=77260 tim=480804990 WAIT #14: nam='db file scattered read' ela= 100 file#=7 block#=2556504 blocks=8 obj#=77260 tim=480805216 WAIT #14: nam='db file scattered read' ela= 175 file#=8 block#=13186 blocks=36 obj#=77260 tim=480805574 WAIT #14: nam='db file scattered read' ela= 81 file#=8 block#=13408 blocks=8 obj#=77262 tim=480805968 WAIT #14: nam='db file sequential read' ela= 58 file#=7 block#=2556515 blocks=1 obj#=77261 tim=480806114 FETCH #14:c=0,e=4284,p=164,cr=161,cu=0,mis=0,r=1,dep=0,og=1,plh=39991016,tim=480806168 WAIT #14: nam='SQL*Net message from client' ela= 660 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480806885 WAIT #14: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480806921 FETCH #14:c=0,e=48,p=0,cr=2,cu=0,mis=0,r=1,dep=0,og=1,plh=39991016,tim=480806957 STAT #14 id=1 cnt=2 pid=0 pos=1 obj=0 op='FILTER (cr=163 pr=164 pw=0 time=0 us)' STAT #14 id=2 cnt=2 pid=1 pos=1 obj=0 op='HASH JOIN (cr=163 pr=164 pw=0 time=0 us cost=51 size=109 card=1)' STAT #14 id=3 cnt=2 pid=2 pos=1 obj=77260 op='TABLE ACCESS FULL T3_1 (cr=158 pr=155 pw=0 time=1 us cost=47 size=8 card=2)' STAT #14 id=4 cnt=2 pid=2 pos=2 obj=77261 op='TABLE ACCESS BY INDEX ROWID T4_1 (cr=5 pr=9 pw=0 time=0 us cost=3 size=210 card=2)' STAT #14 id=5 cnt=2 pid=4 pos=1 obj=77262 op='INDEX RANGE SCAN IND_T4_1 (cr=3 pr=8 pw=0 time=5 us cost=2 size=0 card=2)' WAIT #14: nam='SQL*Net message from client' ela= 916 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480807962 *** SESSION ID:(4.7) 2010-09-06 15:33:24.978 CLOSE #14:c=0,e=11,dep=0,type=0,tim=480829385 ===================== PARSING IN CURSOR #16 len=26 dep=0 uid=185 oct=47 lid=185 tim=480829534 hv=3000858490 ad='467bc4398' sqlid='56ndayftduxvu' BEGIN :N2 := 10000; END; END OF STMT PARSE #16:c=0,e=63,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=0,tim=480829534 BINDS #16: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=24 off=0 kxsbbbfp=2157ab70 bln=22 avl=02 flg=05 value=2 WAIT #16: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480829651 EXEC #16:c=0,e=79,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,plh=0,tim=480829664 WAIT #16: nam='SQL*Net message from client' ela= 16786 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480846479 *** SESSION ID:(4.7) 2010-09-06 15:33:24.994 CLOSE #16:c=0,e=13,dep=0,type=0,tim=480848124 ===================== PARSING IN CURSOR #9 len=101 dep=0 uid=185 oct=3 lid=185 tim=480848185 hv=3458052021 ad='46953aca8' sqlid='4122a6m71vbxp' SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 END OF STMT PARSE #9:c=0,e=29,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=39991016,tim=480848185 BINDS #9: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=48 off=0 kxsbbbfp=1fc575b8 bln=22 avl=02 flg=05 value=1 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24 kxsbbbfp=1fc575d0 bln=22 avl=02 flg=01 value=10000 EXEC #9:c=0,e=878,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=1389598788,tim=480849120 WAIT #9: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480849146 WAIT #9: nam='db file sequential read' ela= 64 file#=7 block#=2556514 blocks=1 obj#=77261 tim=480851978 FETCH #9:c=0,e=3233,p=1,cr=162,cu=0,mis=0,r=1,dep=0,og=1,plh=1389598788,tim=480852394 WAIT #9: nam='SQL*Net message from client' ela= 622 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480853041 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480853074 WAIT #9: nam='db file scattered read' ela= 56 file#=7 block#=2556516 blocks=4 obj#=77261 tim=480853200 WAIT #9: nam='db file scattered read' ela= 54 file#=7 block#=2556520 blocks=8 obj#=77261 tim=480853514 FETCH #9:c=0,e=654,p=12,cr=8,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480853719 WAIT #9: nam='SQL*Net message from client' ela= 1298 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480855067 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480855099 WAIT #9: nam='db file scattered read' ela= 68 file#=7 block#=2556529 blocks=7 obj#=77261 tim=480855450 FETCH #9:c=0,e=528,p=7,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480855618 WAIT #9: nam='SQL*Net message from client' ela= 1296 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480856933 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480856964 WAIT #9: nam='db file scattered read' ela= 52 file#=7 block#=2556536 blocks=8 obj#=77261 tim=480857207 FETCH #9:c=0,e=476,p=8,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480857431 WAIT #9: nam='SQL*Net message from client' ela= 1299 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480858748 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480858779 WAIT #9: nam='db file scattered read' ela= 53 file#=8 block#=13313 blocks=7 obj#=77261 tim=480859035 FETCH #9:c=0,e=474,p=7,cr=8,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480859245 WAIT #9: nam='SQL*Net message from client' ela= 1351 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480860614 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480860671 WAIT #9: nam='db file scattered read' ela= 51 file#=8 block#=13320 blocks=8 obj#=77261 tim=480860899 FETCH #9:c=0,e=465,p=8,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480861127 WAIT #9: nam='SQL*Net message from client' ela= 1284 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480862430 WAIT #9: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480862461 WAIT #9: nam='db file scattered read' ela= 50 file#=8 block#=13329 blocks=7 obj#=77261 tim=480862698 FETCH #9:c=0,e=459,p=7,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480862912 WAIT #9: nam='SQL*Net message from client' ela= 1294 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480864224 WAIT #9: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480864255 WAIT #9: nam='db file scattered read' ela= 50 file#=8 block#=13336 blocks=8 obj#=77261 tim=480864474 FETCH #9:c=0,e=467,p=8,cr=8,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480864713 WAIT #9: nam='SQL*Net message from client' ela= 1300 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480866031 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480866062 WAIT #9: nam='db file scattered read' ela= 51 file#=8 block#=13345 blocks=7 obj#=77261 tim=480866290 FETCH #9:c=0,e=462,p=7,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480866516 WAIT #9: nam='SQL*Net message from client' ela= 1297 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480867832 WAIT #9: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480867863 WAIT #9: nam='db file scattered read' ela= 51 file#=8 block#=13352 blocks=8 obj#=77261 tim=480868069 FETCH #9:c=0,e=467,p=8,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480868321 WAIT #9: nam='SQL*Net message from client' ela= 1304 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480869643 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480869674 WAIT #9: nam='db file scattered read' ela= 52 file#=8 block#=13361 blocks=7 obj#=77261 tim=480869892 FETCH #9:c=0,e=474,p=7,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480870139 WAIT #9: nam='SQL*Net message from client' ela= 1286 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480871444 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480871475 WAIT #9: nam='db file scattered read' ela= 55 file#=8 block#=13368 blocks=8 obj#=77261 tim=480871672 FETCH #9:c=0,e=523,p=8,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480871989 WAIT #9: nam='SQL*Net message from client' ela= 1343 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480873360 WAIT #9: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480873392 WAIT #9: nam='db file scattered read' ela= 54 file#=8 block#=13377 blocks=7 obj#=77261 tim=480873598 FETCH #9:c=0,e=474,p=7,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480873857 WAIT #9: nam='SQL*Net message from client' ela= 1298 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480875174 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480875205 WAIT #9: nam='db file scattered read' ela= 56 file#=8 block#=13384 blocks=8 obj#=77261 tim=480875392 FETCH #9:c=0,e=472,p=8,cr=8,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480875668 WAIT #9: nam='SQL*Net message from client' ela= 1396 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480877083 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480877131 WAIT #9: nam='db file scattered read' ela= 52 file#=8 block#=13393 blocks=7 obj#=77261 tim=480877320 FETCH #9:c=0,e=463,p=7,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480877585 WAIT #9: nam='SQL*Net message from client' ela= 1295 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480878899 WAIT #9: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480878930 WAIT #9: nam='db file scattered read' ela= 52 file#=8 block#=13400 blocks=8 obj#=77261 tim=480879100 FETCH #9:c=0,e=467,p=8,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480879389 WAIT #9: nam='SQL*Net message from client' ela= 1296 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480880704 WAIT #9: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480880735 WAIT #9: nam='db file scattered read' ela= 196 file#=7 block#=2556546 blocks=36 obj#=77261 tim=480881087 FETCH #9:c=0,e=686,p=36,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480881412 WAIT #9: nam='SQL*Net message from client' ela= 1349 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480882780 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480882811 FETCH #9:c=0,e=289,p=0,cr=8,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480883091 WAIT #9: nam='SQL*Net message from client' ela= 1284 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480884393 WAIT #9: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480884424 FETCH #9:c=0,e=289,p=0,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480884704 WAIT #9: nam='SQL*Net message from client' ela= 1292 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480886015 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480886046 FETCH #9:c=0,e=290,p=0,cr=9,cu=0,mis=0,r=500,dep=0,og=1,plh=1389598788,tim=480886327 WAIT #9: nam='SQL*Net message from client' ela= 1291 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480887636 WAIT #9: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480887666 FETCH #9:c=0,e=313,p=0,cr=8,cu=0,mis=0,r=499,dep=0,og=1,plh=1389598788,tim=480887971 STAT #9 id=1 cnt=10000 pid=0 pos=1 obj=0 op='FILTER (cr=336 pr=154 pw=0 time=4999 us)' STAT #9 id=2 cnt=10000 pid=1 pos=1 obj=0 op='HASH JOIN (cr=336 pr=154 pw=0 time=4102 us cost=95 size=1090000 card=10000)' STAT #9 id=3 cnt=10000 pid=2 pos=1 obj=77260 op='TABLE ACCESS FULL T3_1 (cr=158 pr=0 pw=0 time=1771 us cost=47 size=40000 card=10000)' STAT #9 id=4 cnt=10000 pid=2 pos=2 obj=77261 op='TABLE ACCESS FULL T4_1 (cr=178 pr=154 pw=0 time=1281 us cost=47 size=1050000 card=10000)' WAIT #9: nam='SQL*Net message from client' ela= 4000 driver id=1413697536 #bytes=1 p3=0 obj#=77261 tim=480892264 *** SESSION ID:(4.7) 2010-09-06 15:33:25.056 CLOSE #9:c=0,e=8,dep=0,type=0,tim=480915337 ===================== How are the trace files from the two servers the same, and how are they different? To possibly help you see the differences, the TKPROF summaries from the second server: TKPROF N1 = 1, N2 = 2 and N2 = 10,000, first execution, second server: SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 2 0.00 0.00 0 0 0 0 Execute 2 0.00 0.00 0 0 0 0 Fetch 23 0.04 0.03 344 537 0 10002 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 27 0.04 0.04 344 537 0 10002 Misses in library cache during parse: 1 Misses in library cache during execute: 1 Optimizer mode: ALL_ROWS Parsing user id: 185 Rows Row Source Operation ------- --------------------------------------------------- 10000 FILTER (cr=374 pr=183 pw=0 time=53328 us) 10000 HASH JOIN (cr=374 pr=183 pw=0 time=51020 us cost=51 size=109 card=1) 10000 TABLE ACCESS FULL T3_1 (cr=158 pr=0 pw=0 time=1645 us cost=47 size=8 card=2) 10000 TABLE ACCESS BY INDEX ROWID T4_1 (cr=216 pr=183 pw=0 time=46277 us cost=3 size=210 card=2) 10000 INDEX RANGE SCAN IND_T4_1 (cr=42 pr=24 pw=0 time=2404 us cost=2 size=0 card=2)(object id 77262) Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ SQL*Net message to client 23 0.00 0.00 SQL*Net message from client 23 0.00 0.03 db file sequential read 66 0.00 0.00 db file scattered read 33 0.00 0.00 TKPROF N1 = 1, N2 = 2 and N2 = 10,000, second execution, second server: SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.00 0.00 164 163 0 2 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.00 0.00 164 163 0 2 Misses in library cache during parse: 0 Misses in library cache during execute: 1 Optimizer mode: ALL_ROWS Parsing user id: 185 Rows Row Source Operation ------- --------------------------------------------------- 2 FILTER (cr=163 pr=164 pw=0 time=0 us) 2 HASH JOIN (cr=163 pr=164 pw=0 time=0 us cost=51 size=109 card=1) 2 TABLE ACCESS FULL T3_1 (cr=158 pr=155 pw=0 time=1 us cost=47 size=8 card=2) 2 TABLE ACCESS BY INDEX ROWID T4_1 (cr=5 pr=9 pw=0 time=0 us cost=3 size=210 card=2) 2 INDEX RANGE SCAN IND_T4_1 (cr=3 pr=8 pw=0 time=5 us cost=2 size=0 card=2)(object id 77262) Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ SQL*Net message to client 2 0.00 0.00 db file sequential read 2 0.00 0.00 db file scattered read 18 0.00 0.00 SQL*Net message from client 2 0.00 0.00 ******************************************************************************** SELECT T3_1.C1, T4_1.C2 FROM T3_1, T4_1 WHERE T3_1.C1 BETWEEN :N1 AND :N2 AND T3_1.C1=T4_1.C1 call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 21 0.00 0.01 154 336 0 10000 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 23 0.00 0.01 154 336 0 10000 Misses in library cache during parse: 0 Misses in library cache during execute: 1 Optimizer mode: ALL_ROWS Parsing user id: 185 Rows Row Source Operation ------- --------------------------------------------------- 10000 FILTER (cr=336 pr=154 pw=0 time=4999 us) 10000 HASH JOIN (cr=336 pr=154 pw=0 time=4102 us cost=95 size=1090000 card=10000) 10000 TABLE ACCESS FULL T3_1 (cr=158 pr=0 pw=0 time=1771 us cost=47 size=40000 card=10000) 10000 TABLE ACCESS FULL T4_1 (cr=178 pr=154 pw=0 time=1281 us cost=47 size=1050000 card=10000) Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ SQL*Net message to client 21 0.00 0.00 db file sequential read 1 0.00 0.00 SQL*Net message from client 21 0.00 0.02 db file scattered read 17 0.00 0.00 The SQL*Plus statistics output from both servers were virtually identical. What is the same, and what is the difference between the executions on both servers? Comments : 3 Comments » Categories : Bind Variable, Performance From a VBS Script to a 10046 Trace and Back into a VBS Script 12 03 2010 March 12, 2010 I thought that we would try something a bit more difficult today. In previous articles I showed how to generate and read 10046 trace files using various methods, and I also showed a couple of different VBS scripts that could interact with an Oracle database. With some effort we could even read through a 10046 to pull out bits of information, much like TKPROF, but it probably does not make much sense to reproduce what TKPROF already accomplishes. I thought instead what I would do is to create a VBS script that generates a 10046 trace file at level 4, while executing a couple of SQL statements. A second VBS script will read the raw 10046 trace file and convert that trace file back into a VBS script, complete with bind variables. The code for the second VBS script is based on some of the code in my Toy project for performance tuning – something that I originally created just to see if it could be done. First, we need a table to use as the data source for the first VBS script – this is the test table used in this blog article: CREATE TABLE EMPLOYEE_RECORD_TEST AS SELECT DECODE(TRUNC(DBMS_RANDOM.VALUE(0,5)), 0,'MIKE', 1,'ROB', 2,'SAM', 3,'JOE', 4,'ERIC') EMPLOYEE_ID, TRUNC(SYSDATE)-ROUND(DBMS_RANDOM.VALUE(0,1000)) SHIFT_DATE, DECODE(TRUNC(DBMS_RANDOM.VALUE(0,10)), 0,'VAC', 1,'HOL', 2,'BEREAVE', 3,'JURY', 4,'ABS', 5,'EXCUSE', 6,'MIL', 'OTHER') INDIRECT_ID FROM DUAL CONNECT BY LEVEL<=1000; With the test table built, we execute the following simple VBS script (using either CSCRIPT or WSCRIPT on a Windows client): Const adCmdText = 1 Const adNumeric = 131 Const adDate = 7 Const adDBDate = 133 Const adDBTimeStamp = 135 Const adDBTime = 134 Const adVarChar = 200 Const adParamInput = 1 Dim strSQL Dim strUsername Dim strPassword Dim strDatabase Dim dbDatabase Dim snpDataEmployees Dim comDataEmployees Dim snpDataAttend Dim comDataAttend Dim snpDataEmpRecord Dim comDataEmpRecord Set dbDatabase = CreateObject("ADODB.Connection") Set snpDataEmployees = CreateObject("ADODB.Recordset") Set comDataEmployees = CreateObject("ADODB.Command") Set snpDataAttend = CreateObject("ADODB.Recordset") Set comDataAttend = CreateObject("ADODB.Command") Set snpDataEmpRecord = CreateObject("ADODB.Recordset") Set comDataEmpRecord = CreateObject("ADODB.Command") strUsername = "MyUsername" strPassword = "MyPassword" strDatabase = "MyDB" dbDatabase.ConnectionString = "Provider=OraOLEDB.Oracle;Data Source=" & strDatabase & ";User ID=" & strUsername & ";Password=" & strPassword & ";" dbDatabase.Open 'Should verify that the connection attempt was successful, but I will leave that for someone else to code dbDatabase.Execute "ALTER SESSION SET TRACEFILE_IDENTIFIER = 'VBS2TRACE2VBS'" dbDatabase.Execute "ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 4'" strSQL = "INSERT INTO EMPLOYEE_RECORD_TEST(" & VBCrLf strSQL = strSQL & " EMPLOYEE_ID," & VBCrLf strSQL = strSQL & " SHIFT_DATE," & VBCrLf strSQL = strSQL & " INDIRECT_ID)" & VBCrLf strSQL = strSQL & "VALUES(" & VBCrLf strSQL = strSQL & " ?," & VBCrLf strSQL = strSQL & " ?," & VBCrLf strSQL = strSQL & " ?)" With comDataEmpRecord 'Set up the command properties .CommandText = strSQL .CommandType = adCmdText .CommandTimeout = 30 .ActiveConnection = dbDatabase .Parameters.Append .CreateParameter("employee_id", adVarChar, adParamInput, 15, "TEST") .Parameters.Append .CreateParameter("shift_date", adDate, adParamInput, 8, Date) .Parameters.Append .CreateParameter("indirect_id", adVarchar, adParamInput, 15, "HOL") End With 'Rollback Test dbDatabase.BeginTrans comDataEmpRecord.Execute dbDatabase.RollbackTrans strSQL = "SELECT DISTINCT" & vbCrLf strSQL = strSQL & " EMPLOYEE_ID" & vbCrLf strSQL = strSQL & "FROM" & vbCrLf strSQL = strSQL & " EMPLOYEE_RECORD_TEST" & vbCrLf strSQL = strSQL & "WHERE" & vbCrLf strSQL = strSQL & " SHIFT_DATE>= ?" & vbCrLf strSQL = strSQL & " AND INDIRECT_ID= ?" & vbCrLf strSQL = strSQL & "ORDER BY" & vbCrLf strSQL = strSQL & " EMPLOYEE_ID" With comDataEmployees 'Set up the command properties .CommandText = strSQL .CommandType = adCmdText .CommandTimeout = 30 .ActiveConnection = dbDatabase .Parameters.Append .CreateParameter("shift_date", adDate, adParamInput, 8, DateAdd("d", -90, Date)) .Parameters.Append .CreateParameter("indirect_id", adVarChar, adParamInput, 15, "VAC") End With strSQL = "SELECT" & vbCrLf strSQL = strSQL & " SUM(DECODE(TO_CHAR(SHIFT_DATE,'D'),'2',1,0)) MON_COUNT," & vbCrLf strSQL = strSQL & " SUM(DECODE(TO_CHAR(SHIFT_DATE,'D'),'3',1,0)) TUE_COUNT," & vbCrLf strSQL = strSQL & " SUM(DECODE(TO_CHAR(SHIFT_DATE,'D'),'4',1,0)) WED_COUNT," & vbCrLf strSQL = strSQL & " SUM(DECODE(TO_CHAR(SHIFT_DATE,'D'),'5',1,0)) THU_COUNT," & vbCrLf strSQL = strSQL & " SUM(DECODE(TO_CHAR(SHIFT_DATE,'D'),'6',1,0)) FRI_COUNT" & vbCrLf strSQL = strSQL & "FROM" & vbCrLf strSQL = strSQL & " EMPLOYEE_RECORD_TEST" & vbCrLf strSQL = strSQL & "WHERE" & vbCrLf strSQL = strSQL & " EMPLOYEE_ID= ?" & vbCrLf strSQL = strSQL & " AND INDIRECT_ID= ?" With comDataAttend 'Set up the command properties .CommandText = strSQL .CommandType = adCmdText .CommandTimeout = 30 .ActiveConnection = dbDatabase .Parameters.Append .CreateParameter("employee_id", adVarChar, adParamInput, 15, "HOOPER") .Parameters.Append .CreateParameter("indirect_id", adVarChar, adParamInput, 15, "EXCUSE") End With Set snpDataEmployees = comDataEmployees.Execute If Not (snpDataEmployees Is Nothing) Then Do While Not snpDataEmployees.EOF comDataAttend("employee_id") = snpDataEmployees("employee_id") comDataAttend("indirect_id") = "EXCUSE" Set snpDataAttend = comDataAttend.Execute If Not snpDataAttend.EOF Then 'Do Something with the data End If snpDataAttend.Close comDataAttend("indirect_id") = "ABS" Set snpDataAttend = comDataAttend.Execute If Not snpDataAttend.EOF Then 'Do Something with the data End If snpDataAttend.Close snpDataEmployees.MoveNext Loop snpDataEmployees.Close End If dbDatabase.Close Set snpDataEmployees = Nothing Set snpDataAttend = Nothing Set comDataEmployees = Nothing Set comDataAttend = Nothing Set snpDataEmpRecord = Nothing Set comDataEmpRecord = Nothing Set dbDatabase = Nothing (TestScript.vbs – save as TestScript.vbs) In the above, replace MyDB with a valid database name from the tnsnames.ora file, MyUsername with a valid username, and MyPassword with the password for the user. The script starts by starting a transaction (the default behavior is an implicit commit), a row is inserted into the test table, and then a ROLLBACK is performed. The script then submits a SQL statement that retrieves a list of 5 employees from the test table. For each of the (up to) 5 employees a second SQL statement is executed with two different bind variable sets to determine the number of each week day the employee has been out of work on an excused (EXCUSE) or unexcused (ABS) absence. If I were writing a real program to accomplish this task I would combine the three SELECT statements into a single SELECT statement, but I want to demonstrate how the second VBS script handles multiple SQL statements that are open at the same time. Running the above script generated a trace file when executed against Oracle Database 11.2.0.1: or112_ora_5482_VBS2TRACE2VBS.trc (save as C:\or112_ora_5482_VBS2TRACE2VBS.trc – Windows users can view the file with Wordpad and convert the file into a plain text file that can be opened with Notepad). The goal is to take the trace file and transform it back into a VBS script, ignoring SQL statements that appear in the trace file at a depth greater than 0. The output of the VBS script that reads the 10046 trace file and generates a VBS file should look something like this: 'Source File:C:\or112_ora_5482_VBS2TRACE2VBS.trc HyperactiveTrace Sub HyperactiveTrace() Const adCmdText = 1 Const adCmdStoredProc = 4 Const adParamInput = 1 Const adVarNumeric = 139 Const adBigInt = 20 Const adDecimal = 14 Const adDouble = 5 Const adInteger = 3 Const adLongVarBinary = 205 Const adNumeric = 131 Const adSingle = 4 Const adSmallInt = 2 Const adTinyInt = 16 Const adUnsignedBigInt = 21 Const adUnsignedInt = 19 Const adUnsignedSmallInt = 18 Const adUnsignedTinyInt = 17 Const adDate = 7 Const adDBDate = 133 Const adDBTimeStamp = 135 Const adDBTime = 134 Const adVarChar = 200 Const adChar = 129 Const adUseClient = 3 Dim i Dim strSQL Dim strUsername Dim strPassword Dim strDatabase Dim dbDatabase Set dbDatabase = CreateObject("ADODB.Connection") Dim snpData1 Dim comData1 Set snpData1 = CreateObject("ADODB.Recordset") Set comData1 = CreateObject("ADODB.Command") Dim snpData2 Dim comData2 Set snpData2 = CreateObject("ADODB.Recordset") Set comData2 = CreateObject("ADODB.Command") Dim snpData3 Dim comData3 Set snpData3 = CreateObject("ADODB.Recordset") Set comData3 = CreateObject("ADODB.Command") Dim snpData4 Dim comData4 Set snpData4 = CreateObject("ADODB.Recordset") Set comData4 = CreateObject("ADODB.Command") Dim snpData5 Dim comData5 Set snpData5 = CreateObject("ADODB.Recordset") Set comData5 = CreateObject("ADODB.Command") Dim snpData6 Dim comData6 Set snpData6 = CreateObject("ADODB.Recordset") Set comData6 = CreateObject("ADODB.Command") Dim snpData7 Dim comData7 Set snpData7 = CreateObject("ADODB.Recordset") Set comData7 = CreateObject("ADODB.Command") Dim snpData8 Dim comData8 Set snpData8 = CreateObject("ADODB.Recordset") Set comData8 = CreateObject("ADODB.Command") Dim snpData9 Dim comData9 Set snpData9 = CreateObject("ADODB.Recordset") Set comData9 = CreateObject("ADODB.Command") Dim snpData10 Dim comData10 Set snpData10 = CreateObject("ADODB.Recordset") Set comData10 = CreateObject("ADODB.Command") Dim snpData11 Dim comData11 Set snpData11 = CreateObject("ADODB.Recordset") Set comData11 = CreateObject("ADODB.Command") Dim snpData12 Dim comData12 Set snpData12 = CreateObject("ADODB.Recordset") Set comData12 = CreateObject("ADODB.Command") Dim snpData13 Dim comData13 Set snpData13 = CreateObject("ADODB.Recordset") Set comData13 = CreateObject("ADODB.Command") Dim snpData14 Dim comData14 Set snpData14 = CreateObject("ADODB.Recordset") Set comData14 = CreateObject("ADODB.Command") Dim snpData15 Dim comData15 Set snpData15 = CreateObject("ADODB.Recordset") Set comData15 = CreateObject("ADODB.Command") Dim snpData16 Dim comData16 Set snpData16 = CreateObject("ADODB.Recordset") Set comData16 = CreateObject("ADODB.Command") Dim snpData17 Dim comData17 Set snpData17 = CreateObject("ADODB.Recordset") Set comData17 = CreateObject("ADODB.Command") Dim snpData18 Dim comData18 Set snpData18 = CreateObject("ADODB.Recordset") Set comData18 = CreateObject("ADODB.Command") Dim snpData19 Dim comData19 Set snpData19 = CreateObject("ADODB.Recordset") Set comData19 = CreateObject("ADODB.Command") Dim snpData20 Dim comData20 Set snpData20 = CreateObject("ADODB.Recordset") Set comData20 = CreateObject("ADODB.Command") On Error Resume Next strUsername = "MyUsername" strPassword = "MyPassword" strDatabase = "MyDB" dbDatabase.ConnectionString = "Provider=OraOLEDB.Oracle;Data Source=" & strDatabase & ";User ID=" & strUsername & ";Password=" & strPassword & ";" dbDatabase.Open 'Should verify that the connection attempt was successful, but I will leave that for someone else to code 'dbDatabase.BeginTrans 'Transaction Committed and NO Records were Affected, Need to determine transaction start dbDatabase.CommitTrans 'dbDatabase.BeginTrans Set comData3 = CreateObject("ADODB.Command") strSQL = "INSERT INTO EMPLOYEE_RECORD_TEST(" & vbCrLf strSQL = strSQL & " EMPLOYEE_ID," & vbCrLf strSQL = strSQL & " SHIFT_DATE," & vbCrLf strSQL = strSQL & " INDIRECT_ID)" & vbCrLf strSQL = strSQL & "VALUES(" & vbCrLf strSQL = strSQL & " ?," & vbCrLf strSQL = strSQL & " ?," & vbCrLf strSQL = strSQL & " ?)" With comData3 'Set up the command properties .CommandText = strSQL .CommandType = adCmdText .CommandTimeout = 30 .ActiveConnection = dbDatabase 'Bind variables will be defined below, if there are no bind variable, uncomment the next line and add the looping construct End With 'comData3.Execute With comData3 .Parameters.Append .CreateParameter("B1", adChar, adParamInput, 4, "TEST") .Parameters.Append .CreateParameter("B2", adDate, adParamInput, 7, "3/11/2010 0:0:0") .Parameters.Append .CreateParameter("B3", adChar, adParamInput, 3, "HOL") End With comData3("B1") = "TEST" comData3("B2") = cDate("3/11/2010 0:0:0") comData3("B3") = "HOL" comData3.Execute 'Transaction Rolled Back and Records Should have been Affected, Need to determine transaction start dbDatabase.RollbackTrans 'dbDatabase.BeginTrans 'Cursor 2 Closing If snpData2.State = 1 Then snpData2.Close End If Set comData2 = Nothing Set comData2 = CreateObject("ADODB.Command") strSQL = "SELECT DISTINCT" & vbCrLf strSQL = strSQL & " EMPLOYEE_ID" & vbCrLf strSQL = strSQL & "FROM" & vbCrLf strSQL = strSQL & " EMPLOYEE_RECORD_TEST" & vbCrLf strSQL = strSQL & "WHERE" & vbCrLf strSQL = strSQL & " SHIFT_DATE>= ?" & vbCrLf strSQL = strSQL & " AND INDIRECT_ID= ?" & vbCrLf strSQL = strSQL & "ORDER BY" & vbCrLf strSQL = strSQL & " EMPLOYEE_ID" With comData2 'Set up the command properties .CommandText = strSQL .CommandType = adCmdText .CommandTimeout = 30 .ActiveConnection = dbDatabase 'Bind variables will be defined below, if there are no bind variable, uncomment the next line and add the looping construct End With 'Set snpData2 = comData2.Execute 'Cursor 4 Closing If snpData4.State = 1 Then snpData4.Close End If Set comData4 = Nothing 'Cursor 4 Closing If snpData4.State = 1 Then snpData4.Close End If Set comData4 = Nothing 'Cursor 4 Closing If snpData4.State = 1 Then snpData4.Close End If Set comData4 = Nothing With comData2 .Parameters.Append .CreateParameter("B1", adDate, adParamInput, 7, "12/11/2009 0:0:0") .Parameters.Append .CreateParameter("B2", adChar, adParamInput, 3, "VAC") End With comData2("B1") = cDate("12/11/2009 0:0:0") comData2("B2") = "VAC" Set snpData2 = comData2.Execute If Not (snpData2 Is Nothing) Then Do While Not snpData2.EOF snpData2.MoveNext Loop End If 'Cursor 4 Closing If snpData4.State = 1 Then snpData4.Close End If Set comData4 = Nothing Set comData4 = CreateObject("ADODB.Command") strSQL = "SELECT" & vbCrLf strSQL = strSQL & " SUM(DECODE(TO_CHAR(SHIFT_DATE,'D'),'2',1,0)) MON_COUNT," & vbCrLf strSQL = strSQL & " SUM(DECODE(TO_CHAR(SHIFT_DATE,'D'),'3',1,0)) TUE_COUNT," & vbCrLf strSQL = strSQL & " SUM(DECODE(TO_CHAR(SHIFT_DATE,'D'),'4',1,0)) WED_COUNT," & vbCrLf strSQL = strSQL & " SUM(DECODE(TO_CHAR(SHIFT_DATE,'D'),'5',1,0)) THU_COUNT," & vbCrLf strSQL = strSQL & " SUM(DECODE(TO_CHAR(SHIFT_DATE,'D'),'6',1,0)) FRI_COUNT" & vbCrLf strSQL = strSQL & "FROM" & vbCrLf strSQL = strSQL & " EMPLOYEE_RECORD_TEST" & vbCrLf strSQL = strSQL & "WHERE" & vbCrLf strSQL = strSQL & " EMPLOYEE_ID= ?" & vbCrLf strSQL = strSQL & " AND INDIRECT_ID= ?" With comData4 'Set up the command properties .CommandText = strSQL .CommandType = adCmdText .CommandTimeout = 30 .ActiveConnection = dbDatabase 'Bind variables will be defined below, if there are no bind variable, uncomment the next line and add the looping construct End With 'Set snpData4 = comData4.Execute With comData4 .Parameters.Append .CreateParameter("B1", adChar, adParamInput, 4, "ERIC") .Parameters.Append .CreateParameter("B2", adChar, adParamInput, 6, "EXCUSE") End With comData4("B1") = "ERIC" comData4("B2") = "EXCUSE" Set snpData4 = comData4.Execute If Not (snpData4 Is Nothing) Then Do While Not snpData4.EOF snpData4.MoveNext Loop End If 'Cursor 5 Closing If snpData5.State = 1 Then snpData5.Close End If Set comData5 = Nothing comData4("B1") = "ERIC" comData4("B2") = "ABS" Set snpData4 = comData4.Execute If Not (snpData4 Is Nothing) Then Do While Not snpData4.EOF snpData4.MoveNext Loop End If 'Cursor 5 Closing If snpData5.State = 1 Then snpData5.Close End If Set comData5 = Nothing comData4("B1") = "JOE" comData4("B2") = "EXCUSE" Set snpData4 = comData4.Execute If Not (snpData4 Is Nothing) Then Do While Not snpData4.EOF snpData4.MoveNext Loop End If comData4("B1") = "JOE" comData4("B2") = "ABS" Set snpData4 = comData4.Execute If Not (snpData4 Is Nothing) Then Do While Not snpData4.EOF snpData4.MoveNext Loop End If comData4("B1") = "MIKE" comData4("B2") = "EXCUSE" Set snpData4 = comData4.Execute If Not (snpData4 Is Nothing) Then Do While Not snpData4.EOF snpData4.MoveNext Loop End If comData4("B1") = "MIKE" comData4("B2") = "ABS" Set snpData4 = comData4.Execute If Not (snpData4 Is Nothing) Then Do While Not snpData4.EOF snpData4.MoveNext Loop End If comData4("B1") = "SAM" comData4("B2") = "EXCUSE" Set snpData4 = comData4.Execute If Not (snpData4 Is Nothing) Then Do While Not snpData4.EOF snpData4.MoveNext Loop End If comData4("B1") = "SAM" comData4("B2") = "ABS" Set snpData4 = comData4.Execute If Not (snpData4 Is Nothing) Then Do While Not snpData4.EOF snpData4.MoveNext Loop End If 'Transaction Committed and NO Records were Affected, Need to determine transaction start dbDatabase.CommitTrans 'dbDatabase.BeginTrans 'Cursor 5 Closing If snpData5.State = 1 Then snpData5.Close End If Set comData5 = Nothing '************************************************************* 'Maximum Recordset Number Used is 5 - Adjust the Code at the Start Accordingly '************************************************************* If snpData1.State = 1 Then snpData1.Close End If Set snpData1 = Nothing If snpData2.State = 1 Then snpData2.Close End If Set snpData2 = Nothing If snpData3.State = 1 Then snpData3.Close End If Set snpData3 = Nothing If snpData4.State = 1 Then snpData4.Close End If Set snpData4 = Nothing If snpData5.State = 1 Then snpData5.Close End If Set snpData5 = Nothing Set comData1 = Nothing Set comData2 = Nothing Set comData3 = Nothing Set comData4 = Nothing Set comData5 = Nothing dbDatabase.Close Set dbDatabase = Nothing End Sub (TraceToVBSOutput.vbs – save as TraceToVBSOutput.vbs) If you compare the original TestScript.vbs with the above output, we see that the two scripts are similar, but with a couple of distinct differences: It is not necessarily easy to determine when a transaction starts, but it is possible to determine when a transaction ends. The script that reads the trace file inserts ‘dbDatabase.BeginTrans where it believes that a transaction should start – remove the ‘ if that is the correct starting point for the transaction. Looping structures with nested SQL statements (the retrieval of the employee list from the EMPLOYEE_RECORD_TEST table and the probing of matching rows for each of those employees) cannot be reproduced automatically – you will have to recognize when one SQL statement is feeding the bind variable values of a second SQL statement. The VBS script assumes that up to 20 cursors will be open at any one time, but will automatically handle many more than 20 simultaneously open cursors. The resulting VBS file should be cleaned up to remove the unneeded comData and snpData objects. There are spurious snpDatan.Close statements – see the suggestions for improvements. SQL statements submitted without bind variables will not have code written to execute those statements in the generated VBS file – see the suggestions for improvements. Bind variables that are submitted as VARCHAR (adVarchar) are written to the trace file as if the bind variables were declared as CHAR (adChar) – while this does not appear to cause a problem, it might appear to be an unexpected change when comparing the test script with the automatically generated script. Suggestions for improvement: Recognize the EXEC line in the 10046 trace and use that to actually indicate that a SQL statement should execute in the generated script, rather than executing the SQL statement in response to the submission of bind variables. Allow submitting the source trace file name and the destination (generated) VBS filename on the command line. Allow submitting the username, password, and database name on the command line or in a web-based user interface. Correct the script so that it does not attempt to close recordsets when those recordsets were never opened at dep=0 – this is caused by the script seeing a recursive SQL statement that is preparing to open with that cursor number. The VBS script that converts 10046 trace files to VBS script files may be downloaded here: TraceToVBS.vbs (save as TraceToVBS.vbs). There may be bugs in the script, but it should be close enough to provide some degree of educational benefit. Related Blog Articles: 10046 Extended SQL Trace Interpretation Automated DBMS_XPLAN, Trace, and Send to Excel Database Inpector Gadget Simple VBS Script to Retrieve Data from Oracle Toy Project for Performance Tuning 2 Comments : Leave a Comment » Categories : Bind Variable, Performance, VBS Impact of the TRUNC Function on an Indexed Date Column 8 03 2010 March 8, 2010 A recent email from an ERP user’s group mailing list reminded me of a small problem in that ERP program’s modules related to the DATE columns in several of its tables. In DATE columns that should only contain a date component, rows will occasionally be inserted by one of the ERP program’s modules with a date and time component, for example ’08-MAR-2010 13:01:13′ rather than just ’08-MAR-2010 00:00:00′. This bug, or feature, leads to unexpected performance issues when normal B*Tree indexes are present on that date column. To work around the time component in the DATE type columns, the ERP program modules frequently uses a technique like this to perform a comparisons on only the date component of a DATE type columns: SELECT * FROM T3 WHERE TRUNC(DATE_COLUMN) = :d1; In the above D1 is a bind variable. On occasion, the ERP program will instead pass in the date value as a constant/literal rather than as a bind variable. What is wrong with the above syntax? Would the above syntax be considered a bug if the DATE_COLUMN column had a normal B*Tree index? Is there a better way to retrieve the required rows? Incidentally, I started re-reading the book “Troubleshooting Oracle Performance” and I encountered a couple of interesting sentences on page 7 that seem to address this issue: “For all intents and purposes, an application experiencing poor performance is no worse [should probably state no better] than an application failing to fulfill its functional requirements. In both situations, the application is useless.” Let’s try a couple of experiments to see why the above SQL statement requires improvement. First, we will create table T2 that will serve as a nearly sequential ordered row source with a small amount of randomization introduced into the data by the DBMS_RANDOM function. This row source will be used to help duplicate the essentially random arrival rate of transactions into our T3 test table: CREATE TABLE T2 AS SELECT DBMS_RANDOM.VALUE(0,0.55555)+ROWNUM/10000 DAYS FROM (SELECT ROWNUM RN FROM DUAL CONNECT BY LEVEL<=10000) V1, (SELECT ROWNUM RN FROM DUAL CONNECT BY LEVEL<=10000) V2; The date column in our T3 table is derived from SYSDATE, so ideally the rows should be in order by the DAYS column in table T2. In a production environment, on rare occasion someone will slip in a row that is not in sequential order through an edit of the DATE column for a row, so we should be able to simulate that slight randomness by creating another table from table T2 before generating table T3 (the rows will fill the table blocks with an occasional row that is out of date sequence): CREATE TABLE T2_ORDERED NOLOGGING AS SELECT DAYS FROM T2 ORDER BY DAYS; For our simulation, we have a final problem that needs to be addressed. The volume of data entered on a Saturday in the production database is less than that for a Monday through Friday, and the volume of data entered on a Sunday is less than that entered on a Saturday. To add just a little more randomness, we will insert the rows into table T3 based on the following criteria: 90% chance of a row from T2_ORDERED being included if the date falls on a Monday through Friday 20% chance of a row from T2_ORDERED being included if the date falls on a Saturday 10% chance of a row from T2_ORDERED being included if the date falls on a Sunday The SQL statement to build table T3 follows: CREATE TABLE T3 NOLOGGING AS SELECT DAYS+TO_DATE('01-JAN-1990','DD-MON-YYYY') C1, DAYS+TO_DATE('01-JAN-1990','DD-MON-YYYY') C2, LPAD('A',255,'A') C3 FROM T2_ORDERED WHERE DECODE(TO_CHAR(DAYS+TO_DATE('01-JAN-1990','DD-MON-YYYY'),'D'),'1',0.9,'7',0.8,0.1)USER,TABNAME=>'T3',CASCADE=>TRUE) ALTER TABLE T3 MODIFY C1 NOT NULL; ALTER TABLE T3 MODIFY C2 NOT NULL; Let’s check the data distribution: SELECT COUNT(*) NUM_ROWS, SUM(DECODE(TO_CHAR(C1,'D'),'6',1,0)) FRIDAYS, SUM(DECODE(TO_CHAR(C1,'D'),'7',1,0)) SATURDAYS, SUM(DECODE(TO_CHAR(C1,'D'),'1',1,0)) SUNDAYS FROM T3; NUM_ROWS FRIDAYS SATURDAYS SUNDAYS ---------- ---------- ---------- ---------- 68,579,287 12,858,100 2,855,164 1,428,569 From the above we are able to determine that roughly 18.7% of the rows have a date that is on a Friday, roughly 4.2% of the rows have a date that is on a Saturday, and 2.1% of the rows are on a Sunday. This test will be performed on Oracle Database 11.2.0.1 with the __DB_CACHE_SIZE hidden parameter floating at roughly 7,381,975,040 (6.875GB). I will use my toy project for performance tuning to submit the SQL statements and retrieve the DBMS_XPLAN output, but the same could be accomplished with just SQL*Plus (most tests can also be performed using my Automated DBMS_XPLAN tool). Let’s start simple, we will start with a simple SQL statement to retrieve the rows with today’s date (March 8, 2010) using literals against the indexed column. I will execute each SQL statement twice to take advantage of any previously cached blocks in the buffer cache, and eliminate the time consumed by the hard parse: SELECT C1, C2, C3 FROM T3 WHERE TRUNC(C2) = TO_DATE('08-MAR-2010','DD-MON-YYYY'); SQL_ID 3us49wsdzdun3, child number 1 ------------------------------------- SELECT C1, C2, C3 FROM T3 WHERE TRUNC(C2) = TO_DATE('08-MAR-2010','DD-MON-YYYY') Plan hash value: 4161002650 --------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | --------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 9114 |00:02:31.93 | 2743K| 2743K| |* 1 | TABLE ACCESS FULL| T3 | 1 | 9114 | 9114 |00:02:31.93 | 2743K| 2743K| --------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(TRUNC(INTERNAL_FUNCTION("C2"))=TO_DATE(' 2010-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss')) Note ----- - cardinality feedback used for this statement Roughly 2 minutes and 32 seconds. Notice the Note at the bottom of the DBMS_XPLAN output, cardinality feedback (apparently not documented) is a new feature in Oracle Database 11.2.0.1 (see here for a related blog article). The first execution required 2 minutes and 33 seconds, but a predicted cardinality of 685,000 rows (1% of the total) was returned for the first execution. The second execution generated a second child cursor with a corrected cardinality estimate based on the actual number of rows returned during the first execution. 2 minutes and 32 seconds is not bad, unless this is an OLTP application and an end user is waiting for the application to return the rows. Let’s try again with a modified, equivalent SQL statement, again executing the SQL statement twice: SELECT C1, C2, C3 FROM T3 WHERE C2 >= TO_DATE('08-MAR-2010','DD-MON-YYYY') AND C2 < TO_DATE('08-MAR-2010','DD-MON-YYYY')+1; SQL_ID c7jfpa0rpt95a, child number 0 ------------------------------------- SELECT C1, C2, C3 FROM T3 WHERE C2 >= TO_DATE('08-MAR-2010','DD-MON-YYYY') AND C2 < TO_DATE('08-MAR-2010','DD-MON-YYYY')+1 Plan hash value: 4176467757 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 9114 |00:00:00.02 | 575 | | 1 | TABLE ACCESS BY INDEX ROWID| T3 | 1 | 6859 | 9114 |00:00:00.02 | 575 | |* 2 | INDEX RANGE SCAN | IND_T3_C2 | 1 | 6859 | 9114 |00:00:00.01 | 118 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C2">=TO_DATE(' 2010-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "C2"=TO_DATE(' 2010-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "C2"<=TO_DATE(' 2010-03-08 23:59:59', 'syyyy-mm-dd hh24:mi:ss')) 0.02 seconds again, and the estimated number of rows is roughly the same as we achieved with the previous SQL statement. By checking the Predicate Information section of the DBMS_XPLAN output we see that the optimizer has transformed the BETWEEN syntax into roughly the same syntax as was used in the previous SQL statement. Let’s try again with bind variables (the bind variable names are automatically changed by ADO into generic names, and that is why the bind variable appears in the execution plan as :1 rather than :d1): SELECT C1, C2, C3 FROM T3 WHERE TRUNC(C2) = :d1; SQL_ID cub25jm7y8zck, child number 0 ------------------------------------- SELECT C1, C2, C3 FROM T3 WHERE TRUNC(C2) = :1 Plan hash value: 4161002650 --------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | --------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 9114 |00:02:33.37 | 2743K| 2743K| |* 1 | TABLE ACCESS FULL| T3 | 1 | 685K| 9114 |00:02:33.37 | 2743K| 2743K| --------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(TRUNC(INTERNAL_FUNCTION("C2"))=:1) Notice that our friendly note about Cardinality Feedback did not appear this time, and that the cardinality estimate was not corrected when the SQL statement was executed for the second time, even though bind variable peeking did happen. The incorrect cardinality estimate would not have changed the execution plan for this SQL statement, but could impact the execution plan if table T3 were joined to another table. Let’s try the other equivalent SQL statement with bind variables: SELECT C1, C2, C3 FROM T3 WHERE C2 >= :d1 AND C2 < :d2 +1; SQL_ID 9j2a54zbzb9cz, child number 0 ------------------------------------- SELECT C1, C2, C3 FROM T3 WHERE C2 >= :1 AND C2 < :2 +1 Plan hash value: 3025660695 ---------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | ---------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 9114 |00:00:00.02 | 575 | |* 1 | FILTER | | 1 | | 9114 |00:00:00.02 | 575 | | 2 | TABLE ACCESS BY INDEX ROWID| T3 | 1 | 6859 | 9114 |00:00:00.02 | 575 | |* 3 | INDEX RANGE SCAN | IND_T3_C2 | 1 | 6859 | 9114 |00:00:00.01 | 118 | ---------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:1<:2+1) 3 - access("C2">=:1 AND "C2"<:2+1) The optimizer estimated that 6,859 rows would be returned, just as it did when we used literals in the SQL statement because of bind variable peeking. In case you are wondering, the same estimated row cardinality was returned when the D2 bind variable was set to ’09-MAR-2010′ in the application and the +1 was removed from the SQL statement. Quite the problem we caused by pretending to not understand the impact of using a function in the WHERE clause on an indexed column. We could create a function based index to work around the problem of the application programmers not knowing how to specify a specific date without using the TRUNC function on a DATE column: CREATE INDEX IND_T3_C2_FBI ON T3(TRUNC(C2)); ALTER SYSTEM FLUSH SHARED_POOL; But is creating a function based index the best approach, or have we just created another problem rather than attacking the root cause of the original problem? We now have two indexes on the same column that need to be updated every time a row is inserted or deleted in table T3, and also maintained every time that column is updated (even when updated with the same value). Let’s experiment with the function based index. SELECT C1, C2, C3 FROM T3 WHERE TRUNC(C2) = TO_DATE('08-MAR-2010','DD-MON-YYYY'); SQL_ID 3us49wsdzdun3, child number 1 ------------------------------------- SELECT C1, C2, C3 FROM T3 WHERE TRUNC(C2) = TO_DATE('08-MAR-2010','DD-MON-YYYY') Plan hash value: 3662266936 ------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | ------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 9114 |00:00:00.01 | 576 | | 1 | TABLE ACCESS BY INDEX ROWID| T3 | 1 | 9114 | 9114 |00:00:00.01 | 576 | |* 2 | INDEX RANGE SCAN | IND_T3_C2_FBI | 1 | 9114 | 9114 |00:00:00.01 | 119 | ------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("T3"."SYS_NC00004$"=TO_DATE(' 2010-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss')) Note ----- - cardinality feedback used for this statement Cardinality feedback again helps out the cardinality estimate on the second execution, but look at the Predicate Information section of the execution plan. We have now increased the difficulty of walking through a complicated execution plan with the help of the Predicate Information section of the execution plan to see how the predicates in the WHERE clause are applied to the execution plan. Not so bad, right? What happens if this column C2 is joined to a column in another table, or even specified as being equal to column C1 in this table? Let’s take a look: SELECT C1, C2, C3 FROM T3 WHERE C2 >= TO_DATE('08-MAR-2010','DD-MON-YYYY') AND C2 < TO_DATE('08-MAR-2010','DD-MON-YYYY')+1 AND C2=C1; SQL_ID 27rqhg1mpmzt9, child number 1 ------------------------------------- SELECT C1, C2, C3 FROM T3 WHERE C2 >= TO_DATE('08-MAR-2010','DD-MON-YYYY') AND C2 < TO_DATE('08-MAR-2010','DD-MON-YYYY')+1 AND C2=C1 Plan hash value: 4176467757 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 9114 |00:00:00.01 | 575 | |* 1 | TABLE ACCESS BY INDEX ROWID| T3 | 1 | 9114 | 9114 |00:00:00.01 | 575 | |* 2 | INDEX RANGE SCAN | IND_T3_C2 | 1 | 9114 | 9114 |00:00:00.01 | 118 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(("C2"="C1" AND "C1">=TO_DATE(' 2010-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "C1"=TO_DATE(' 2010-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "C2"USER,TABNAME=>'T3',CASCADE=>TRUE) I received the following after several minutes: BEGIN DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T3',CASCADE=>TRUE); END; * ERROR at line 1: ORA-00600: internal error code, arguments: [15851], [3], [2], [1], [1], [], [], [], [], [], [], [] ORA-06512: at "SYS.DBMS_STATS", line 20337 ORA-06512: at "SYS.DBMS_STATS", line 20360 ORA-06512: at line 1 The same error appeared without the CASCADE option, but a call to collect statistics on the indexes for the table, as well as other tables completes successfully. I may look at this problem again later. Continuing, we will create another table: CREATE TABLE T4 NOLOGGING AS SELECT * FROM T3 WHERE C2 BETWEEN TO_DATE('01-JAN-2010','DD-MON-YYYY') AND TO_DATE('08-MAR-2010','DD-MON-YYYY') + (1-1/24/60/60); CREATE INDEX IND_T4_C2 ON T4(C2); EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T4',CASCADE=>TRUE) Before we start, let’s take a look at the disk space used by the objects and the automatically allocated extent sizes: SELECT SEGMENT_NAME SEGMENT, (SUM(BYTES))/1048576 TOTAL_MB FROM DBA_EXTENTS WHERE OWNER=USER AND SEGMENT_NAME IN ('IND_T3_C2','IND_T3_C2_FBI','T3','T4','IND_T4_C2') GROUP BY SEGMENT_NAME ORDER BY SEGMENT_NAME; SEGMENT TOTAL_MB --------------- ---------- IND_T3_C2 1469 IND_T3_C2_FBI 1472 IND_T4_C2 10 T3 21480 T4 144 SELECT SEGMENT_NAME SEGMENT, COUNT(*) EXTENTS, BYTES/1024 EXT_SIZE_KB, (COUNT(*) * BYTES)/1048576 TOTAL_MB FROM DBA_EXTENTS WHERE OWNER=USER AND SEGMENT_NAME IN ('IND_T3_C2','IND_T3_C2_FBI','T3','T4','IND_T4_C2') GROUP BY SEGMENT_NAME, BYTES ORDER BY SEGMENT_NAME, BYTES; SEGMENT EXTENTS EXT_SIZE_KB TOTAL_MB --------------- ---------- ----------- ---------- IND_T3_C2 16 64 1 IND_T3_C2 63 1024 63 IND_T3_C2 120 8192 960 IND_T3_C2 1 27648 27 IND_T3_C2 1 34816 34 IND_T3_C2 6 65536 384 IND_T3_C2_FBI 16 64 1 IND_T3_C2_FBI 63 1024 63 IND_T3_C2_FBI 120 8192 960 IND_T3_C2_FBI 7 65536 448 IND_T4_C2 16 64 1 IND_T4_C2 9 1024 9 T3 16 64 1 T3 63 1024 63 T3 120 8192 960 T3 1 19456 19 T3 1 43008 42 T3 1 44032 43 T3 318 65536 20352 T4 16 64 1 T4 63 1024 63 T4 10 8192 80 Table T3 is using about 21GB of space while table T4 is using about 144MB of space. We occasionally received an extent size that is not a power of 2 in size – a bit unexpected. Let’s try a couple of SQL statements that access the two tables: SELECT T3.C1, T3.C2, T4.C3 FROM T3, T4 WHERE TRUNC(T3.C2) = TO_DATE('08-MAR-2010','DD-MON-YYYY') AND T3.C2=T4.C2; SQL_ID f2v7cf7w2bwqq, child number 0 ------------------------------------- SELECT T3.C1, T3.C2, T4.C3 FROM T3, T4 WHERE TRUNC(T3.C2) = TO_DATE('08-MAR-2010','DD-MON-YYYY') AND T3.C2=T4.C2 Plan hash value: 1631978485 -------------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem | -------------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 10044 |00:00:00.38 | 18622 | 25 | | | | |* 1 | HASH JOIN | | 1 | 7095 | 10044 |00:00:00.38 | 18622 | 25 | 1223K| 1223K| 1593K (0)| | 2 | TABLE ACCESS BY INDEX ROWID| T3 | 1 | 6857 | 9114 |00:00:00.04 | 394 | 25 | | | | |* 3 | INDEX RANGE SCAN | IND_T3_C2_FBI | 1 | 6857 | 9114 |00:00:00.03 | 28 | 25 | | | | | 4 | TABLE ACCESS FULL | T4 | 1 | 452K| 452K|00:00:00.12 | 18228 | 0 | | | | -------------------------------------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - access("T3"."C2"="T4"."C2") 3 - access("T3"."SYS_NC00004$"=TO_DATE(' 2010-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss')) The above used a full table scan on table T4, and you will notice that a filter predicate is not applied to table T4 to reduce the number of rows entering the hash join. Transitive closure did not take place. Let’s try again with the SQL statement with the other syntax that does not use the TRUNC function, nor the function based index: SELECT T3.C1, T3.C2, T4.C3 FROM T3, T4 WHERE T3.C2 BETWEEN TO_DATE('08-MAR-2010','DD-MON-YYYY') AND TO_DATE('08-MAR-2010','DD-MON-YYYY') + (1-1/24/60/60) AND T3.C2=T4.C2; SQL_ID 5swqbjak147vk, child number 0 ------------------------------------- SELECT T3.C1, T3.C2, T4.C3 FROM T3, T4 WHERE T3.C2 BETWEEN TO_DATE('08-MAR-2010','DD-MON-YYYY') AND TO_DATE('08-MAR-2010','DD-MON-YYYY') + (1-1/24/60/60) AND T3.C2=T4.C2 Plan hash value: 3991319422 ---------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem | ---------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 10044 |00:00:00.06 | 983 | 1 | | | | |* 1 | HASH JOIN | | 1 | 6761 | 10044 |00:00:00.06 | 983 | 1 | 1223K| 1223K| 1618K (0)| | 2 | TABLE ACCESS BY INDEX ROWID| T3 | 1 | 6860 | 9114 |00:00:00.01 | 393 | 0 | | | | |* 3 | INDEX RANGE SCAN | IND_T3_C2 | 1 | 6860 | 9114 |00:00:00.01 | 27 | 0 | | | | | 4 | TABLE ACCESS BY INDEX ROWID| T4 | 1 | 6762 | 9114 |00:00:00.03 | 590 | 1 | | | | |* 5 | INDEX RANGE SCAN | IND_T4_C2 | 1 | 6762 | 9114 |00:00:00.01 | 127 | 1 | | | | ---------------------------------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - access("T3"."C2"="T4"."C2") 3 - access("T3"."C2">=TO_DATE(' 2010-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "T3"."C2"<=TO_DATE(' 2010-03-08 23:59:59', 'syyyy-mm-dd hh24:mi:ss')) 5 - access("T4"."C2">=TO_DATE(' 2010-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "T4"."C2"<=TO_DATE(' 2010-03-08 23:59:59', 'syyyy-mm-dd hh24:mi:ss')) Notice this time that transitive closure happened, allowing the optimizer to take advantage of the IND_T4_C2 index on table T4. You are probably thinking, it must be that we need a function based index on the C2 column of table T4 also to allow transitive closure to happen. Let’s try: CREATE INDEX IND_T4_C2_FBI ON T4(TRUNC(C2)); ALTER SYSTEM FLUSH SHARED_POOL; Now our SQL statement again: SELECT T3.C1, T3.C2, T4.C3 FROM T3, T4 WHERE TRUNC(T3.C2) = TO_DATE('08-MAR-2010','DD-MON-YYYY') AND T3.C2=T4.C2; SQL_ID f2v7cf7w2bwqq, child number 0 ------------------------------------- SELECT T3.C1, T3.C2, T4.C3 FROM T3, T4 WHERE TRUNC(T3.C2) = TO_DATE('08-MAR-2010','DD-MON-YYYY') AND T3.C2=T4.C2 Plan hash value: 1631978485 ----------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | OMem | 1Mem | Used-Mem | ----------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 10044 |00:00:00.33 | 18622 | | | | |* 1 | HASH JOIN | | 1 | 7095 | 10044 |00:00:00.33 | 18622 | 1223K| 1223K| 1584K (0)| | 2 | TABLE ACCESS BY INDEX ROWID| T3 | 1 | 6857 | 9114 |00:00:00.01 | 394 | | | | |* 3 | INDEX RANGE SCAN | IND_T3_C2_FBI | 1 | 6857 | 9114 |00:00:00.01 | 28 | | | | | 4 | TABLE ACCESS FULL | T4 | 1 | 452K| 452K|00:00:00.11 | 18228 | | | | ----------------------------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - access("T3"."C2"="T4"."C2") 3 - access("T3"."SYS_NC00004$"=TO_DATE(' 2010-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss')) As expected, the function based index on column C4 of table T4 was not used because transitive closure did not happen. Do we still want to do it the wrong way? The execution time could have been much longer than 0.33 seconds, of course, if table T4 were much larger and a large number of physical reads were required. Try again using a larger table T4: DROP TABLE T4 PURGE; CREATE TABLE T4 NOLOGGING AS SELECT * FROM T3 WHERE C2 BETWEEN TO_DATE('01-JAN-2000','DD-MON-YYYY') AND TO_DATE('08-MAR-2010','DD-MON-YYYY') + (1-1/24/60/60); CREATE INDEX IND_T4_C2 ON T4(C2); EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T4',CASCADE=>TRUE) Table T4 now requires about 7.9GB of disk space. Now a range scan that accesses tables T3 and T4 (each SQL statement is executed twice, with the last execution plan reported): SELECT T3.C1, T3.C2, T4.C3 FROM T3, T4 WHERE TRUNC(T3.C2) BETWEEN TO_DATE('08-MAR-2009','DD-MON-YYYY') AND TO_DATE('01-JUL-2009','DD-MON-YYYY') AND T3.C2=T4.C2; SQL_ID 2d4f5x92axqgn, child number 0 ------------------------------------- SELECT T3.C1, T3.C2, T4.C3 FROM T3, T4 WHERE TRUNC(T3.C2) BETWEEN TO_DATE('08-MAR-2009','DD-MON-YYYY') AND TO_DATE('01-JUL-2009','DD-MON-YYYY') AND T3.C2=T4.C2 Plan hash value: 1631978485 -------------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem | -------------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 874K|00:00:33.24 | 1062K| 302K| | | | |* 1 | HASH JOIN | | 1 | 849K| 874K|00:00:33.24 | 1062K| 302K| 33M| 5591K| 50M (0)| | 2 | TABLE ACCESS BY INDEX ROWID| T3 | 1 | 802K| 795K|00:00:00.56 | 33957 | 0 | | | | |* 3 | INDEX RANGE SCAN | IND_T3_C2_FBI | 1 | 802K| 795K|00:00:00.20 | 2115 | 0 | | | | | 4 | TABLE ACCESS FULL | T4 | 1 | 25M| 25M|00:00:17.13 | 1028K| 302K| | | | -------------------------------------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - access("T3"."C2"="T4"."C2") 3 - access("T3"."SYS_NC00004$">=TO_DATE(' 2009-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "T3"."SYS_NC00004$"<=TO_DATE(' 2009-07-01 00:00:00', 'syyyy-mm-dd hh24:mi:ss')) Notice the full table scan of table T4. Now the other SQL statement: SELECT T3.C1, T3.C2, T4.C3 FROM T3, T4 WHERE T3.C2 BETWEEN TO_DATE('08-MAR-2009','DD-MON-YYYY') AND TO_DATE('01-JUL-2009','DD-MON-YYYY') + (1-1/24/60/60) AND T3.C2=T4.C2; SQL_ID 539d93k50ruz3, child number 0 ------------------------------------- SELECT T3.C1, T3.C2, T4.C3 FROM T3, T4 WHERE T3.C2 BETWEEN TO_DATE('08-MAR-2009','DD-MON-YYYY') AND TO_DATE('01-JUL-2009','DD-MON-YYYY') + (1-1/24/60/60) AND T3.C2=T4.C2 Plan hash value: 1243183227 ----------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem | ----------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 874K|00:00:05.80 | 85051 | 574 | | | | | 1 | MERGE JOIN | | 1 | 795K| 874K|00:00:05.80 | 85051 | 574 | | | | | 2 | TABLE ACCESS BY INDEX ROWID | T4 | 1 | 795K| 795K|00:00:02.43 | 51097 | 574 | | | | |* 3 | INDEX RANGE SCAN | IND_T4_C2 | 1 | 795K| 795K|00:00:00.41 | 10841 | 0 | | | | |* 4 | SORT JOIN | | 795K| 795K| 874K|00:00:02.00 | 33954 | 0 | 30M| 1977K| 26M (0)| | 5 | TABLE ACCESS BY INDEX ROWID| T3 | 1 | 795K| 795K|00:00:00.50 | 33954 | 0 | | | | |* 6 | INDEX RANGE SCAN | IND_T3_C2 | 1 | 795K| 795K|00:00:00.17 | 2114 | 0 | | | | ----------------------------------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 3 - access("T4"."C2">=TO_DATE(' 2009-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "T4"."C2"<=TO_DATE(' 2009-07-01 23:59:59', 'syyyy-mm-dd hh24:mi:ss')) 4 - access("T3"."C2"="T4"."C2") filter("T3"."C2"="T4"."C2") 6 - access("T3"."C2">=TO_DATE(' 2009-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "T3"."C2"<=TO_DATE(' 2009-07-01 23:59:59', 'syyyy-mm-dd hh24:mi:ss')) Notice that the above used the index on table T4, but performed a sort-merge join between the two tables. We are able to force a hash join, as was used with the other SQL statement, by applying a hint: SQL_ID b9q6tf6p6x2m0, child number 0 ------------------------------------- SELECT /*+ USE_HASH (T3 T4) */ T3.C1, T3.C2, T4.C3 FROM T3, T4 WHERE T3.C2 BETWEEN TO_DATE('08-MAR-2009','DD-MON-YYYY') AND TO_DATE('01-JUL-2009','DD-MON-YYYY') + (1-1/24/60/60) AND T3.C2=T4.C2 Plan hash value: 3991319422 ------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | OMem | 1Mem | Used-Mem | ------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 874K|00:00:03.60 | 85051 | | | | |* 1 | HASH JOIN | | 1 | 795K| 874K|00:00:03.60 | 85051 | 33M| 5591K| 50M (0)| | 2 | TABLE ACCESS BY INDEX ROWID| T3 | 1 | 795K| 795K|00:00:00.54 | 33954 | | | | |* 3 | INDEX RANGE SCAN | IND_T3_C2 | 1 | 795K| 795K|00:00:00.19 | 2114 | | | | | 4 | TABLE ACCESS BY INDEX ROWID| T4 | 1 | 795K| 795K|00:00:01.44 | 51097 | | | | |* 5 | INDEX RANGE SCAN | IND_T4_C2 | 1 | 795K| 795K|00:00:00.40 | 10841 | | | | ------------------------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - access("T3"."C2"="T4"."C2") 3 - access("T3"."C2">=TO_DATE(' 2009-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "T3"."C2"<=TO_DATE(' 2009-07-01 23:59:59', 'syyyy-mm-dd hh24:mi:ss')) 5 - access("T4"."C2">=TO_DATE(' 2009-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "T4"."C2"<=TO_DATE(' 2009-07-01 23:59:59', 'syyyy-mm-dd hh24:mi:ss')) 9.2 times faster (just 5.7 times faster without the hint) by not using the TRUNC function and function-based index combination. Are we able to just agree to do it the right way, or should I continue? Without the function based index we receive an execution plan like this: SQL_ID 2d4f5x92axqgn, child number 0 ------------------------------------- SELECT T3.C1, T3.C2, T4.C3 FROM T3, T4 WHERE TRUNC(T3.C2) BETWEEN TO_DATE('08-MAR-2009','DD-MON-YYYY') AND TO_DATE('01-JUL-2009','DD-MON-YYYY') AND T3.C2=T4.C2 Plan hash value: 1396201636 ------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem | ------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 874K|00:03:05.88 | 3771K| 3013K| | | | |* 1 | HASH JOIN | | 1 | 849K| 874K|00:03:05.88 | 3771K| 3013K| 33M| 5591K| 52M (0)| |* 2 | TABLE ACCESS FULL| T3 | 1 | 802K| 795K|00:02:34.36 | 2743K| 2743K| | | | | 3 | TABLE ACCESS FULL| T4 | 1 | 25M| 25M|00:00:15.72 | 1028K| 270K| | | | ------------------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - access("T3"."C2"="T4"."C2") 2 - filter((TRUNC(INTERNAL_FUNCTION("C2"))>=TO_DATE(' 2009-03-08 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND TRUNC(INTERNAL_FUNCTION("C2"))<=TO_DATE(' 2009-07-01 00:00:00', 'syyyy-mm-dd hh24:mi:ss'))) A full table scan of a 21GB table and 7.9GB table, with 795,000 rows from the large table and 25,000,000 rows from the small table entering the hash join – probably not too good for performance. Fix the performance bug in the application and let the user get back to counting the pencils in the pencil jar 9.2 times faster (or 51.6 times faster if there is no function based index). While you might not frequently join two tables on a DATE column as I have done in this demonstration, how common is it to store numeric data in a VARCHAR2 column, and then need to be able to compare those values with numbers stored in NUMBER type columns, with literals, or numeric bind variables? Comments : 2 Comments » Categories : Bind Variable, Performance, SQL Plan Cardinality Estimates Problem with 11.1.0.7 and 11.2.0.1 15 02 2010 February 15, 2010 Here is a fun test on Oracle Database 11.1.0.7 and 11.2.0.1 (all posted tests are from 11.2.0.1). First, we will create a table and collect statistics for the table with indexes on a couple of the columns and histograms on a couple of the columns: CREATE TABLE T1 ( C1 NUMBER NOT NULL, C2 NUMBER NOT NULL, C3 NUMBER NOT NULL, C4 NUMBER NOT NULL, C5 VARCHAR2(30) NOT NULL, C6 VARCHAR2(30) NOT NULL, FILLER VARCHAR2(200), PRIMARY KEY (C1)); INSERT INTO T1 SELECT ROWNUM, ROWNUM, TRUNC(ROWNUM/100+1), TRUNC(ROWNUM/100+1), CHR(65+TRUNC(ROWNUM/10000))||TRUNC(ROWNUM/100+1), CHR(65+TRUNC(ROWNUM/10000))||TRUNC(ROWNUM/100+1), LPAD('A',200,'A') FROM DUAL CONNECT BY LEVEL<=100000; COMMIT; CREATE INDEX IND_T1_C3 ON T1(C3); CREATE INDEX IND_T1_C4 ON T1(C4); CREATE INDEX IND_T1_C5 ON T1(C5); EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T1',CASCADE=>TRUE,ESTIMATE_PERCENT=>100,METHOD_OPT=>'FOR COLUMNS SIZE 254 C2, C4, C6') Just for a review of that GATHER_TABLE_STATS procedure, from the documentation: When gathering statistics on a table, DBMS_STATS gathers information about the data distribution of the columns within the table. The most basic information about the data distribution is the maximum value and minimum value of the column. However, this level of statistics may be insufficient for the optimizer’s needs if the data within the column is skewed. For skewed data distributions, histograms can also be created as part of the column statistics to describe the data distribution of a given column. Histograms are described in more details in “Viewing Histograms“. Histograms are specified using the METHOD_OPT argument of the DBMS_STATS gathering procedures. Oracle recommends setting the METHOD_OPT to FOR ALL COLUMNS SIZE AUTO. With this setting, Oracle Database automatically determines which columns require histograms and the number of buckets (size) of each histogram. You can also manually specify which columns should have histograms and the size of each histogram. The table created by the above script will have 100,000 rows with indexes on columns C3, C4, and C5. Columns C2, C4, and C6 will have histograms with 254 buckets. Let’s see the maximum values for the table columns (column C2 values are identical to C1, column C4 values are identical to C3, column C6 values are identical to C5)): SELECT MAX(C1) MAX_C1, MAX(C3) MAX_C3, MAX(C5) MAX_C5 FROM T1; MAX_C1 MAX_C3 MAX_C5 ------ ---------- ------ 100000 1001 K1001 The maximum value for column C3 is 1001 – most of the distinct values in that column have 100 matching rows, except for the value 1001 which has a single row. We will try a couple of tests with my VBS tool for automatically generating DBMS_XPLANs, with the DBMS_XPLAN Type set to “ALLSTATS LAST” and Stats Level set to “ALL“. First, we will try the maximum value for column C3 (and also C4): SELECT * FROM T1 WHERE C3=1001; The returned execution plan is displayed below (after this point the execution plan will be displayed directly below the SQL statement): SQL_ID 0vcvak7bgbdzt, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C3=1001 Plan hash value: 1220227203 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 1 |00:00:00.01 | 3 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 1000 | 1 |00:00:00.01 | 3 | |* 2 | INDEX RANGE SCAN | IND_T1_C3 | 1 | 400 | 1 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C3"=1001) OK, it was more or less expected that the cardinality estimate would be incorrect here because there is no histogram on column C3. It is a bit odd that the optimizer predicts that the index will return 400 rows that then causes 1000 rows to be returned from the table. Now let’s try the same SQL statement referencing column C4: SELECT * FROM T1 WHERE C4=1001; SQL_ID d3zfa447tsjrz, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C4=1001 Plan hash value: 7035821 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 1 |00:00:00.01 | 3 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 100 | 1 |00:00:00.01 | 3 | |* 2 | INDEX RANGE SCAN | IND_T1_C4 | 1 | 100 | 1 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C4"=1001) 100 is closer to the actual number of rows than were predicted for the same SQL statement using column C3, so the histogram probably helped. What happens when we specify values for C3 and C4 that exceed the maximum values in those columns? SELECT * FROM T1 WHERE C3=1101; SQL_ID 7hy399svng33n, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C3=1101 Plan hash value: 1220227203 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 2 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 1000 | 0 |00:00:00.01 | 2 | |* 2 | INDEX RANGE SCAN | IND_T1_C3 | 1 | 400 | 0 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C3"=1101) - SELECT * FROM T1 WHERE C4=1101; SQL_ID at676unwj7uk1, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C4=1101 Plan hash value: 7035821 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 2 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 90 | 0 |00:00:00.01 | 2 | |* 2 | INDEX RANGE SCAN | IND_T1_C4 | 1 | 90 | 0 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C4"=1101) In the above, the SQL statement that accesses column C3 continued to estimate the same number of rows would be returned when the value exceeded the maximum value for the column by roughly 10%. When the same restriction was applied to column C4, the optimizer predicted that 10% less rows would be returned. Interesting… SELECT * FROM T1 WHERE C3=1201; SQL_ID f94b21zwvsn11, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C3=1201 Plan hash value: 1220227203 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 2 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 1000 | 0 |00:00:00.01 | 2 | |* 2 | INDEX RANGE SCAN | IND_T1_C3 | 1 | 400 | 0 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C3"=1201) - SELECT * FROM T1 WHERE C4=1201; SQL_ID 4sf3hjx44u1sn, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C4=1201 Plan hash value: 7035821 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 2 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 80 | 0 |00:00:00.01 | 2 | |* 2 | INDEX RANGE SCAN | IND_T1_C4 | 1 | 80 | 0 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C4"=1201) Exceeding the maximum value by 20% returned the same cardinality estimate for the SQL statement using column C3 as seen earlier, while the cardinality estimate decreased by 20% for the SQL statement accessing column C4. SELECT * FROM T1 WHERE C3=1901; SQL_ID 86z7nbb5u2p26, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C3=1901 Plan hash value: 1220227203 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 2 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 1000 | 0 |00:00:00.01 | 2 | |* 2 | INDEX RANGE SCAN | IND_T1_C3 | 1 | 400 | 0 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C3"=1901) - SELECT * FROM T1 WHERE C4=1901; SQL_ID 08rg4uf562h4x, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C4=1901 Plan hash value: 7035821 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 2 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 10 | 0 |00:00:00.01 | 2 | |* 2 | INDEX RANGE SCAN | IND_T1_C4 | 1 | 10 | 0 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C4"=1901) When exceeding the maximum value by 90% we see the same pattern, the cardinality estimates for the first SQL statement were unaffected, while the cardinality estimate for the second SQL statement decreased by 90%. With a value of 2001 specified for column C4 the optimizer’s predicted cardinality decreased to 1 row. What about value ranges? SELECT * FROM T1 WHERE C3 BETWEEN 1101 AND 1201; SQL_ID 95zzq8vb523gf, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C3 BETWEEN 1101 AND 1201 Plan hash value: 1220227203 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 2 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 250 | 0 |00:00:00.01 | 2 | |* 2 | INDEX RANGE SCAN | IND_T1_C3 | 1 | 450 | 0 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C3">=1101 AND "C3"<=1201) Notice that the optimizer is estimating that 250 rows will be returned, which is less than the sum of the estimated 1,000 rows that would be returned for the value 1101 and the estimated 1,000 rows for value 1201. Let’s try again specifying column C4: SELECT * FROM T1 WHERE C4 BETWEEN 1101 AND 1201; SQL_ID 9t2fv8dcz7jsv, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C4 BETWEEN 1101 AND 1201 Plan hash value: 7035821 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 2 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 100 | 0 |00:00:00.01 | 2 | |* 2 | INDEX RANGE SCAN | IND_T1_C4 | 1 | 90 | 0 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C4">=1101 AND "C4"<=1201) Interesting – the index is predicted to return 90 rows which then causes the table to return 100 rows (edit: Oracle 10.2.0.2 predicts 90 rows for both the index and the table). Let’s try again specifying ranges that exceed the maximum values by 90% to 100%: SELECT * FROM T1 WHERE C3 BETWEEN 1901 AND 2001; SQL_ID 99btm350uvvbp, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C3 BETWEEN 1901 AND 2001 Plan hash value: 1220227203 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 2 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 250 | 0 |00:00:00.01 | 2 | |* 2 | INDEX RANGE SCAN | IND_T1_C3 | 1 | 450 | 0 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C3">=1901 AND "C3"<=2001) - SELECT * FROM T1 WHERE C4 BETWEEN 1901 AND 2001; SQL_ID bnwyf98m74q26, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C4 BETWEEN 1901 AND 2001 Plan hash value: 7035821 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 0 |00:00:00.01 | 2 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 100 | 0 |00:00:00.01 | 2 | |* 2 | INDEX RANGE SCAN | IND_T1_C4 | 1 | 10 | 0 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C4">=1901 AND "C4"<=2001) The first query is predicting the same number of rows will be returned as seen earlier. The second query is again predicting that 100 rows will be returned from the table as a result of the 10 predicted rows that will be returned from the index (edit: Oracle 10.2.0.2 predicts 10 rows for both the index and the table). When the range was changed to 1101 through 2001, the same cardinality estimates displayed for the range of 1101 to 1201 were again returned for both SQL statements. If the above accurately depicts what happens when the maximum recorded value for a column is exceeded, what might happen when statistics are not gathered on a regular basis? Hold that thought. Let’s try again using a value range for C3 and C4 that are not beyond the maximum values for the columns: SELECT * FROM T1 WHERE C3 BETWEEN 101 AND 201; SQL_ID 8jd9h693mbkvc, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C3 BETWEEN 101 AND 201 Plan hash value: 1220227203 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 10100 |00:00:00.02 | 547 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 250 | 10100 |00:00:00.02 | 547 | |* 2 | INDEX RANGE SCAN | IND_T1_C3 | 1 | 450 | 10100 |00:00:00.01 | 124 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C3">=101 AND "C3"<=201) - SELECT * FROM T1 WHERE C4 BETWEEN 101 AND 201; SQL_ID 2bk0njs0atfpw, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C4 BETWEEN 101 AND 201 Plan hash value: 7035821 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 10100 |00:00:00.02 | 547 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 10039 | 10100 |00:00:00.02 | 547 | |* 2 | INDEX RANGE SCAN | IND_T1_C4 | 1 | 10039 | 10100 |00:00:00.01 | 124 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C4">=101 AND "C4"<=201) Once again, the first SQL statement is estimating that the index range scan will return 450 rows which will then cause the table to return 250 rows. In actuality, both the index and the table return 10,100 rows (do we possibly have a statistics problem here?). The second query returns cardinality estimates that are closer to the actual number of rows returned – the histogram helped here. Let’s try again with a wider value range: SELECT * FROM T1 WHERE C3 BETWEEN 101 AND 1401; SQL_ID 0bapwrbn3ch8j, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C3 BETWEEN 101 AND 1401 Plan hash value: 1220227203 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 90001 |00:00:00.13 | 4865 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 250 | 90001 |00:00:00.13 | 4865 | |* 2 | INDEX RANGE SCAN | IND_T1_C3 | 1 | 450 | 90001 |00:00:00.04 | 1090 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C3">=101 AND "C3"<=1401) - SELECT * FROM T1 WHERE C4 BETWEEN 101 AND 1401; SQL_ID 3ppn43z5ukur6, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C4 BETWEEN 101 AND 1401 Plan hash value: 3617692013 ------------------------------------------------------------------------------------ | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | ------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 1 | | 90001 |00:00:00.05 | 4147 | |* 1 | TABLE ACCESS FULL| T1 | 1 | 89961 | 90001 |00:00:00.05 | 4147 | ------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(("C4">=101 AND "C4"<=1401)) Again, the cardinality estimates and execution plan are the same as before for the query using column C3, even though we are selecting 90% of the rows in the table. The query using column C4 switched to a full table scan, and has a much more accurate cardinality estimate. This same pattern holds true regardless of the values specified for the low and high ends of the range. Let’s switch back to SQL*Plus and try a couple of experiments with bind variables (note that I am actually submitting the SQL statements using my Toy Project so that I do not have to watch all of the rows scroll on the screen): VARIABLE N1 NUMBER VARIABLE N2 NUMBER ALTER SESSION SET STATISTICS_LEVEL='ALL'; EXEC :N1:=101 EXEC :N2:=1401 SET ARRAYSIZE 100 SELECT * FROM T1 WHERE C3 BETWEEN :N1 AND :N2; ... SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); SQL_ID 2ksn64btq6fx4, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C3 BETWEEN :1 AND :2 Plan hash value: 108045900 ---------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | ---------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 90001 |00:00:00.18 | 4865 | |* 1 | FILTER | | 1 | | 90001 |00:00:00.18 | 4865 | | 2 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 250 | 90001 |00:00:00.13 | 4865 | |* 3 | INDEX RANGE SCAN | IND_T1_C3 | 1 | 450 | 90001 |00:00:00.04 | 1090 | ---------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:1<=:2) 3 - access("C3">=:1 AND "C3"<=:2) The same execution plan as returned when we used literals (constants), so bind peeking is probably working. Now let’s try the query that accesses column C4: SELECT * FROM T1 WHERE C4 BETWEEN :N1 AND :N2; ... SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); SQL_ID gwgk5h8u3k4tp, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C4 BETWEEN :1 AND :2 Plan hash value: 3332582666 ------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | ------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 90001 |00:00:00.10 | 4147 | |* 1 | FILTER | | 1 | | 90001 |00:00:00.10 | 4147 | |* 2 | TABLE ACCESS FULL| T1 | 1 | 89961 | 90001 |00:00:00.05 | 4147 | ------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:1<=:2) 2 - filter(("C4">=:1 AND "C4"<=:2)) Bind variable peeking was used to obtain a close estimate for the cardinalities. So, what happens when we change the value of bind variable N2? EXEC :N2:=101 SELECT * FROM T1 WHERE C3 BETWEEN :N1 AND :N2; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); SQL_ID 2ksn64btq6fx4, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C3 BETWEEN :1 AND :2 Plan hash value: 108045900 ---------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | ---------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 100 |00:00:00.01 | 7 | |* 1 | FILTER | | 1 | | 100 |00:00:00.01 | 7 | | 2 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 250 | 100 |00:00:00.01 | 7 | |* 3 | INDEX RANGE SCAN | IND_T1_C3 | 1 | 450 | 100 |00:00:00.01 | 3 | ---------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:1<=:2) 3 - access("C3">=:1 AND "C3"<=:2) - SELECT * FROM T1 WHERE C3 BETWEEN :N1 AND :N2; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); SQL_ID gwgk5h8u3k4tp, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C4 BETWEEN :1 AND :2 Plan hash value: 3332582666 ------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | ------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 100 |00:00:00.01 | 3263 | |* 1 | FILTER | | 1 | | 100 |00:00:00.01 | 3263 | |* 2 | TABLE ACCESS FULL| T1 | 1 | 89961 | 100 |00:00:00.01 | 3263 | ------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:1<=:2) 2 - filter(("C4">=:1 AND "C4"<=:2)) The bind variable in the presence of bind peeking with a histogram came back to bit us – ideally Oracle would have used an index range scan for this set of bind variables. Let’s add another 10,000 rows to the table without gathering statistics and see what happens to the cardinality estimates in the plans: INSERT INTO T1 SELECT (100000+ROWNUM), (100000+ROWNUM), TRUNC((100000+ROWNUM)/100+1), TRUNC((100000+ROWNUM)/100+1), CHR(65+TRUNC((100000+ROWNUM)/10000))||TRUNC((100000+ROWNUM)/100+1), CHR(65+TRUNC((100000+ROWNUM)/10000))||TRUNC((100000+ROWNUM)/100+1), LPAD('A',200,'A') FROM DUAL CONNECT BY LEVEL<=100000; COMMIT; ALTER SYSTEM FLUSH SHARED_POOL; Here are the execution plans: SELECT * FROM T1 WHERE C3=1901; SQL_ID 86z7nbb5u2p26, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C3=1901 Plan hash value: 1220227203 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 100 |00:00:00.01 | 8 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 1000 | 100 |00:00:00.01 | 8 | |* 2 | INDEX RANGE SCAN | IND_T1_C3 | 1 | 400 | 100 |00:00:00.01 | 4 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C3"=1901) - SELECT * FROM T1 WHERE C4=1901; SQL_ID 08rg4uf562h4x, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C4=1901 Plan hash value: 7035821 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 100 |00:00:00.01 | 8 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 10 | 100 |00:00:00.01 | 8 | |* 2 | INDEX RANGE SCAN | IND_T1_C4 | 1 | 10 | 100 |00:00:00.01 | 4 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C4"=1901) The cardinality estimates are unchanged from what we saw earlier. The first query is estimating 10 times too many rows will be returned, and the second query is estimating 10 times too few rows will be returned. Let’s delete the rows that were just added and check a couple of statistics: DELETE FROM T1 WHERE C1>100000; 100000 rows deleted. COMMIT; SELECT SUBSTR(COLUMN_NAME,1,3) COL, DENSITY, NUM_BUCKETS, LAST_ANALYZED FROM DBA_TAB_COLUMNS WHERE TABLE_NAME='T1' ORDER BY 1; COL DENSITY NUM_BUCKETS LAST_ANAL --- ---------- ----------- --------- C1 C2 .00001 254 14-FEB-10 C3 C4 .00099998 254 14-FEB-10 C5 C6 .00099998 254 14-FEB-10 FIL No density values for columns C1, C3, or C5 and the LAST_ANALYZED column is NULL for those same entries (edit: same results on Oracle 10.2.0.2). Let’s try collecting statistics again without specifying the columns for which histograms should be created: EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T1',CASCADE=>TRUE,ESTIMATE_PERCENT=>100,NO_INVALIDATE=>FALSE) SELECT SUBSTR(COLUMN_NAME,1,3) COL, DENSITY, NUM_BUCKETS, LAST_ANALYZED FROM DBA_TAB_COLUMNS WHERE TABLE_NAME='T1' ORDER BY 1; COL DENSITY NUM_BUCKETS LAST_ANAL --- ---------- ----------- --------- C1 .00001 1 14-FEB-10 C2 .00001 1 14-FEB-10 C3 .000999001 1 14-FEB-10 C4 .000999001 1 14-FEB-10 C5 .000999001 1 14-FEB-10 C6 .000999001 1 14-FEB-10 FIL 1 1 14-FEB-10 Now we have no histograms, but the density and LAST_ANALYZED columns are populated for all rows returned by the above query (edit: same results on Oracle 10.2.0.2). For fun let’s retry one of the queries that returned odd cardinality estimates earlier: SELECT * FROM T1 WHERE C3 BETWEEN 101 AND 201; SQL_ID 8jd9h693mbkvc, child number 0 ------------------------------------- SELECT * FROM T1 WHERE C3 BETWEEN 101 AND 201 Plan hash value: 1220227203 --------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 10100 |00:00:00.02 | 547 | | 1 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 10200 | 10100 |00:00:00.02 | 547 | |* 2 | INDEX RANGE SCAN | IND_T1_C3 | 1 | 10200 | 10100 |00:00:00.01 | 124 | --------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("C3">=101 AND "C3"<=201) The estimated cardinality is just about as accurate as it was with the query that accessed column C4 which had a histogram. What are your thoughts? Is the METHOD_OPT parameter of DBMS_STATS.GATHER_TABLE_STATS only used to specify how histograms will be collected for columns, or does that parameter also specify for which columns column-level statistics should be gathered? If you again collect statistics using the DBMS_STATS.GATHER_TABLE_STATS command found near the start of this article, the query of DBA_TAB_COLUMNS returns the following, which is what was originally expected (edit: same results on Oracle 10.2.0.2): COL DENSITY NUM_BUCKETS LAST_ANAL --- ---------- ----------- --------- C1 .00001 1 14-FEB-10 C2 .00001 254 14-FEB-10 C3 .000999001 1 14-FEB-10 C4 .00099998 254 14-FEB-10 C5 .000999001 1 14-FEB-10 C6 .00099998 254 14-FEB-10 FIL 1 1 14-FEB-10 ——- Follow-up (8 hours after this article was scheduled to appear): It is important at times to check more than one source in the Oracle documentation (bold/italic emphasis is mine): “METHOD_OPT – The value controls column statistics collection and histogram creation.” Here is an example where Christian Antognini tried to drive that point home: “Since the syntax is FOR ALL INDEXED COLUMNS, you are gathering statistics for all indexed columns only. I.e. not for all columns. FOR ALL COLUMNS should be used for that…” “When the option FOR ALL INDEXED COLUMNS is specified, columns statistics are gathered only for the indexed columns.” “My point was that the parameter method_opt not only impacts histograms, but also column statistics.” Greg Rahn also made the point in this blog article. “The METHOD_OPT parameter of DBMS_STATS controls two things: on which columns statistics will be collected on which columns histograms will be collected (and how many buckets)” The message of this blog article is to make certain that you know what the GATHER_TABLE_STATS procedure is actually doing when you use the METHOD_OPT parameter (or alter the default value for the parameter). The apparent benefit from having the histogram in place might actually be a false benefit – it might be that previously you were not updating the column statistics for that column, until you started creating a histogram on that column (assuming that you were previously collecting histograms on other columns in the table, and you assumed that column statistics were updated for the remaining columns). Incidentally, if instead of using this to collect statistics for the table: EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T1',CASCADE=>TRUE,ESTIMATE_PERCENT=>100,METHOD_OPT=>'FOR COLUMNS SIZE 254 C2, C4, C6') You used this: EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T1',CASCADE=>TRUE,ESTIMATE_PERCENT=>100,METHOD_OPT=>'FOR ALL INDEXED COLUMNS size skewonly') You would end up with the following when executing the query against DBA_TAB_COLUMNS: COL DENSITY NUM_BUCKETS LAST_ANAL --- ---------- ----------- --------- C1 .00001 1 15-FEB-10 C2 C3 .000999001 1 15-FEB-10 C4 .000999001 1 15-FEB-10 C5 .00099998 254 15-FEB-10 C6 FIL The above output confirms Christian Antognini’s comment in the OTN thread – column-level statistics were not collected for columns C2, C6, and FILLER. There is effectively no histogram on columns C3 and C4 (a single bucket, therefore no histogram), but the cardinality estimates in the plan will be close because column-level statistics are present for columns C3 and C4. Comments : 5 Comments » Categories : Bind Variable, Histogram, Performance 10046 Extended SQL Trace Interpretation 2 26 01 2010 January 26, 2010 (Back to the Previous Post in the Series) (Forward to the Next Post in the Series) In an earlier blog article I described several methods for enabling and disabling 10046 extended SQL traces, listed several keywords that are found in 10046 traces, and demonstrated output generated by my Toy Project for Performance Tuning as it processed the raw trace file. Another way to enable a 10046 trace for a session is through the use of a logon trigger created by the SYS user. For example, the following trigger will enable a 10046 trace at level 12 for any program that begins with the letters MS or VB, even if the path to the program is included in the PROGRAM column of V$SESSION. With the use of DECODE, it is very easy to allow the trigger to enable tracing for an additional list of programs: CREATE OR REPLACE TRIGGER LOGON_10046_TRACE AFTER LOGON ON DATABASE DECLARE SHOULD_EXECUTE INTEGER; BEGIN SELECT DECODE(SUBSTR(UPPER(PROGRAM),1,2),'MS',1,'VB',1,0) +DECODE(INSTR(PROGRAM,'\',-1),0,0,DECODE(SUBSTR(UPPER(SUBSTR(PROGRAM,INSTR(PROGRAM,'\',-1)+1)),1,2),'MS',1,'VB',1,0)) INTO SHOULD_EXECUTE FROM V$SESSION WHERE SID=(SELECT SID FROM V$MYSTAT WHERE ROWNUM=1); IF SHOULD_EXECUTE > 0 THEN EXECUTE IMMEDIATE 'ALTER SESSION SET EVENTS ''10046 TRACE NAME CONTEXT FOREVER, LEVEL 12'''; END IF; END; / Obviously, if you create the trigger, you should drop the trigger when it is no longer needed using the following command. DROP TRIGGER LOGON_10046_TRACE; Let’s try creating a test trace file with Oracle Database 10.2.0.4. First, we need to create a couple of test tables with 10,000 rows each and collect statistics for the tables and primary key indexes: CREATE TABLE T1 ( C1 NUMBER, C2 VARCHAR2(255), PRIMARY KEY (C1)); CREATE TABLE T2 ( C1 NUMBER, C2 VARCHAR2(255), PRIMARY KEY (C1)); INSERT INTO T1 SELECT ROWNUM, RPAD(TO_CHAR(ROWNUM),255,'A') FROM DUAL CONNECT BY LEVEL<=10000; INSERT INTO T2 SELECT ROWNUM, RPAD(TO_CHAR(ROWNUM),255,'A') FROM DUAL CONNECT BY LEVEL<=10000; COMMIT; EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T1',CASCADE=>TRUE) EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T2',CASCADE=>TRUE) We now have two tables, each with 10,000 rows. This test case in SQL*Plus will: Flush the buffer cache (twice) to force physical reads Set the fetch array size to the SQL*Plus default value of 15 rows per fetch call Disable output of the rows returned from the database to limit client-side delays Create two bind variables with the value of 1 and 2 Enable a 10046 extended SQL trace at level 12 Give the trace file an easy to identify name Execute a query that joins the two tables Increases the fetch array size to 50 rows per fetch call Increase the value of the second bind variable from 2 to 10,000 Execute the same SQL statement executed in step 7 ALTER SYSTEM FLUSH BUFFER_CACHE; ALTER SYSTEM FLUSH BUFFER_CACHE; SET ARRAYSIZE 15 SET AUTOTRACE TRACEONLY STATISTICS VARIABLE N1 NUMBER VARIABLE N2 NUMBER EXEC :N1 := 1 EXEC :N2 := 2 EXEC DBMS_SESSION.SESSION_TRACE_ENABLE(WAITS=>TRUE, BINDS=>TRUE) ALTER SESSION SET TRACEFILE_IDENTIFIER='10046_FIND_ME'; SELECT T1.C1, T2.C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SET ARRAYSIZE 50 EXEC :N1 := 1 EXEC :N2 := 10000 SELECT T1.C1, T2.C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT SYSDATE FROM DUAL; EXEC DBMS_SESSION.SESSION_TRACE_DISABLE; Rather than scrolling all of the rows on the screen, SQL*Plus output the following from the two executions: Statistics --------------------------------------------------- 1 recursive calls 0 db block gets 9 consistent gets 5 physical reads 0 redo size 921 bytes sent via SQL*Net to client 334 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 2 rows processed Statistics --------------------------------------------------- 0 recursive calls 0 db block gets 10982 consistent gets 404 physical reads 0 redo size 2647906 bytes sent via SQL*Net to client 2523 bytes received via SQL*Net from client 201 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 10000 rows processed From the above, the first execution required 9 consistent block gets, and 5 of those block gets involved reading the block from disk. The server sent 921 bytes to the client in 2 round trips, and 2 rows were retrieved. 2 round trips (we will see why later)? The second execution required 10,982 consistent block gets, and 404 of those involved physical reads. The server sent about 2.53MB to the client in 10,000 rows, using 201 round trips. Nice, but we can find out more about what happened. We could process the trace file with tkprof using a command like this: tkprof testdb_ora_4148_10046_find_me.trc testdb_ora_4148_10046_find_me.txt A portion of the TKPROF output might look like this (see Metalink Doc ID 41634.1 for help with reading the tkprof output): ******************************************************************************** count = number of times OCI procedure was executed cpu = cpu time in seconds executing elapsed = elapsed time in seconds executing disk = number of physical reads of buffers from disk query = number of buffers gotten for consistent read current = number of buffers gotten in current mode (usually for update) rows = number of rows processed by the fetch or execute call ******************************************************************************** SELECT T1.C1, T2.C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 2 0.00 0.00 0 0 0 0 Execute 3 0.00 0.00 0 0 0 0 Fetch 203 0.28 0.38 409 10991 0 10002 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 208 0.28 0.38 409 10991 0 10002 Misses in library cache during parse: 1 Misses in library cache during execute: 1 Optimizer mode: ALL_ROWS Parsing user id: 31 Rows Row Source Operation ------- --------------------------------------------------- 2 FILTER (cr=9 pr=5 pw=0 time=19577 us) 2 NESTED LOOPS (cr=9 pr=5 pw=0 time=19569 us) 2 TABLE ACCESS BY INDEX ROWID T2 (cr=5 pr=3 pw=0 time=13843 us) 2 INDEX RANGE SCAN SYS_C0020548 (cr=3 pr=2 pw=0 time=9231 us)(object id 114211) 2 INDEX UNIQUE SCAN SYS_C0020547 (cr=4 pr=2 pw=0 time=5788 us)(object id 114210) Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ SQL*Net message to client 204 0.00 0.00 db file sequential read 409 0.00 0.21 SQL*Net message from client 204 0.00 0.07 SQL*Net more data to client 1200 0.00 0.02 The above is a nice summary of what was found in the trace file for our specific test SQL statement, but what is it telling us? There were 2 parse calls (1 for each execution), and one of those parse calls resulted in a hard parse. There were 3 execution calls (I am only able to explain 2 of the execution calls). There were 203 fetch calls that retrieved a total of 10,002 rows – from this we could derive that on average the client fetched 49.27 rows per fetch call. All of the time for the execution happened on the fetch calls, which required 0.28 seconds of CPU time and a total of 0.38 clock seconds to execute. A total of 10,991 consistent gets were required and 409 blocks were read from disk. The execution plan displayed is a little misleading, since it shows that only 2 rows were retrieved (note that if this example were executed on Oracle 11.1.0.6 or higher, the first and second executions of the SQL statement could have had different execution plans). An index range scan is performed on the index SYS_C0020548 (the primary key index for table T2) to locate all of the C1 values between 1 and 2 – note that the optimizer used transitive closure here since the restriction in the SQL statement was actually placed on the column C1 of table T1. The top line in the plan indicates that in total the query required 9 consistent gets, 5 physical block reads, and 0.019577 seconds to execute. The index range scan on SYS_C0020548 required 3 consistent gets, and 2 physical block reads were required to satisfy the 3 consistent gets. Table T2 required an additional 2 consistent gets and 1 physical block read. A nested loop operation was performed, driving into the primary key index for table T1 – this required an additional 4 consistent gets and 2 physical block reads. But, what about the second execution of the SQL statement? The wait events show 409 waits on the “db file sequential read” wait event, which indicates physically reading 1 block at a time from disk – note that this exactly matches the “disk” column in the “fetch” line of the tkprof summary. Every block that had to be read from disk was read one block at a time, with an average read time of 0.000513 seconds per block read (the extremely fast average time likely indicate caching of blocks at the file system, RAID controller, SAN, or hard drive). There were 1200 waits on the “SQL*Net more data to client” wait, indicating that the SDU size was filled 1200 times when sending the data to the client computer. We could also run the trace file though my Toy Project (below is one of 4 outputs from my program): Total for Trace File: |PARSEs 6|CPU S 0.000000|CLOCK S 0.008073|ROWs 0|PHY RD BLKs 0|CON RD BLKs (Mem) 0|CUR RD BLKs (Mem) 0|SHARED POOL MISs 4| |EXECs 6|CPU S 0.015625|CLOCK S 0.007978|ROWs 3|PHY RD BLKs 0|CON RD BLKs (Mem) 0|CUR RD BLKs (Mem) 0|SHARED POOL MISs 2| |FETCHs 205|CPU S 0.281250|CLOCK S 0.382338|ROWs 10003|PHY RD BLKs 409|CON RD BLKs (Mem) 10991|CUR RD BLKs (Mem) 0|SHARED POOL MISs 0| Wait Event Summary: SQL*Net message to client 0.000318 On Client/Network Min Wait: 0.000001 Avg Wait: 0.000001 Max Wait: 0.000005 SQL*Net message from client 3.916585 On Client/Network Min Wait: 0.000211 Avg Wait: 0.018302 Max Wait: 3.829540 db file sequential read 0.211736 On DB Server Min Wait: 0.000182 Avg Wait: 0.000518 Max Wait: 0.005905 SQL*Net more data to client 0.020220 On Client/Network Min Wait: 0.000010 Avg Wait: 0.000017 Max Wait: 0.000258 Total for Similar SQL Statements in Each Group: ---------------------------------------------------------------------------------- Similar SQL Statements in Group: 2 |PARSEs 2|CPU S 0.000000|CLOCK S 0.001231|ROWs 0|PHY RD BLKs 0|CON RD BLKs (Mem) 0|CUR RD BLKs (Mem) 0|SHARED POOL MISs 1| |EXECs 2|CPU S 0.000000|CLOCK S 0.004237|ROWs 0|PHY RD BLKs 0|CON RD BLKs (Mem) 0|CUR RD BLKs (Mem) 0|SHARED POOL MISs 1| |FETCHs 203|CPU S 0.281250|CLOCK S 0.382314|ROWs 10002|PHY RD BLKs 409|CON RD BLKs (Mem) 10991|CUR RD BLKs (Mem) 0|SHARED POOL MISs 0| CPU S 94.74% CLOCK S 97.34% * 0.211736 seconds of time related data file I/O * 0.096212 seconds of time related to client/network events | +++++++++++++++++++|| +++++++++++++++++++| Cursor 3 Ver 1 Parse at 0.000000 Similar Cnt 1 |PARSEs 1|CPU S 0.000000|CLOCK S 0.001113|ROWs 0|PHY RD BLKs 0|CON RD BLKs (Mem) 0|CUR RD BLKs (Mem) 0|SHARED POOL MISs 1| |EXECs 1|CPU S 0.000000|CLOCK S 0.003538|ROWs 0|PHY RD BLKs 0|CON RD BLKs (Mem) 0|CUR RD BLKs (Mem) 0|SHARED POOL MISs 1| |FETCHs 2|CPU S 0.015625|CLOCK S 0.019769|ROWs 2|PHY RD BLKs 5|CON RD BLKs (Mem) 9|CUR RD BLKs (Mem) 0|SHARED POOL MISs 0| CPU S 5.26% CLOCK S 6.13% | +|| +| SELECT T1.C1, T2.C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 (Rows 2) FILTER (cr=9 pr=5 pw=0 time=19577 us) (Rows 2) NESTED LOOPS (cr=9 pr=5 pw=0 time=19569 us) (Rows 2) TABLE ACCESS BY INDEX ROWID T2 (cr=5 pr=3 pw=0 time=13843 us) (Rows 2) INDEX RANGE SCAN SYS_C0020548 (cr=3 pr=2 pw=0 time=9231 us) (Rows 2) INDEX UNIQUE SCAN SYS_C0020547 (cr=4 pr=2 pw=0 time=5788 us) Bind Variables:BINDS #3: -0.000008 Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=48 off=0 kxsbbbfp=13ce8870 bln=22 avl=02 flg=05 value=1 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24 kxsbbbfp=13ce8888 bln=22 avl=02 flg=01 value=2 ------------ Cursor 4 Ver 1 Parse at 0.041427 (TD Prev 0.007600) Similar Cnt 2 |PARSEs 1|CPU S 0.000000|CLOCK S 0.000118|ROWs 0|PHY RD BLKs 0|CON RD BLKs (Mem) 0|CUR RD BLKs (Mem) 0|SHARED POOL MISs 0| |EXECs 1|CPU S 0.000000|CLOCK S 0.000699|ROWs 0|PHY RD BLKs 0|CON RD BLKs (Mem) 0|CUR RD BLKs (Mem) 0|SHARED POOL MISs 0| |FETCHs 201|CPU S 0.265625|CLOCK S 0.362545|ROWs 10000|PHY RD BLKs 404|CON RD BLKs (Mem) 10982|CUR RD BLKs (Mem) 0|SHARED POOL MISs 0| CPU S 89.47% CLOCK S 91.21% | ++++++++++++++++++|| ++++++++++++++++++| SELECT T1.C1, T2.C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 (Rows 10000) FILTER (cr=10982 pr=404 pw=0 time=680024 us) (Rows 10000) NESTED LOOPS (cr=10982 pr=404 pw=0 time=670018 us) (Rows 10000) TABLE ACCESS BY INDEX ROWID T2 (cr=781 pr=387 pw=0 time=590006 us) (Rows 10000) INDEX RANGE SCAN SYS_C0020548 (cr=218 pr=17 pw=0 time=10038 us) (Rows 10000) INDEX UNIQUE SCAN SYS_C0020547 (cr=10201 pr=17 pw=0 time=77882 us) Bind Variables:BINDS #4: 0.041421 Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=48 off=0 kxsbbbfp=13ce8870 bln=22 avl=02 flg=05 value=1 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24 kxsbbbfp=13ce8888 bln=22 avl=02 flg=01 value=10000 ---------------------------------------------------------------------------------- The above shows basically the same output as tkprof, just with greater resolution, both sets of bind variables, and both sets of execution plans. We could also use any number of other 10046 trace file parsers including TRCANLZR (see Metalink Doc ID 224270.1), TVD$XTAT (see the book “Troubleshooting Oracle Performance”), ESQLTRCPROF (see the book “Secrets of the Oracle Database”), the Hotsos Profiler (Method R), OraSRP (www.oracledba.ru/orasrp/), or one of several other programs. I typically either look at the output from my program or the raw 10046 trace file. That brings us to the raw 10046 trace file. So, what does the raw output of the trace file look like? Before diving into the raw trace file, let’s find a little information to help us later: COLUMN TABLE_NAME FORMAT A10 SELECT TABLE_NAME, INDEX_NAME FROM DBA_INDEXES WHERE TABLE_NAME IN ('T1','T2') ORDER BY TABLE_NAME; TABLE_NAME INDEX_NAME ---------- ------------ T1 SYS_C0020547 T2 SYS_C0020548 COLUMN OBJECT_NAME FORMAT A15 SELECT OBJECT_ID, OBJECT_NAME, OBJECT_TYPE FROM DBA_OBJECTS WHERE OBJECT_NAME IN ('T1','T2','SYS_C0020547','SYS_C0020548') ORDER BY OBJECT_NAME; OBJECT_ID OBJECT_NAME OBJECT_TYPE ---------- --------------- ----------- 114210 SYS_C0020547 INDEX 114211 SYS_C0020548 INDEX 114209 T1 TABLE 114207 T2 TABLE From the above output, the index on table T1 is named SYS_C0020547 and it has an OBJECT_ID of 114210. The index on table T2 is named SYS_C0020548 and it has an OBJECT_ID of 114211. Table T1 has an OBJECT_ID of 114209, and table T2 has an OBJECT_ID of 114207. Now on to a portion of the raw 10046 trace file: ===================== PARSING IN CURSOR #3 len=91 dep=0 uid=31 oct=3 lid=31 tim=4963261335 hv=3021110247 ad='982ab100' SELECT T1.C1, T2.C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 END OF STMT PARSE #3:c=0,e=1113,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,tim=4963261327 BINDS #3: kkscoacd Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=48 off=0 kxsbbbfp=13ce8870 bln=22 avl=02 flg=05 value=1 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24 kxsbbbfp=13ce8888 bln=22 avl=02 flg=01 value=2 From the above, we see that the first parse was a hard parse that required 0 CPU seconds and 0.001113 clock seconds. Additionally, two bind variables were passed in. A level 4 or level 12 10046 extended SQL trace file will include the submitted bind variable values, as shown above. It is possible to use the following list to decode the bind variable data type (oacdty), in the process determining that the bind variables are in fact, defined as numbers. See Metalink Doc IDs 67701.1 and 154170.1, the Oracle OCI documentation, or Julian Dyke’s site for a more complete list of datatype constants: 0 - This row is a placeholder for a procedure with no arguments. 1 - VARCHAR2 (or NVARCHAR) 2 - NUMBER 3 - NATIVE INTEGER (for PL/SQL's BINARY_INTEGER) 8 - LONG 11 - ROWID 12 - DATE 23 - RAW 24 - LONG RAW 96 - CHAR (or NCHAR) 112 - CLOB or NCLOB 113 - BLOB 114 - BFILE 106 - MLSLABEL 250 - PL/SQL RECORD 251 - PL/SQL TABLE 252 - PL/SQL BOOLEAN The trace file continues below: EXEC #3:c=0,e=3538,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,tim=4963265927 WAIT #3: nam='SQL*Net message to client' ela= 5 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=4963266031 WAIT #3: nam='db file sequential read' ela= 4816 file#=4 block#=1138316 blocks=1 obj#=114211 tim=4963271120 WAIT #3: nam='db file sequential read' ela= 3934 file#=4 block#=1138318 blocks=1 obj#=114211 tim=4963275219 WAIT #3: nam='db file sequential read' ela= 4443 file#=4 block#=1138310 blocks=1 obj#=114207 tim=4963279805 WAIT #3: nam='db file sequential read' ela= 5221 file#=4 block#=1093148 blocks=1 obj#=114210 tim=4963285172 WAIT #3: nam='db file sequential read' ela= 236 file#=4 block#=1093150 blocks=1 obj#=114210 tim=4963285569 FETCH #3:c=15625,e=19589,p=5,cr=5,cu=0,mis=0,r=1,dep=0,og=1,tim=4963285717 WAIT #3: nam='SQL*Net message from client' ela= 364 driver id=1413697536 #bytes=1 p3=0 obj#=114210 tim=4963286240 WAIT #3: nam='SQL*Net message to client' ela= 4 driver id=1413697536 #bytes=1 p3=0 obj#=114210 tim=4963286494 FETCH #3:c=0,e=180,p=0,cr=4,cu=0,mis=0,r=1,dep=0,og=1,tim=4963286588 Above, we see the 5 physical block reads of blocks 1138316 and 1138318 of OBJECT_ID 114211 (index on table T2, SYS_C0020548), followed by a single block read of OBJECT_ID 114207 (table T2), and 2 single block reads of object 114210 (index on table T1, SYS_C0020547) – note that the final of the 5 physical block reads completed in 0.000236 seconds, which is about 20 times faster than the time it takes for 1 revolution of a 15,000 RPM hard drive platter. The first fetch call returned a single row, even though the array fetch size was explicitly set to 15 rows. That fetch required 5 consistent gets, which then required the 5 physical block reads. The 1 row was sent to the client which then fetched a second row (4963286588 – 4963286240 4963285717)/1,000,000 = 0.000348 0.000871 seconds later. The trace file continues: WAIT #3: nam='SQL*Net message from client' ela= 319 driver id=1413697536 #bytes=1 p3=0 obj#=114210 tim=4963287035 *** SESSION ID:(211.17427) 2010-01-25 15:41:01.540 STAT #3 id=1 cnt=2 pid=0 pos=1 obj=0 op='FILTER (cr=9 pr=5 pw=0 time=19577 us)' STAT #3 id=2 cnt=2 pid=1 pos=1 obj=0 op='NESTED LOOPS (cr=9 pr=5 pw=0 time=19569 us)' STAT #3 id=3 cnt=2 pid=2 pos=1 obj=114207 op='TABLE ACCESS BY INDEX ROWID T2 (cr=5 pr=3 pw=0 time=13843 us)' STAT #3 id=4 cnt=2 pid=3 pos=1 obj=114211 op='INDEX RANGE SCAN SYS_C0020548 (cr=3 pr=2 pw=0 time=9231 us)' STAT #3 id=5 cnt=2 pid=2 pos=2 obj=114210 op='INDEX UNIQUE SCAN SYS_C0020547 (cr=4 pr=2 pw=0 time=5788 us)' WAIT #0: nam='SQL*Net message to client' ela= 2 driver id=1413697536 #bytes=1 p3=0 obj#=114210 tim=4963289021 WAIT #0: nam='SQL*Net message from client' ela= 2329 driver id=1413697536 #bytes=1 p3=0 obj#=114210 tim=4963291420 ===================== Cursor #3 was closed, so Oracle output the STAT (row source operation) lines, as we saw in the tkprof output. The trace file continues (with a couple of lines removed): ... ===================== PARSING IN CURSOR #4 len=91 dep=0 uid=31 oct=3 lid=31 tim=4963302762 hv=3021110247 ad='982ab100' SELECT T1.C1, T2.C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 END OF STMT PARSE #4:c=0,e=118,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,tim=4963302756 BINDS #4: kkscoacd Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=48 off=0 kxsbbbfp=13ce8870 bln=22 avl=02 flg=05 value=1 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24 kxsbbbfp=13ce8888 bln=22 avl=02 flg=01 value=10000 EXEC #4:c=0,e=699,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,tim=4963304299 WAIT #4: nam='SQL*Net message to client' ela= 3 driver id=1413697536 #bytes=1 p3=0 obj#=114210 tim=4963304383 FETCH #4:c=0,e=94,p=0,cr=5,cu=0,mis=0,r=1,dep=0,og=1,tim=4963304564 WAIT #4: nam='SQL*Net message from client' ela= 718 driver id=1413697536 #bytes=1 p3=0 obj#=114210 tim=4963305403 WAIT #4: nam='SQL*Net message to client' ela= 2 driver id=1413697536 #bytes=1 p3=0 obj#=114210 tim=4963305590 WAIT #4: nam='SQL*Net more data to client' ela= 29 driver id=1413697536 #bytes=2146 p3=0 obj#=114210 tim=4963305766 WAIT #4: nam='SQL*Net more data to client' ela= 20 driver id=1413697536 #bytes=1862 p3=0 obj#=114210 tim=4963305913 WAIT #4: nam='SQL*Net more data to client' ela= 17 driver id=1413697536 #bytes=2128 p3=0 obj#=114210 tim=4963306065 WAIT #4: nam='db file sequential read' ela= 2272 file#=4 block#=1138311 blocks=1 obj#=114207 tim=4963308471 WAIT #4: nam='SQL*Net more data to client' ela= 27 driver id=1413697536 #bytes=1868 p3=0 obj#=114207 tim=4963308686 WAIT #4: nam='SQL*Net more data to client' ela= 18 driver id=1413697536 #bytes=2122 p3=0 obj#=114207 tim=4963308841 WAIT #4: nam='SQL*Net more data to client' ela= 13 driver id=1413697536 #bytes=2128 p3=0 obj#=114207 tim=4963309001 FETCH #4:c=0,e=3573,p=1,cr=54,cu=0,mis=0,r=50,dep=0,og=1,tim=4963309109 Note that there was no hard parse this time. The first two fetches are complete at this point. Again, the first fetch returned a single row, while the second fetch returned 50 rows. Note the presence of the “SQL*Net more data to client” wait before the second fetch line printed – each of these lines indicates that the SDU size was filled on the previous send to the client. Notice that there was only a single physical block read of OBJECT_ID 114207 (table T2), requiring 0.002272 seconds, when fetching the first 51 rows (the other blocks were already in the buffer cache). The trace file continues below: WAIT #4: nam='SQL*Net message from client' ela= 256 driver id=1413697536 #bytes=1 p3=0 obj#=114207 tim=4963309476 WAIT #4: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=114207 tim=4963309654 WAIT #4: nam='db file sequential read' ela= 226 file#=4 block#=1138312 blocks=1 obj#=114207 tim=4963309995 WAIT #4: nam='SQL*Net more data to client' ela= 32 driver id=1413697536 #bytes=2116 p3=0 obj#=114207 tim=4963310197 WAIT #4: nam='SQL*Net more data to client' ela= 14 driver id=1413697536 #bytes=2096 p3=0 obj#=114207 tim=4963310353 WAIT #4: nam='SQL*Net more data to client' ela= 13 driver id=1413697536 #bytes=1834 p3=0 obj#=114207 tim=4963310488 WAIT #4: nam='db file sequential read' ela= 1762 file#=4 block#=1138308 blocks=1 obj#=114207 tim=4963312390 WAIT #4: nam='SQL*Net more data to client' ela= 23 driver id=1413697536 #bytes=2096 p3=0 obj#=114207 tim=4963312551 WAIT #4: nam='SQL*Net more data to client' ela= 16 driver id=1413697536 #bytes=2096 p3=0 obj#=114207 tim=4963312783 WAIT #4: nam='SQL*Net more data to client' ela= 13 driver id=1413697536 #bytes=1834 p3=0 obj#=114207 tim=4963312904 FETCH #4:c=0,e=3345,p=2,cr=55,cu=0,mis=0,r=50,dep=0,og=1,tim=4963312955 Two more physical block reads of OBJECT_ID 114207 (table T2) to return the next 50 rows to the client. Jumping forward in the trace file to the last two fetches: ... FETCH #4:c=0,e=1259,p=2,cr=55,cu=0,mis=0,r=50,dep=0,og=1,tim=4963757700 WAIT #4: nam='SQL*Net message from client' ela= 842 driver id=1413697536 #bytes=1 p3=0 obj#=114207 tim=4963758586 WAIT #4: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=114207 tim=4963758653 WAIT #4: nam='SQL*Net more data to client' ela= 16 driver id=1413697536 #bytes=2124 p3=0 obj#=114207 tim=4963758766 WAIT #4: nam='db file sequential read' ela= 242 file#=4 block#=1773782 blocks=1 obj#=114207 tim=4963759071 WAIT #4: nam='SQL*Net more data to client' ela= 17 driver id=1413697536 #bytes=2104 p3=0 obj#=114207 tim=4963759182 WAIT #4: nam='SQL*Net more data to client' ela= 13 driver id=1413697536 #bytes=1841 p3=0 obj#=114207 tim=4963759268 WAIT #4: nam='SQL*Net more data to client' ela= 17 driver id=1413697536 #bytes=2104 p3=0 obj#=114207 tim=4963759365 WAIT #4: nam='SQL*Net more data to client' ela= 13 driver id=1413697536 #bytes=1841 p3=0 obj#=114207 tim=4963759453 WAIT #4: nam='db file sequential read' ela= 226 file#=4 block#=1773852 blocks=1 obj#=114207 tim=4963759715 WAIT #4: nam='SQL*Net more data to client' ela= 20 driver id=1413697536 #bytes=2104 p3=0 obj#=114207 tim=4963759867 FETCH #4:c=0,e=1290,p=2,cr=54,cu=0,mis=0,r=49,dep=0,og=1,tim=4963759912 From the above, we see that Oracle is still performing an average of two single block physical reads per fetch call, and the final fetch call retrieved just 49 rows. The trace file continues: WAIT #4: nam='SQL*Net message from client' ela= 792 driver id=1413697536 #bytes=1 p3=0 obj#=114207 tim=4963760775 *** SESSION ID:(211.17427) 2010-01-25 15:41:02.008 STAT #4 id=1 cnt=10000 pid=0 pos=1 obj=0 op='FILTER (cr=10982 pr=404 pw=0 time=680024 us)' STAT #4 id=2 cnt=10000 pid=1 pos=1 obj=0 op='NESTED LOOPS (cr=10982 pr=404 pw=0 time=670018 us)' STAT #4 id=3 cnt=10000 pid=2 pos=1 obj=114207 op='TABLE ACCESS BY INDEX ROWID T2 (cr=781 pr=387 pw=0 time=590006 us)' STAT #4 id=4 cnt=10000 pid=3 pos=1 obj=114211 op='INDEX RANGE SCAN SYS_C0020548 (cr=218 pr=17 pw=0 time=10038 us)' STAT #4 id=5 cnt=10000 pid=2 pos=2 obj=114210 op='INDEX UNIQUE SCAN SYS_C0020547 (cr=10201 pr=17 pw=0 time=77882 us)' WAIT #0: nam='SQL*Net message to client' ela= 2 driver id=1413697536 #bytes=1 p3=0 obj#=114207 tim=4963764774 WAIT #0: nam='SQL*Net message from client' ela= 2789 driver id=1413697536 #bytes=1 p3=0 obj#=114207 tim=4963767585 From the above, we see the execution plan for the second execution – this information was missing from the tkprof output. A hash join with two full table scans probably would have been more efficient than a nested loop join with index lookups, especially if the number of rows were larger. This is one of the potential problems with using bind variables, especially when bind variable peeking is enabled (by default in recent releases) – the execution plan is essentially locked after the initial hard parse. Oracle 11.1.0.6 introduced a feature known as adaptive cursor sharing that could potentially alter the plan on a future execution if Oracle senses that there will be significant changes in the number of rows returned when different bind variable values are submitted. Incidentally, you may have noticed the keyword “oct” on the “PARSING IN CURSOR” lines in the above trace file. This keyword identifies the Oracle command type, which is related to the V$SESSION.COMMAND column and the V$SQL.COMMAND_TYPE column. Common command type values include: 1 - CREATE TABLE 2 - INSERT 3 - SELECT 6 - UPDATE 7 - DELETE 9 - CREATE INDEX See the “Command Column of V$SESSION and Corresponding Commands” table in the Oracle Reference documentation (Table 8-2 in the Oracle Database Reference 11g Release 2 book) for a complete list of command types. For more information about 10046 trace files, see Chapter 8, “Understanding Performance Optimization Methods”, in the book “Expert Oracle Practices: Oracle Database Administration from the Oak Table” (the chapter was co-written by Randolf Geist and myself). The book “Optimizing Oracle Performance” is also highly recommended. Comments : 1 Comment » Categories : Bind Variable, Performance Explain Plan Lies, Autotrace Lies, TKPROF Lies, What is the Plan? 11 01 2010 January 11, 2010 As some of you might be aware, EXPLAIN PLAN will occasionally show the wrong execution plan for a SQL statement. SQL*Plus’ AUTOTRACE feature experiences a similar problem with generating accurate plans from time to time, especially when the SQL statement uses bind variables. Did you know that TKPROF may also show the wrong execution plan from time to time? I set up a simple test case to demonstrate this behavior. The test case tables: CREATE TABLE T1( C1 NUMBER, C2 VARCHAR2(255), PRIMARY KEY (C1)); CREATE TABLE T2( C1 NUMBER, C2 VARCHAR2(255), PRIMARY KEY (C1)); INSERT INTO T1 SELECT ROWNUM, LPAD('A',255,'A') FROM (SELECT ROWNUM RN FROM DUAL CONNECT BY LEVEL<=10000) V1, (SELECT ROWNUM RN FROM DUAL CONNECT BY LEVEL<=30) V2; INSERT INTO T2 SELECT ROWNUM, LPAD('A',255,'A') FROM (SELECT ROWNUM RN FROM DUAL CONNECT BY LEVEL<=10000) V1, (SELECT ROWNUM RN FROM DUAL CONNECT BY LEVEL<=30) V2; COMMIT; EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T1',CASCADE=>TRUE) EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T2',CASCADE=>TRUE) The above creates two tables, each with 300,000 rows with a primary key column (thus an index exists for the column), and a second column that acts as padding to make the row longer (to discourage full table scans). The test script that I built looks like this: ALTER SESSION SET STATISTICS_LEVEL='ALL'; VARIABLE N1 NUMBER VARIABLE N2 NUMBER SET AUTOTRACE TRACEONLY EXPLAIN SET LINESIZE 150 SET PAGESIZE 2000 SET ARRAYSIZE 100 SPOOL DIFF_EXPLAIN_PLAN_TEST.TXT EXEC :N1:=1 EXEC :N2:=2 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; EXEC :N1:=1 EXEC :N2:=10 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; EXEC :N1:=1 EXEC :N2:=100 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; EXEC :N1:=1 EXEC :N2:=1000 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; EXEC :N1:=1 EXEC :N2:=10000 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; EXEC :N1:=1 EXEC :N2:=100000 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; EXEC :N1:=1 EXEC :N2:=300000 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; /* --------------------------------------------------- */ ALTER SYSTEM FLUSH SHARED_POOL; SET AUTOTRACE OFF EXEC :N1:=1 EXEC :N2:=2 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); EXEC :N1:=1 EXEC :N2:=10 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); EXEC :N1:=1 EXEC :N2:=100 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); EXEC :N1:=1 EXEC :N2:=1000 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); EXEC :N1:=1 EXEC :N2:=10000 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); EXEC :N1:=1 EXEC :N2:=100000 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); EXEC :N1:=1 EXEC :N2:=300000 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); /* ------------------------------ */ SET AUTOTRACE TRACEONLY STATISTICS ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 8'; ALTER SESSION SET TRACEFILE_IDENTIFIER = 'DIFF_EXPLAIN_PLAN_TEST'; EXEC :N1:=1 EXEC :N2:=2 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; EXEC :N1:=1 EXEC :N2:=10 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; EXEC :N1:=1 EXEC :N2:=100 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; EXEC :N1:=1 EXEC :N2:=1000 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; EXEC :N1:=1 EXEC :N2:=10000 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; EXEC :N1:=1 EXEC :N2:=100000 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; EXEC :N1:=1 EXEC :N2:=300000 SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2; ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT OFF'; SET AUTOTRACE OFF SPOOL OFF There are three sections in the script: Display the AUTOTRACE generated execution plans without executing the queries, adjusting the bind variable values before each execution. Execute the SQL statement and display the actual execution plan using DBMS_XPLAN, adjusting the bind variable values before each execution (note that the flush of the shared pool seems to be required, the SQL statement is executed a couple times with each set of bind variable values to trigger adaptive cursor sharing in Oracle 11.1.0.6 and above). Execute the SQL statement displaying only the execution statistics, with a 10046 trace at level 8 enabled. What were the results on Oracle 11.1.0.7 (you may see different results on 11.1.0.6 and 11.2.0.1)? Section 1 of the output (slightly cleaned up) follows – keep on eye on the predicted number of rows and the Predicate Information section: SQL> EXEC :N1:=1 SQL> EXEC :N2:=2 SQL> SQL> SELECT 2 T1.C1, 3 T2.C1, 4 SUBSTR(T1.C2,1,1) T1_C2, 5 SUBSTR(T2.C2,1,1) T2_C2 6 FROM 7 T1, 8 T2 9 WHERE 10 T1.C1=T2.C1 11 AND T1.C1 BETWEEN :N1 AND :N2; Execution Plan ---------------------------------------------------------- Plan hash value: 2267210268 ------------------------------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 15000 | 7617K| | 498 (1)| 00:00:06 | |* 1 | FILTER | | | | | | | |* 2 | HASH JOIN | | 15000 | 7617K| 3992K| 498 (1)| 00:00:06 | | 3 | TABLE ACCESS BY INDEX ROWID| T1 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 4 | INDEX RANGE SCAN | SYS_C0030339 | 1350 | | | 4 (0)| 00:00:01 | | 5 | TABLE ACCESS BY INDEX ROWID| T2 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 6 | INDEX RANGE SCAN | SYS_C0030340 | 1350 | | | 4 (0)| 00:00:01 | ------------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(TO_NUMBER(:N1)<=TO_NUMBER(:N2)) 2 - access("T1"."C1"="T2"."C1") 4 - access("T1"."C1">=TO_NUMBER(:N1) AND "T1"."C1"<=TO_NUMBER(:N2)) 6 - access("T2"."C1">=TO_NUMBER(:N1) AND "T2"."C1"<=TO_NUMBER(:N2)) SQL> SQL> EXEC :N1:=1 SQL> EXEC :N2:=10 SQL> SQL> SELECT 2 T1.C1, 3 T2.C1, 4 SUBSTR(T1.C2,1,1) T1_C2, 5 SUBSTR(T2.C2,1,1) T2_C2 6 FROM 7 T1, 8 T2 9 WHERE 10 T1.C1=T2.C1 11 AND T1.C1 BETWEEN :N1 AND :N2; Execution Plan ---------------------------------------------------------- Plan hash value: 2267210268 ------------------------------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 15000 | 7617K| | 498 (1)| 00:00:06 | |* 1 | FILTER | | | | | | | |* 2 | HASH JOIN | | 15000 | 7617K| 3992K| 498 (1)| 00:00:06 | | 3 | TABLE ACCESS BY INDEX ROWID| T1 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 4 | INDEX RANGE SCAN | SYS_C0030339 | 1350 | | | 4 (0)| 00:00:01 | | 5 | TABLE ACCESS BY INDEX ROWID| T2 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 6 | INDEX RANGE SCAN | SYS_C0030340 | 1350 | | | 4 (0)| 00:00:01 | ------------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(TO_NUMBER(:N1)<=TO_NUMBER(:N2)) 2 - access("T1"."C1"="T2"."C1") 4 - access("T1"."C1">=TO_NUMBER(:N1) AND "T1"."C1"<=TO_NUMBER(:N2)) 6 - access("T2"."C1">=TO_NUMBER(:N1) AND "T2"."C1"<=TO_NUMBER(:N2)) SQL> SQL> EXEC :N1:=1 SQL> EXEC :N2:=100 SQL> SQL> SELECT 2 T1.C1, 3 T2.C1, 4 SUBSTR(T1.C2,1,1) T1_C2, 5 SUBSTR(T2.C2,1,1) T2_C2 6 FROM 7 T1, 8 T2 9 WHERE 10 T1.C1=T2.C1 11 AND T1.C1 BETWEEN :N1 AND :N2; Execution Plan ---------------------------------------------------------- Plan hash value: 2267210268 ------------------------------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 15000 | 7617K| | 498 (1)| 00:00:06 | |* 1 | FILTER | | | | | | | |* 2 | HASH JOIN | | 15000 | 7617K| 3992K| 498 (1)| 00:00:06 | | 3 | TABLE ACCESS BY INDEX ROWID| T1 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 4 | INDEX RANGE SCAN | SYS_C0030339 | 1350 | | | 4 (0)| 00:00:01 | | 5 | TABLE ACCESS BY INDEX ROWID| T2 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 6 | INDEX RANGE SCAN | SYS_C0030340 | 1350 | | | 4 (0)| 00:00:01 | ------------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(TO_NUMBER(:N1)<=TO_NUMBER(:N2)) 2 - access("T1"."C1"="T2"."C1") 4 - access("T1"."C1">=TO_NUMBER(:N1) AND "T1"."C1"<=TO_NUMBER(:N2)) 6 - access("T2"."C1">=TO_NUMBER(:N1) AND "T2"."C1"<=TO_NUMBER(:N2)) SQL> SQL> EXEC :N1:=1 SQL> EXEC :N2:=1000 SQL> SQL> SELECT 2 T1.C1, 3 T2.C1, 4 SUBSTR(T1.C2,1,1) T1_C2, 5 SUBSTR(T2.C2,1,1) T2_C2 6 FROM 7 T1, 8 T2 9 WHERE 10 T1.C1=T2.C1 11 AND T1.C1 BETWEEN :N1 AND :N2; Execution Plan ---------------------------------------------------------- Plan hash value: 2267210268 ------------------------------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 15000 | 7617K| | 498 (1)| 00:00:06 | |* 1 | FILTER | | | | | | | |* 2 | HASH JOIN | | 15000 | 7617K| 3992K| 498 (1)| 00:00:06 | | 3 | TABLE ACCESS BY INDEX ROWID| T1 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 4 | INDEX RANGE SCAN | SYS_C0030339 | 1350 | | | 4 (0)| 00:00:01 | | 5 | TABLE ACCESS BY INDEX ROWID| T2 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 6 | INDEX RANGE SCAN | SYS_C0030340 | 1350 | | | 4 (0)| 00:00:01 | ------------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(TO_NUMBER(:N1)<=TO_NUMBER(:N2)) 2 - access("T1"."C1"="T2"."C1") 4 - access("T1"."C1">=TO_NUMBER(:N1) AND "T1"."C1"<=TO_NUMBER(:N2)) 6 - access("T2"."C1">=TO_NUMBER(:N1) AND "T2"."C1"<=TO_NUMBER(:N2)) SQL> SQL> EXEC :N1:=1 SQL> EXEC :N2:=10000 SQL> SQL> SELECT 2 T1.C1, 3 T2.C1, 4 SUBSTR(T1.C2,1,1) T1_C2, 5 SUBSTR(T2.C2,1,1) T2_C2 6 FROM 7 T1, 8 T2 9 WHERE 10 T1.C1=T2.C1 11 AND T1.C1 BETWEEN :N1 AND :N2; Execution Plan ---------------------------------------------------------- Plan hash value: 2267210268 ------------------------------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 15000 | 7617K| | 498 (1)| 00:00:06 | |* 1 | FILTER | | | | | | | |* 2 | HASH JOIN | | 15000 | 7617K| 3992K| 498 (1)| 00:00:06 | | 3 | TABLE ACCESS BY INDEX ROWID| T1 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 4 | INDEX RANGE SCAN | SYS_C0030339 | 1350 | | | 4 (0)| 00:00:01 | | 5 | TABLE ACCESS BY INDEX ROWID| T2 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 6 | INDEX RANGE SCAN | SYS_C0030340 | 1350 | | | 4 (0)| 00:00:01 | ------------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(TO_NUMBER(:N1)<=TO_NUMBER(:N2)) 2 - access("T1"."C1"="T2"."C1") 4 - access("T1"."C1">=TO_NUMBER(:N1) AND "T1"."C1"<=TO_NUMBER(:N2)) 6 - access("T2"."C1">=TO_NUMBER(:N1) AND "T2"."C1"<=TO_NUMBER(:N2)) SQL> SQL> EXEC :N1:=1 SQL> EXEC :N2:=100000 SQL> SQL> SELECT 2 T1.C1, 3 T2.C1, 4 SUBSTR(T1.C2,1,1) T1_C2, 5 SUBSTR(T2.C2,1,1) T2_C2 6 FROM 7 T1, 8 T2 9 WHERE 10 T1.C1=T2.C1 11 AND T1.C1 BETWEEN :N1 AND :N2; Execution Plan ---------------------------------------------------------- Plan hash value: 2267210268 ------------------------------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 15000 | 7617K| | 498 (1)| 00:00:06 | |* 1 | FILTER | | | | | | | |* 2 | HASH JOIN | | 15000 | 7617K| 3992K| 498 (1)| 00:00:06 | | 3 | TABLE ACCESS BY INDEX ROWID| T1 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 4 | INDEX RANGE SCAN | SYS_C0030339 | 1350 | | | 4 (0)| 00:00:01 | | 5 | TABLE ACCESS BY INDEX ROWID| T2 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 6 | INDEX RANGE SCAN | SYS_C0030340 | 1350 | | | 4 (0)| 00:00:01 | ------------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(TO_NUMBER(:N1)<=TO_NUMBER(:N2)) 2 - access("T1"."C1"="T2"."C1") 4 - access("T1"."C1">=TO_NUMBER(:N1) AND "T1"."C1"<=TO_NUMBER(:N2)) 6 - access("T2"."C1">=TO_NUMBER(:N1) AND "T2"."C1"<=TO_NUMBER(:N2)) SQL> SQL> EXEC :N1:=1 SQL> EXEC :N2:=300000 SQL> SQL> SELECT 2 T1.C1, 3 T2.C1, 4 SUBSTR(T1.C2,1,1) T1_C2, 5 SUBSTR(T2.C2,1,1) T2_C2 6 FROM 7 T1, 8 T2 9 WHERE 10 T1.C1=T2.C1 11 AND T1.C1 BETWEEN :N1 AND :N2; Execution Plan ---------------------------------------------------------- Plan hash value: 2267210268 ------------------------------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 15000 | 7617K| | 498 (1)| 00:00:06 | |* 1 | FILTER | | | | | | | |* 2 | HASH JOIN | | 15000 | 7617K| 3992K| 498 (1)| 00:00:06 | | 3 | TABLE ACCESS BY INDEX ROWID| T1 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 4 | INDEX RANGE SCAN | SYS_C0030339 | 1350 | | | 4 (0)| 00:00:01 | | 5 | TABLE ACCESS BY INDEX ROWID| T2 | 15000 | 3808K| | 55 (0)| 00:00:01 | |* 6 | INDEX RANGE SCAN | SYS_C0030340 | 1350 | | | 4 (0)| 00:00:01 | ------------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(TO_NUMBER(:N1)<=TO_NUMBER(:N2)) 2 - access("T1"."C1"="T2"."C1") 4 - access("T1"."C1">=TO_NUMBER(:N1) AND "T1"."C1"<=TO_NUMBER(:N2)) 6 - access("T2"."C1">=TO_NUMBER(:N1) AND "T2"."C1"<=TO_NUMBER(:N2)) Notice that the optimizer is estimating that each table will return 15,000 of the 300,000 rows, 5% of the rows in the tables. Also notice in the Predicate Information section that AUTOTRACE is treating each of the bind variables as if it were defined as a VARCHAR2, rather than a NUMBER. Next, we will move on to the second section of the script to see the actual execution plans. Keep an eye on the E-Rows column, the execution plan, and the order of the query results. SQL> EXEC :N1:=1 SQL> EXEC :N2:=2 SQL> SELECT 2 T1.C1, 3 T2.C1, 4 SUBSTR(T1.C2,1,1) T1_C2, 5 SUBSTR(T2.C2,1,1) T2_C2 6 FROM 7 T1, 8 T2 9 WHERE 10 T1.C1=T2.C1 11 AND T1.C1 BETWEEN :N1 AND :N2; C1 C1 T T ---------- ---------- - - 1 1 A A 2 2 A A SQL_ID bgjafhqwjt1zt, child number 0 Plan hash value: 4256353061 --------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 2 |00:00:00.01 | 11 | |* 1 | FILTER | | 1 | | 2 |00:00:00.01 | 11 | | 2 | NESTED LOOPS | | 1 | | 2 |00:00:00.01 | 11 | | 3 | NESTED LOOPS | | 1 | 1 | 2 |00:00:00.01 | 9 | | 4 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 2 | 2 |00:00:00.01 | 5 | |* 5 | INDEX RANGE SCAN | SYS_C0030339 | 1 | 2 | 2 |00:00:00.01 | 3 | |* 6 | INDEX UNIQUE SCAN | SYS_C0030340 | 2 | 1 | 2 |00:00:00.01 | 4 | | 7 | TABLE ACCESS BY INDEX ROWID | T2 | 2 | 1 | 2 |00:00:00.01 | 2 | --------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 5 - access("T1"."C1">=:N1 AND "T1"."C1"<=:N2) 6 - access("T1"."C1"="T2"."C1") filter(("T2"."C1"<=:N2 AND "T2"."C1">=:N1)) SQL> EXEC :N1:=1 SQL> EXEC :N2:=10 SQL> SELECT 2 T1.C1, 3 T2.C1, 4 SUBSTR(T1.C2,1,1) T1_C2, 5 SUBSTR(T2.C2,1,1) T2_C2 6 FROM 7 T1, 8 T2 9 WHERE 10 T1.C1=T2.C1 11 AND T1.C1 BETWEEN :N1 AND :N2; C1 C1 T T ---------- ---------- - - 1 1 A A 2 2 A A 3 3 A A 4 4 A A 5 5 A A 6 6 A A 7 7 A A 8 8 A A 9 9 A A 10 10 A A SQL_ID bgjafhqwjt1zt, child number 0 Plan hash value: 4256353061 --------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 10 |00:00:00.01 | 24 | |* 1 | FILTER | | 1 | | 10 |00:00:00.01 | 24 | | 2 | NESTED LOOPS | | 1 | | 10 |00:00:00.01 | 24 | | 3 | NESTED LOOPS | | 1 | 1 | 10 |00:00:00.01 | 14 | | 4 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 2 | 10 |00:00:00.01 | 5 | |* 5 | INDEX RANGE SCAN | SYS_C0030339 | 1 | 2 | 10 |00:00:00.01 | 3 | |* 6 | INDEX UNIQUE SCAN | SYS_C0030340 | 10 | 1 | 10 |00:00:00.01 | 9 | | 7 | TABLE ACCESS BY INDEX ROWID | T2 | 10 | 1 | 10 |00:00:00.01 | 10 | --------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 5 - access("T1"."C1">=:N1 AND "T1"."C1"<=:N2) 6 - access("T1"."C1"="T2"."C1") filter(("T2"."C1"<=:N2 AND "T2"."C1">=:N1)) SQL_ID bgjafhqwjt1zt, child number 0 Plan hash value: 4256353061 --------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 100 |00:00:00.01 | 117 | |* 1 | FILTER | | 1 | | 100 |00:00:00.01 | 117 | | 2 | NESTED LOOPS | | 1 | | 100 |00:00:00.01 | 117 | | 3 | NESTED LOOPS | | 1 | 1 | 100 |00:00:00.01 | 17 | | 4 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 2 | 100 |00:00:00.01 | 8 | |* 5 | INDEX RANGE SCAN | SYS_C0030339 | 1 | 2 | 100 |00:00:00.01 | 3 | |* 6 | INDEX UNIQUE SCAN | SYS_C0030340 | 100 | 1 | 100 |00:00:00.01 | 9 | | 7 | TABLE ACCESS BY INDEX ROWID | T2 | 100 | 1 | 100 |00:00:00.01 | 100 | --------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 5 - access("T1"."C1">=:N1 AND "T1"."C1"<=:N2) 6 - access("T1"."C1"="T2"."C1") filter(("T2"."C1"<=:N2 AND "T2"."C1">=:N1)) SQL> EXEC :N1:=1 SQL> EXEC :N2:=1000 C1 C1 T T ---------- ---------- - - 1 1 A A 2 2 A A 3 3 A A 4 4 A A 5 5 A A ... 997 997 A A 998 998 A A 999 999 A A 1000 1000 A A SQL_ID bgjafhqwjt1zt, child number 0 Plan hash value: 4256353061 --------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | --------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 1000 |00:00:00.01 | 1087 | |* 1 | FILTER | | 1 | | 1000 |00:00:00.01 | 1087 | | 2 | NESTED LOOPS | | 1 | | 1000 |00:00:00.01 | 1087 | | 3 | NESTED LOOPS | | 1 | 1 | 1000 |00:00:00.01 | 87 | | 4 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 2 | 1000 |00:00:00.01 | 61 | |* 5 | INDEX RANGE SCAN | SYS_C0030339 | 1 | 2 | 1000 |00:00:00.01 | 13 | |* 6 | INDEX UNIQUE SCAN | SYS_C0030340 | 1000 | 1 | 1000 |00:00:00.01 | 26 | | 7 | TABLE ACCESS BY INDEX ROWID | T2 | 1000 | 1 | 1000 |00:00:00.01 | 1000 | --------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 5 - access("T1"."C1">=:N1 AND "T1"."C1"<=:N2) 6 - access("T1"."C1"="T2"."C1") filter(("T2"."C1"<=:N2 AND "T2"."C1">=:N1)) SQL> EXEC :N1:=1 SQL> EXEC :N2:=10000 C1 C1 T T ---------- ---------- - - 1 1 A A 2 2 A A 3 3 A A 4 4 A A 5 5 A A ... 9997 9997 A A 9998 9998 A A 9999 9999 A A 10000 10000 A A SQL_ID bgjafhqwjt1zt, child number 1 Plan hash value: 2267210268 ----------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | OMem | 1Mem | Used-Mem | ----------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 10000 |00:00:00.03 | 975 | | | | |* 1 | FILTER | | 1 | | 10000 |00:00:00.03 | 975 | | | | |* 2 | HASH JOIN | | 1 | 9999 | 10000 |00:00:00.03 | 975 | 3337K| 921K| 3324K (0)| | 3 | TABLE ACCESS BY INDEX ROWID| T1 | 1 | 10000 | 10000 |00:00:00.01 | 390 | | | | |* 4 | INDEX RANGE SCAN | SYS_C0030339 | 1 | 10000 | 10000 |00:00:00.01 | 19 | | | | | 5 | TABLE ACCESS BY INDEX ROWID| T2 | 1 | 10000 | 10000 |00:00:00.01 | 585 | | | | |* 6 | INDEX RANGE SCAN | SYS_C0030340 | 1 | 10000 | 10000 |00:00:00.01 | 118 | | | | ----------------------------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 2 - access("T1"."C1"="T2"."C1") 4 - access("T1"."C1">=:N1 AND "T1"."C1"<=:N2) 6 - access("T2"."C1">=:N1 AND "T2"."C1"<=:N2) SQL> EXEC :N1:=1 SQL> EXEC :N2:=100000 C1 C1 T T ---------- ---------- - - 28 28 A A 29 29 A A 30 30 A A 31 31 A A 32 32 A A ... 98763 98763 A A 98764 98764 A A 98765 98765 A A 98766 98766 A A SQL_ID bgjafhqwjt1zt, child number 2 Plan hash value: 487071653 ----------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | OMem | 1Mem | Used-Mem | ----------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 100K|00:00:00.29 | 23219 | | | | |* 1 | FILTER | | 1 | | 100K|00:00:00.29 | 23219 | | | | |* 2 | HASH JOIN | | 1 | 100K| 100K|00:00:00.29 | 23219 | 28M| 3683K| 29M (0)| |* 3 | TABLE ACCESS FULL| T1 | 1 | 100K| 100K|00:00:00.03 | 11128 | | | | |* 4 | TABLE ACCESS FULL| T2 | 1 | 100K| 100K|00:00:00.03 | 12091 | | | | ----------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 2 - access("T1"."C1"="T2"."C1") 3 - filter(("T1"."C1"<=:N2 AND "T1"."C1">=:N1)) 4 - filter(("T2"."C1"<=:N2 AND "T2"."C1">=:N1)) SQL> EXEC :N1:=1 SQL> EXEC :N2:=300000 C1 C1 T T ---------- ---------- - - 31 31 A A 34 34 A A 35 35 A A 37 37 A A 44 44 A A ... 276467 276467 A A 276910 276910 A A 277750 277750 A A 277771 277771 A A SQL_ID bgjafhqwjt1zt, child number 3 Plan hash value: 487071653 --------------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | Writes | OMem | 1Mem | Used-Mem | Used-Tmp| --------------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 300K|00:00:06.67 | 23351 | 15996 | 15996 | | | | | |* 1 | FILTER | | 1 | | 300K|00:00:06.67 | 23351 | 15996 | 15996 | | | | | |* 2 | HASH JOIN | | 1 | 300K| 300K|00:00:06.37 | 23351 | 15996 | 15996 | 85M| 7366K| 43M (1)| 132K| |* 3 | TABLE ACCESS FULL| T1 | 1 | 300K| 300K|00:00:00.01 | 11128 | 0 | 0 | | | | | |* 4 | TABLE ACCESS FULL| T2 | 1 | 300K| 300K|00:00:00.01 | 12223 | 0 | 0 | | | | | --------------------------------------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 2 - access("T1"."C1"="T2"."C1") 3 - filter(("T1"."C1">=:N1 AND "T1"."C1"<=:N2)) 4 - filter(("T2"."C1">=:N1 AND "T2"."C1"<=:N2)) Next, we will move on to the third section of the script to see the output of TKPROF. The Oracle 11.1.0.6 client home happened to be first in my path before any other Oracle home. I renamed the trace file from the server and processed it with TKPROF: tkprof DIFF_EXPLAIN_PLAN_TEST.trc DIFF_EXPLAIN_PLAN_TEST_TRC.txt What was in the DIFF_EXPLAIN_PLAN_TEST_TRC.txt file generated by TKPROF? ******************************************************************************** SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 7 0.00 0.00 0 0 0 0 Execute 7 0.01 0.03 0 0 0 0 Fetch 4120 3.52 10.91 18216 47844 0 411112 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4134 3.54 10.95 18216 47844 0 411112 Misses in library cache during parse: 0 Misses in library cache during execute: 4 Optimizer mode: ALL_ROWS Parsing user id: 518 Rows Row Source Operation ------- --------------------------------------------------- 2 FILTER (cr=11 pr=0 pw=0 time=0 us) 2 NESTED LOOPS (cr=11 pr=0 pw=0 time=0 us) 2 NESTED LOOPS (cr=9 pr=0 pw=0 time=0 us cost=5 size=520 card=1) 2 TABLE ACCESS BY INDEX ROWID T1 (cr=5 pr=0 pw=0 time=0 us cost=3 size=520 card=2) 2 INDEX RANGE SCAN SYS_C0030337 (cr=3 pr=0 pw=0 time=0 us cost=2 size=0 card=2)(object id 87187) 2 INDEX UNIQUE SCAN SYS_C0030338 (cr=4 pr=0 pw=0 time=0 us cost=0 size=0 card=1)(object id 87189) 2 TABLE ACCESS BY INDEX ROWID T2 (cr=2 pr=0 pw=0 time=0 us cost=1 size=260 card=1) Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ SQL*Net message to client 4120 0.00 0.02 SQL*Net message from client 4120 0.01 3.21 direct path write temp 1128 0.99 2.06 direct path read temp 1128 0.17 5.10 ******************************************************************************** Interesting, all seven executions with different bind variable values used the same execution plan. That means that this command lied?: SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); Wait a minute, that is not in the title of this article. If only I knew how to read a raw 10046 extended SQL trace, I could read this: STAT #8 id=1 cnt=2 pid=0 pos=1 obj=0 op='FILTER (cr=11 pr=0 pw=0 time=0 us)' STAT #8 id=2 cnt=2 pid=1 pos=1 obj=0 op='NESTED LOOPS (cr=11 pr=0 pw=0 time=0 us)' STAT #8 id=3 cnt=2 pid=2 pos=1 obj=0 op='NESTED LOOPS (cr=9 pr=0 pw=0 time=0 us cost=5 size=520 card=1)' STAT #8 id=4 cnt=2 pid=3 pos=1 obj=87264 op='TABLE ACCESS BY INDEX ROWID T1 (cr=5 pr=0 pw=0 time=0 us cost=3 size=520 card=2)' STAT #8 id=5 cnt=2 pid=4 pos=1 obj=87265 op='INDEX RANGE SCAN SYS_C0030339 (cr=3 pr=0 pw=0 time=0 us cost=2 size=0 card=2)' STAT #8 id=6 cnt=2 pid=3 pos=2 obj=87267 op='INDEX UNIQUE SCAN SYS_C0030340 (cr=4 pr=0 pw=0 time=0 us cost=0 size=0 card=1)' STAT #8 id=7 cnt=2 pid=2 pos=2 obj=87266 op='TABLE ACCESS BY INDEX ROWID T2 (cr=2 pr=0 pw=0 time=0 us cost=1 size=260 card=1)' ... STAT #5 id=1 cnt=300000 pid=0 pos=1 obj=0 op='FILTER (cr=23515 pr=15705 pw=15705 time=4741368 us)' STAT #5 id=2 cnt=300000 pid=1 pos=1 obj=0 op='HASH JOIN (cr=23515 pr=15705 pw=15705 time=4741368 us cost=13774 size=156000000 card=300000)' STAT #5 id=3 cnt=300000 pid=2 pos=1 obj=87264 op='TABLE ACCESS FULL T1 (cr=11128 pr=0 pw=0 time=31126 us cost=3024 size=78000000 card=300000)' STAT #5 id=4 cnt=300000 pid=2 pos=2 obj=87266 op='TABLE ACCESS FULL T2 (cr=12387 pr=0 pw=0 time=155851 us cost=3024 size=78000000 card=300000)' If you do know how to read the STAT lines in a 10046 extended SQL trace, you would know that the execution plan definitely did change between the first and last execution when 10046 extended SQL tracing was enabled. We might be inclined to do some checking, like this: SELECT CHILD_NUMBER, EXECUTIONS, ROWS_PROCESSED, BUFFER_GETS, BIND_SET_HASH_VALUE FROM V$SQL_CS_STATISTICS WHERE SQL_ID='bgjafhqwjt1zt' ORDER BY CHILD_NUMBER; CHILD_NUMBER EXECUTIONS ROWS_PROCESSED BUFFER_GETS BIND_SET_HASH_VALUE ------------ ---------- -------------- ----------- ------------------- 0 1 12001 1087 722894381 0 1 25 124 2702357211 1 1 180000 975 982654583 2 1 1000000 23219 2772294946 3 1 3000000 23351 1545127490 4 1 25 11 2702357211 5 1 140 8 6258482 6 1 1400 14 4281096765 7 1 18000 102 722894381 Or, we might try doing something like this: SPOOL 'DIFF_EXPLAIN_PLANS_ADAPT.TXT' SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR('bgjafhqwjt1zt',NULL,'TYPICAL')); SPOOL OFF PLAN_TABLE_OUTPUT ------------------------------------------------------------------------------------------------------------------------------------------------------ SQL_ID bgjafhqwjt1zt, child number 0 ------------------------------------- SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 Plan hash value: 4256353061 ----------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ----------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 5 (100)| | |* 1 | FILTER | | | | | | | 2 | NESTED LOOPS | | | | | | | 3 | NESTED LOOPS | | 1 | 520 | 5 (0)| 00:00:01 | | 4 | TABLE ACCESS BY INDEX ROWID| T1 | 2 | 520 | 3 (0)| 00:00:01 | |* 5 | INDEX RANGE SCAN | SYS_C0030339 | 2 | | 2 (0)| 00:00:01 | |* 6 | INDEX UNIQUE SCAN | SYS_C0030340 | 1 | | 0 (0)| | | 7 | TABLE ACCESS BY INDEX ROWID | T2 | 1 | 260 | 1 (0)| 00:00:01 | ----------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 5 - access("T1"."C1">=:N1 AND "T1"."C1"<=:N2) 6 - access("T1"."C1"="T2"."C1") filter(("T2"."C1"<=:N2 AND "T2"."C1">=:N1)) SQL_ID bgjafhqwjt1zt, child number 1 ------------------------------------- SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 Plan hash value: 2267210268 ------------------------------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | | | | 1042 (100)| | |* 1 | FILTER | | | | | | | |* 2 | HASH JOIN | | 9999 | 5077K| 2664K| 1042 (1)| 00:00:13 | | 3 | TABLE ACCESS BY INDEX ROWID| T1 | 10000 | 2539K| | 391 (0)| 00:00:05 | |* 4 | INDEX RANGE SCAN | SYS_C0030339 | 10000 | | | 20 (0)| 00:00:01 | | 5 | TABLE ACCESS BY INDEX ROWID| T2 | 10000 | 2539K| | 391 (0)| 00:00:05 | |* 6 | INDEX RANGE SCAN | SYS_C0030340 | 10000 | | | 20 (0)| 00:00:01 | ------------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 2 - access("T1"."C1"="T2"."C1") 4 - access("T1"."C1">=:N1 AND "T1"."C1"<=:N2) 6 - access("T2"."C1">=:N1 AND "T2"."C1"<=:N2) SQL_ID bgjafhqwjt1zt, child number 2 ------------------------------------- SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 Plan hash value: 487071653 ------------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | | | | 8623 (100)| | |* 1 | FILTER | | | | | | | |* 2 | HASH JOIN | | 100K| 49M| 25M| 8623 (1)| 00:01:44 | |* 3 | TABLE ACCESS FULL| T1 | 100K| 24M| | 3023 (1)| 00:00:37 | |* 4 | TABLE ACCESS FULL| T2 | 100K| 24M| | 3023 (1)| 00:00:37 | ------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 2 - access("T1"."C1"="T2"."C1") 3 - filter(("T1"."C1"<=:N2 AND "T1"."C1">=:N1)) 4 - filter(("T2"."C1"<=:N2 AND "T2"."C1">=:N1)) SQL_ID bgjafhqwjt1zt, child number 3 ------------------------------------- SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 Plan hash value: 487071653 ------------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | | | | 13774 (100)| | |* 1 | FILTER | | | | | | | |* 2 | HASH JOIN | | 300K| 148M| 77M| 13774 (1)| 00:02:46 | |* 3 | TABLE ACCESS FULL| T1 | 300K| 74M| | 3024 (1)| 00:00:37 | |* 4 | TABLE ACCESS FULL| T2 | 300K| 74M| | 3024 (1)| 00:00:37 | ------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 2 - access("T1"."C1"="T2"."C1") 3 - filter(("T1"."C1">=:N1 AND "T1"."C1"<=:N2)) 4 - filter(("T2"."C1">=:N1 AND "T2"."C1"<=:N2)) SQL_ID bgjafhqwjt1zt, child number 4 ------------------------------------- SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 Plan hash value: 4256353061 ----------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ----------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 5 (100)| | |* 1 | FILTER | | | | | | | 2 | NESTED LOOPS | | | | | | | 3 | NESTED LOOPS | | 1 | 520 | 5 (0)| 00:00:01 | | 4 | TABLE ACCESS BY INDEX ROWID| T1 | 2 | 520 | 3 (0)| 00:00:01 | |* 5 | INDEX RANGE SCAN | SYS_C0030339 | 2 | | 2 (0)| 00:00:01 | |* 6 | INDEX UNIQUE SCAN | SYS_C0030340 | 1 | | 0 (0)| | | 7 | TABLE ACCESS BY INDEX ROWID | T2 | 1 | 260 | 1 (0)| 00:00:01 | ----------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 5 - access("T1"."C1">=:N1 AND "T1"."C1"<=:N2) 6 - access("T1"."C1"="T2"."C1") filter(("T2"."C1"<=:N2 AND "T2"."C1">=:N1)) SQL_ID bgjafhqwjt1zt, child number 5 ------------------------------------- SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 Plan hash value: 2267210268 ---------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ---------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 7 (100)| | |* 1 | FILTER | | | | | | |* 2 | HASH JOIN | | 9 | 4680 | 7 (15)| 00:00:01 | | 3 | TABLE ACCESS BY INDEX ROWID| T1 | 10 | 2600 | 3 (0)| 00:00:01 | |* 4 | INDEX RANGE SCAN | SYS_C0030339 | 10 | | 2 (0)| 00:00:01 | | 5 | TABLE ACCESS BY INDEX ROWID| T2 | 10 | 2600 | 3 (0)| 00:00:01 | |* 6 | INDEX RANGE SCAN | SYS_C0030340 | 10 | | 2 (0)| 00:00:01 | ---------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 2 - access("T1"."C1"="T2"."C1") 4 - access("T1"."C1">=:N1 AND "T1"."C1"<=:N2) 6 - access("T2"."C1">=:N1 AND "T2"."C1"<=:N2) SQL_ID bgjafhqwjt1zt, child number 6 ------------------------------------- SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 Plan hash value: 2267210268 ---------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ---------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 13 (100)| | |* 1 | FILTER | | | | | | |* 2 | HASH JOIN | | 99 | 51480 | 13 (8)| 00:00:01 | | 3 | TABLE ACCESS BY INDEX ROWID| T1 | 100 | 26000 | 6 (0)| 00:00:01 | |* 4 | INDEX RANGE SCAN | SYS_C0030339 | 100 | | 2 (0)| 00:00:01 | | 5 | TABLE ACCESS BY INDEX ROWID| T2 | 100 | 26000 | 6 (0)| 00:00:01 | |* 6 | INDEX RANGE SCAN | SYS_C0030340 | 100 | | 2 (0)| 00:00:01 | ---------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 2 - access("T1"."C1"="T2"."C1") 4 - access("T1"."C1">=:N1 AND "T1"."C1"<=:N2) 6 - access("T2"."C1">=:N1 AND "T2"."C1"<=:N2) SQL_ID bgjafhqwjt1zt, child number 7 ------------------------------------- SELECT T1.C1, T2.C1, SUBSTR(T1.C2,1,1) T1_C2, SUBSTR(T2.C2,1,1) T2_C2 FROM T1, T2 WHERE T1.C1=T2.C1 AND T1.C1 BETWEEN :N1 AND :N2 Plan hash value: 2267210268 ---------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ---------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 83 (100)| | |* 1 | FILTER | | | | | | |* 2 | HASH JOIN | | 999 | 507K| 83 (2)| 00:00:01 | | 3 | TABLE ACCESS BY INDEX ROWID| T1 | 1000 | 253K| 41 (0)| 00:00:01 | |* 4 | INDEX RANGE SCAN | SYS_C0030339 | 1000 | | 3 (0)| 00:00:01 | | 5 | TABLE ACCESS BY INDEX ROWID| T2 | 1000 | 253K| 41 (0)| 00:00:01 | |* 6 | INDEX RANGE SCAN | SYS_C0030340 | 1000 | | 3 (0)| 00:00:01 | ---------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(:N1<=:N2) 2 - access("T1"."C1"="T2"."C1") 4 - access("T1"."C1">=:N1 AND "T1"."C1"<=:N2) 6 - access("T2"."C1">=:N1 AND "T2"."C1"<=:N2) You might be wondering if TKPROF in 11.1.0.7 or 11.2.0.1 lie about the execution plan captured in an extended 10046 SQL trace. I would tell you, but this blog article is now too long. 🙂 Comments : 3 Comments » Categories : Bind Variable, SQL Hard Parses when Using Bind Variables? 31 12 2009 December 31, 2009 Assume that the following tables are created and then statistics are gathered: CREATE TABLE T3 AS SELECT ROWNUM C1, LPAD('A',100,'A') C2 FROM DUAL CONNECT BY LEVEL<=10000; CREATE TABLE T4 AS SELECT ROWNUM C1, LPAD('A',100,'A') C2 FROM DUAL CONNECT BY LEVEL<=10000; CREATE INDEX IND_T4 ON T4(C1); EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T3',CASCADE=>TRUE) EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T4',CASCADE=>TRUE) If you then set up SQL*Plus with the following commands: VARIABLE N1 NUMBER VARIABLE N2 NUMBER SET AUTOTRACE TRACEONLY STATISTICS SET ARRAYSIZE 1000 How many hard parses would you see for the session, and how many child cursors for the SQL statement will be in the library cache, if you do the following in SQL*Plus: EXEC :N1 := 1 EXEC :N2 := 1 SELECT T3.C1, T4.C2 FROM T3, T4 WHERE T3.C1 BETWEEN :N1 AND :N2 AND T3.C1=T4.C1; SELECT T3.C1, T4.C2 FROM T3, T4 WHERE T3.C1 BETWEEN :N1 AND :N2 AND T3.C1=T4.C1; SELECT T3.C1, T4.C2 FROM T3, T4 WHERE T3.C1 BETWEEN :N1 AND :N2 AND T3.C1=T4.C1; SELECT T3.C1, T4.C2 FROM T3, T4 WHERE T3.C1 BETWEEN :N1 AND :N2 AND T3.C1=T4.C1; Now, repeat the above 9,999 times in the same session, specifying random values for N1 and N2 such that: N1 >= 1 N2 <= 10,000 N1 <= N2 Does it matter if you test with Oracle 8.0.5 (assuming that you use the ANALYZE command rather than DBMS_STATS), 11.2.0.1, or something in between? Comments : Leave a Comment » Categories : Bind Variable, Performance Tracking Performance Problems – Inserting a Hint into SQL in a Compiled Program 18 12 2009 December 18, 2009 What follows is part of a presentation that I gave in 2008. The lead into this slide is that a report in an ERP package was running far slower than expected following an upgrade of the ERP package. Of course the SQL statements are hard coded into the ERP package, so there is not much that can be done, right? I created a 10046 extended SQL trace file at level 12, and then passed the trace file through my Toy Project for Performance Tuning (https://hoopercharles.wordpress.com/2009/12/13/toy-project-for-performance-tuning-2/). One of the outputs of my program’s 10046 trace file parser provides an overview of the trace file, as well as an overview of each SQL statement. A screen shot of that output follows: The screen shot shows that there were 10,399 execute calls in the trace file that consumed 47.67 seconds of the server’s CPU time, and that the elapsed time/wall clock time from the server’s perspective for executing the SQL statements is 50.32 seconds for the executions. There were 16,146 fetch calls that consumed 263.48 seconds of the server’s CPU time, and the elapsed time/wall clock time from the server’s perspective for fetching the rows for the SQL statements is 263.64 seconds. Note that there were 0 physical reads and 8,804,970 consistent gets during the fetch (what’s that buffer cache hit ratio?). The SQL*Net message from client wait totaled 107.33 seconds, but 42.62 seconds were in a single block of time (likely at the end of the trace file, just before tracing was disabled), so the actual total time waiting for the next request from the client is about 64.7 seconds in 22,825 round-trips. If we scroll down a bit, we might find a couple of the greatest contributors to the server-side processing: The screen shot shows that there are two groups of SQL statements that combined contributed to 86% of the total server-side processing time for the report. The first SQL statement group is identified as Cursor 16 Ver 1, and the second is identified as Cursor 17 Ver 1. We could then search the remainder of this file to locate those identifiers. The above screen shot shows that there are two identical SQL statements in the first group – the first SQL statement (Cursor 16 Ver 1) was parsed, but never executed. The row source execution plan from the 10046 trace file (in the STAT lines) shows that the optimizer decided to use the index X_RECEIVABLE_3 that is on the column ENTITY_ID (there are two distinct values for this column), rather than the much more selective index on the INVOICE_ID column. Notice the number of consistent gets and the CPU utilization for this one SQL statement that was executed 934 times. Why was the index selected? Would a DBMS_XPLAN help? The DBMS_XPLAN output shows that the optimizer estimated that the X_RECEIVABLE_3 index would return a single row, when in fact it returned 66124 rows. A problem where statistics were not collected in the last 15 years? No. Maybe a 10053 trace will help: In the above we see that the predicted number of rows returned with the X_RECEIVABLE_3 index will be less than that for the other indexes, and the expected CPU resources will also be slightly less, but something is not right. The optimizer selected not to use the primary key index on the RECEIVABLE table. Note that the calculated cost for all three indexes is 2. Let’s try an experiment with a NO_INDEX hint to prevent the optimizer from using the X_RECEIVABLE_3 index: In the above, notice that the primary key index (SYS_C006885) was selected by the optimizer, that the execution time dropped from 0.31 seconds to 0.01 seconds (or less), and the number of consistent gets dropped from 1311 to 5. A rather nice improvement, but how do we force the ERP package to not use the index without the ability to modify the program’s source code? We could do something like this, if we assume that bind variable peeking is the source of the problem: CREATE OR REPLACE TRIGGER LOGON_FIX_APP_PERF AFTER LOGON ON DATABASE DECLARE SHOULD_EXECUTE INTEGER; BEGIN SELECT DECODE(SUBSTR(UPPER(PROGRAM),1,2),'VM',1,'VF',1,0)+DECODE(INSTR(PROGRAM,'\',-1),0,0,DECODE(SUBSTR(UPPER(SUBSTR(PROGRAM,INSTR(PROGRAM,'\',-1)+1)),1,2),'VM',1,'VF',1,0)) INTO SHOULD_EXECUTE FROM V$SESSION WHERE SID=(SELECT SID FROM V$MYSTAT WHERE ROWNUM=1); IF SHOULD_EXECUTE > 0 THEN EXECUTE IMMEDIATE 'ALTER SESSION SET "_optim_peek_user_binds"=FALSE'; END IF; END; / The above logon trigger results in the following row source execution plans: But, the above approach may be too broad as it will affect every SQL statement executed by a program that begins with the letters VM or VF. It did help this report: The time to fetch all rows dropped from 264 seconds to just 11 seconds, and the full report displays in just over a minute. Let’s see if we are able to trick the optimizer into not using the X_RECEIVABLE_3 index without disabling bind variable peeking. First, we need to enable private outlines in the current session: ALTER SESSION SET USE_PRIVATE_OUTLINES=TRUE; For demonstration purposes, I will explicitly tell the optimizer to use the X_RECEIVABLE_3 index when creating the outline (the exact SQL statement used by the program should be used, without the hint): CREATE OR REPLACE PRIVATE OUTLINE P_RECEIVABLE1 ON select /*+ INDEX(I X_RECEIVABLE_3) */ sum(a.apply_amount) from RECV_MEMO_APPLY a, RECEIVABLE i where a.inv_invoice_id = :1 and a.apply_date <= :2 and a.inv_invoice_id = i.invoice_id and i.status != 'X' and i.total_amount != 0 and i.recv_gl_acct_id = :3 and ENTITY_ID = :4; Note that I had to specify a hint in this case as I was at the time testing in Oracle 11g R1 (11.1.0.6), which refused to use the X_RECEIVABLE_3 on its own to reproduce the problem seen with Oracle 10.2.0.x. Let’s view the hints generated by the optimizer when it created the outline: SELECT HINT#, HINT_TEXT FROM OL$HINTS WHERE OL_NAME='P_RECEIVABLE1'; HINT# HINT_TEXT 1 USE_NL(@"SEL$1" "A"@"SEL$1") 2 LEADING(@"SEL$1" "I"@"SEL$1" "A"@"SEL$1") 3 INDEX(@"SEL$1" "A"@"SEL$1" ("RECV_MEMO_APPLY"."INV_INVOICE_ID" "RECV_MEMO_APPLY"."MEMO_INVOICE_ID")) 4 INDEX(@"SEL$1" "I"@"SEL$1" ("RECEIVABLE"."ENTITY_ID")) 5 OUTLINE_LEAF(@"SEL$1") 6 ALL_ROWS 7 OPTIMIZER_FEATURES_ENABLE('10.2.0.2') 8 IGNORE_OPTIM_EMBEDDED_HINTS We can then verify that in fact the slow execution plan was selected: EXPLAIN PLAN FOR select /*+ INDEX(I X_RECEIVABLE_3) */ sum(a.apply_amount) from RECV_MEMO_APPLY a, RECEIVABLE i where a.inv_invoice_id = :1 and a.apply_date <= :2 and a.inv_invoice_id = i.invoice_id and i.status != 'X' and i.total_amount != 0 and i.recv_gl_acct_id = :3 and ENTITY_ID = :4; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY(NULL,NULL,'BASIC +NOTE')); ------------------------------------------------------------ | Id | Operation | Name | ------------------------------------------------------------ | 0 | SELECT STATEMENT | | | 1 | SORT AGGREGATE | | | 2 | TABLE ACCESS BY INDEX ROWID | RECV_MEMO_APPLY | | 3 | NESTED LOOPS | | | 4 | TABLE ACCESS BY INDEX ROWID| RECEIVABLE | | 5 | INDEX RANGE SCAN | X_RECEIVABLE_3 | | 6 | INDEX RANGE SCAN | X_RECV_MEMO_APP_1 | ------------------------------------------------------------ Next, we will create a stored outline that explicitly specifies not to use the X_RECEIVABLE_3 index: CREATE OR REPLACE PRIVATE OUTLINE P_RECEIVABLE_TEMP ON select /*+ NO_INDEX(I X_RECEIVABLE_3) */ sum(a.apply_amount) from RECV_MEMO_APPLY a, RECEIVABLE i where a.inv_invoice_id = :1 and a.apply_date <= :2 and a.inv_invoice_id = i.invoice_id and i.status != 'X' and i.total_amount != 0 and i.recv_gl_acct_id = :3 and ENTITY_ID = :4; Let’s view the hints generated by the optimizer when it created the outline: SELECT HINT#, HINT_TEXT FROM OL$HINTS WHERE OL_NAME='P_RECEIVABLE_TEMP'; HINT# HINT_TEXT 1 USE_NL(@"SEL$1" "A"@"SEL$1") 2 LEADING(@"SEL$1" "I"@"SEL$1" "A"@"SEL$1") 3 INDEX(@"SEL$1" "A"@"SEL$1" ("RECV_MEMO_APPLY"."INV_INVOICE_ID" "RECV_MEMO_APPLY"."MEMO_INVOICE_ID")) 4 INDEX(@"SEL$1" "I"@"SEL$1" ("RECEIVABLE"."INVOICE_ID")) 5 OUTLINE_LEAF(@"SEL$1") 6 ALL_ROWS 7 OPTIMIZER_FEATURES_ENABLE('10.2.0.2') 8 IGNORE_OPTIM_EMBEDDED_HINTS We can then verify that the faster execution plan was selected: EXPLAIN PLAN FOR select /*+ NO_INDEX(I X_RECEIVABLE_3) */ sum(a.apply_amount) from RECV_MEMO_APPLY a, RECEIVABLE i where a.inv_invoice_id = :1 and a.apply_date <= :2 and a.inv_invoice_id = i.invoice_id and i.status != 'X' and i.total_amount != 0 and i.recv_gl_acct_id = :3 and ENTITY_ID = :4; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY(NULL,NULL,'BASIC +NOTE')); ----------------------------------------------------------- | Id | Operation | Name | ----------------------------------------------------------- | 0 | SELECT STATEMENT | | | 1 | SORT AGGREGATE | | | 2 | NESTED LOOPS | | | 3 | TABLE ACCESS BY INDEX ROWID| RECEIVABLE | | 4 | INDEX UNIQUE SCAN | SYS_C0011925 | | 5 | TABLE ACCESS BY INDEX ROWID| RECV_MEMO_APPLY | | 6 | INDEX RANGE SCAN | X_RECV_MEMO_APP_1 | ----------------------------------------------------------- Next is the potentially dangerous part – refer to Metalink Notes:604022.1, 726802.1, 730062.1, 144194.1, 728647.1, 5893396.8, as well as http://www.jlcomp.demon.co.uk/04_outlines.rtf We delete the outline hints from the outline with the slow execution plan (the one with the INDEX(I X_RECEIVABLE_3) hint in this case): DELETE FROM OL$HINTS WHERE OL_NAME='P_RECEIVABLE1'; Then we copy the hints from the outline for the fast execution plan (with our NO_INDEX hint): INSERT INTO OL$HINTS SELECT 'P_RECEIVABLE1', HINT#, CATEGORY, HINT_TYPE, HINT_TEXT, STAGE#, NODE#, TABLE_NAME, TABLE_TIN, TABLE_POS, REF_ID, USER_TABLE_NAME, COST, CARDINALITY, BYTES, HINT_TEXTOFF, HINT_TEXTLEN, JOIN_PRED, SPARE1, SPARE2, HINT_STRING FROM OL$HINTS WHERE OL_NAME='P_RECEIVABLE_TEMP'; COMMIT; Then we instruct Oracle to refresh the private outline: EXEC DBMS_OUTLN_EDIT.REFRESH_PRIVATE_OUTLINE('P_RECEIVABLE1') Now, let’s try generating the plan for the original query again (with the forced/hinted X_RECEIVABLE_3 index usage): EXPLAIN PLAN FOR select /*+ INDEX(I X_RECEIVABLE_3) */ sum(a.apply_amount) from RECV_MEMO_APPLY a, RECEIVABLE i where a.inv_invoice_id = :1 and a.apply_date <= :2 and a.inv_invoice_id = i.invoice_id and i.status != 'X' and i.total_amount != 0 and i.recv_gl_acct_id = :3 and ENTITY_ID = :4; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY(NULL,NULL,'BASIC +NOTE')); ----------------------------------------------------------- | Id | Operation | Name | ----------------------------------------------------------- | 0 | SELECT STATEMENT | | | 1 | SORT AGGREGATE | | | 2 | NESTED LOOPS | | | 3 | TABLE ACCESS BY INDEX ROWID| RECEIVABLE | | 4 | INDEX UNIQUE SCAN | SYS_C0011925 | | 5 | TABLE ACCESS BY INDEX ROWID| RECV_MEMO_APPLY | | 6 | INDEX RANGE SCAN | X_RECV_MEMO_APP_1 | ----------------------------------------------------------- Note that even though we hinted/forced Oracle to use the X_RECEIVABLE_3 index, it now selected to use the primary key index SYS_C0011925 due to the hacked private outline. Oracle IGNORED MY HINT (actually, it did exactly as the outline instructed). Now, let’s make the outline a bit more permanent, converting it from a private outline to a public outline: CREATE PUBLIC OUTLINE PP_RECEIVABLE1 FROM PRIVATE P_RECEIVABLE1; ALTER SYSTEM SET USE_STORED_OUTLINES=TRUE; DROP PRIVATE OUTLINE P_RECEIVABLE1; DROP PRIVATE OUTLINE P_RECEIVABLE_TEMP; If we then generate a DBMS_XPLAN for the original query (in my example, the one with the forced index hint), we see the following: ------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | ------------------------------------------------------------------------------------------------------------- | 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:00.01 | 5 | | 2 | NESTED LOOPS | | 1 | 1 | 0 |00:00:00.01 | 5 | |* 3 | TABLE ACCESS BY INDEX ROWID| RECEIVABLE | 1 | 1 | 1 |00:00:00.01 | 3 | |* 4 | INDEX UNIQUE SCAN | SYS_C0011925 | 1 | 1 | 1 |00:00:00.01 | 2 | |* 5 | TABLE ACCESS BY INDEX ROWID| RECV_MEMO_APPLY | 1 | 1 | 0 |00:00:00.01 | 2 | |* 6 | INDEX RANGE SCAN | X_RECV_MEMO_APP_1 | 1 | 1 | 0 |00:00:00.01 | 2 | ------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 3 - filter(("I"."STATUS"<>'X' AND "I"."TOTAL_AMOUNT"<>0 AND "I"."RECV_GL_ACCT_ID"=:3 AND "ENTITY_ID"=:4)) 4 - access("I"."INVOICE_ID"=:1) 5 - filter("A"."APPLY_DATE"<=:2) 6 - access("A"."INV_INVOICE_ID"=:1) Note ----- - outline "PP_RECEIVABLE1" used for this statement But there is a catch. The USE_STORED_OUTLINES parameter cannot be set in the init.ora or spfile, so we need a STARTUP trigger to set the parameter: CREATE OR REPLACE TRIGGER ENABLE_OUTLINES_TRIG AFTER STARTUP ON DATABASE BEGIN EXECUTE IMMEDIATE('ALTER SYSTEM SET USE_STORED_OUTLINES=TRUE'); END; After implementing the stored outlines there is a new top SQL statement in the 10046 trace profile file. But, at some point we have to stop (before the effects of compulsive tuning disorder are achieved). My original idea for hacking the stored outlines came from the book “Troubleshooting Oracle Performance”. Note that hacking stored outlines should not be the first step. Instead, see if it is possible to modify optimizer parameters at the session level first to achieve the desired execution plan. Once the desired execution plan is achieved, create the stored outline to freeze the execution plan. Comments : 1 Comment » Categories : Bind Variable, Performance, SQL Using Histograms to Fix Bind Peeking Problems? 30 11 2009 November 30, 2009 The following question appeared on the OTN forums (http://forums.oracle.com/forums/thread.jspa?threadID=993929): What is the solution to [bind] variable peeking without going to 11g? i got answer before as stored outline , but i don’t think this will fix it as stored outlines will stablise the plan which we don’t want , i think histogram is a better solution? Consider the following test case, which might leave you wondering if creating a histogram on a column used by bind variables is a good idea. The set up: SHOW PARAMETER OPTIMIZER NAME TYPE VALUE ------------------------------------ ----------- -------- optimizer_dynamic_sampling integer 2 optimizer_features_enable string 10.2.0.4 optimizer_index_caching integer 0 optimizer_index_cost_adj integer 100 optimizer_mode string ALL_ROWS optimizer_secure_view_merging boolean TRUE CREATE TABLE T10 AS SELECT ROWNUM COL1, DECODE(MOD(ROWNUM,1000),1,1,2,2,3,3,DECODE(MOD(ROWNUM,25),10,10,11,11,25)) COL2, LPAD('A',255,'A') COL3 FROM (SELECT ROWNUM RN FROM DUAL CONNECT BY LEVEL<=10000) V1, (SELECT ROWNUM RN FROM DUAL CONNECT BY LEVEL<=1000) V2; CREATE INDEX IND_T10_1 ON T10(COL1); CREATE INDEX IND_T10_2 ON T10(COL2); EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T10',CASCADE=>TRUE,METHOD_OPT=>'FOR COLUMNS SIZE 254 COL2') SELECT COL2, COUNT(*) NUM, COUNT(*)/10000000*100 PERCENT FROM T10 GROUP BY COL2 ORDER BY COL2; COL2 NUM PERCENT ---- ---------- ---------- 1 10000 .1 2 10000 .1 3 10000 .1 10 400000 4 11 400000 4 25 9170000 91.7 The above created a 10,000,000 row table with 6 distinct values in COL2. 0.1% of the rows have a value of 1 in COL2, and 91.7% of the rows have a value of 25 in COL2. There is an index with a histogram on COL2. Obviously (or not) if we have only COL2=1 in the WHERE clause, we probably would want to use the index on the COL2 column to retrieve rows. Obviously (or not) if we have only COL2=25 in the WHERE clause, we probably would want to use a full table scan to retrieve rows. So, what happens when bind variable peeking takes place when a histogram is present on COL2? Ignore for a moment the elapsed time that is output in the following (note that I flush the buffer cache to force physical reads for consistency – direct I/O is enabled): VARIABLE N1 NUMBER EXEC :N1:=1 ALTER SYSTEM FLUSH SHARED_POOL; ALTER SYSTEM FLUSH BUFFER_CACHE; ALTER SYSTEM FLUSH BUFFER_CACHE; SET TIMING ON SELECT /*+ GATHER_PLAN_STATISTICS */ COL2, COUNT(COL1) C1 FROM T10 WHERE COL2= :N1 GROUP BY COL2; COL2 C1 ---------- ---------- 1 10000 Elapsed: 00:00:42.72 SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); ------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows |A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------------------- | 1 | SORT GROUP BY NOSORT | | 1 | 1 | 1 |00:00:42.29 | 10022 | 10022 | | 2 | TABLE ACCESS BY INDEX ROWID| T10 | 1 | 8856 | 10000 |00:00:39.03 | 10022 | 10022 | |* 3 | INDEX RANGE SCAN | IND_T10_2 | 1 | 8856 | 10000 |00:00:00.06 | 22 | 22 | ------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 3 - access("COL2"=:N1) EXEC :N1:=25 ALTER SYSTEM FLUSH BUFFER_CACHE; ALTER SYSTEM FLUSH BUFFER_CACHE; SELECT /*+ GATHER_PLAN_STATISTICS */ COL2, COUNT(COL1) C1 FROM T10 WHERE COL2= :N1 GROUP BY COL2; COL2 C1 ---------- ---------- 25 9170000 Elapsed: 00:00:32.37 SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); ------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows |A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------------------- | 1 | SORT GROUP BY NOSORT | | 1 | 1 | 1 |00:00:32.35 | 402K| 402K| | 2 | TABLE ACCESS BY INDEX ROWID| T10 | 1 | 8856 | 9170K|00:00:27.57 | 402K| 402K| |* 3 | INDEX RANGE SCAN | IND_T10_2 | 1 | 8856 | 9170K|00:00:09.22 | 17879 | 17879 | ------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 3 - access("COL2"=:N1) EXEC :N1:=25 ALTER SYSTEM FLUSH SHARED_POOL; ALTER SYSTEM FLUSH BUFFER_CACHE; ALTER SYSTEM FLUSH BUFFER_CACHE; SELECT /*+ GATHER_PLAN_STATISTICS */ COL2, COUNT(COL1) C1 FROM T10 WHERE COL2= :N1 GROUP BY COL2; COL2 C1 ---------- ---------- 25 9170000 Elapsed: 00:00:20.76 SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); ------------------------------------------------------------------------------------------------ | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------ | 1 | SORT GROUP BY NOSORT| | 1 | 1 | 1 |00:00:20.57 | 384K| 384K| |* 2 | TABLE ACCESS FULL | T10 | 1 | 9234K| 9170K|00:00:27.54 | 384K| 384K| ------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 2 - filter("COL2"=:N1) EXEC :N1:=1 ALTER SYSTEM FLUSH BUFFER_CACHE; ALTER SYSTEM FLUSH BUFFER_CACHE; SELECT /*+ GATHER_PLAN_STATISTICS */ COL2, COUNT(COL1) C1 FROM T10 WHERE COL2= :N1 GROUP BY COL2; COL2 C1 ---------- ---------- 1 10000 Elapsed: 00:00:20.20 SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST')); ------------------------------------------------------------------------------------------------ | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------ | 1 | SORT GROUP BY NOSORT| | 1 | 1 | 1 |00:00:20.19 | 384K| 384K| |* 2 | TABLE ACCESS FULL | T10 | 1 | 9234K| 10000 |00:00:28.73 | 384K| 384K| ------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 2 - filter("COL2"=:N1) The above shows that the first time the SQL statement is hard parsed, a bind variable value of 1 is set, which causes an index range scan regardless if the query will select 0.1% of the rows or 91.7% of the rows. OK, if we then flush the shared pool and first set the bind variable value to 25, a full table scan is used regardless if we select 91.7% of the rows or 0.1% of the rows. You will note that when the full table scan is used when the bind variable was set to 25 the query completed in 20.76 seconds, and when an index range scan was used with the same bind variable value the query completed in 32.37 seconds. OK so far, now the potentially confusing part. When an index range scan was used for both bind variable values, Oracle counted the 0.1% of the matching rows (10000) in 42.72 seconds, while counting 91.7% of the rows (9,170,000) in just 32.37 seconds. You might be wondering why Oracle is able to return the result of counting 91.7% of the rows by the index range scan faster than it is able to count 0.1% of the rows – I will leave that for your investigation. Now, reviewing the above, what is better?: * Allow the bind variable values submitted during the hard parse to determine the execution plan. * Use a stored outline to lock the execution plan to always use an index range scan. * Use a stored outline to lock the execution plan to always use a full table scan. * Disable bind variable peeking. * Not enough information is available.

OracleAppsDBAKK1

Thursday, July 8, 2021

Histograms and Bind Variables, But Why?

No comments:

Post a Comment

Database Options/Management Packs Usage Reporting for Oracle Databases 11.2 and later (Doc ID 1317265.1)

Search This Blog