更多精彩內(nèi)容盡在
www.leonarding.com一 數(shù)據(jù)庫版本
SYS@LEO1>select* from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database
11g Enterprise EditionRelease 11.2.0.1.0 - 64
bit Production
PL/SQLRelease 11.2.0.1.0 - Production
CORE 11.2.0.1.0 Production
TNS forLinux: Version 11.2.0.1.0 - Production
NLSRTLVersion 11.2.0.1.0 – Production
操作系統(tǒng)信息
[
oracle@leonarding1admin]$ uname -a
Linuxleonarding1.oracle.com 2.6.32-200.13.1.el5uek #1 SMP Wed Jul 27 21:02:33 EDT 2011x86_64 x86_64 x86_64 GNU/Linux
二比較使用sql*loader的直接加載方式和傳統(tǒng)加載方式的性能差異,給出演示過程和結(jié)論。
第一 我們先要生成平面數(shù)據(jù)(文本數(shù)據(jù))
LEO1@LEO1>create table leo2 as select *from dba_objects; 創(chuàng)建數(shù)據(jù)源,我們的平面數(shù)據(jù)就是從這個(gè)表中取出
Table created.
第二 我們利用spool工具將屏幕中顯示出來的記錄寫入到指定文件,這樣我們就可以得到一個(gè)平面文件啦
set termout off; 是否在屏幕上顯示輸出內(nèi)容,off屏幕不顯示查詢語句,主要與spool結(jié)合使用
set feedback off; 關(guān)閉本次sql命令處理的記錄條數(shù),默認(rèn)為on即去掉最后的已經(jīng)選擇的行數(shù)
set echo off; 關(guān)閉腳本中正在執(zhí)行的SQL語句的顯示
set heading off; 關(guān)閉標(biāo)題的輸出,設(shè)置為off就去掉了select結(jié)果的字段名只顯示數(shù)據(jù)
set trimout on; 去除標(biāo)準(zhǔn)輸出每行后面多余的空格
set trimspool on; 將每行后面多余的空格去掉【linesize-實(shí)際字符數(shù)=多余空格】
spool /home/oracle/sql_loader/leo3.txt 在屏幕上的所有內(nèi)容都包含在該文件中
select owner||','||object_name||','||object_id||','||object_typefrom leo2;
spool off 只有關(guān)閉spool輸出,才會(huì)在輸出文件中看到輸出的內(nèi)容
備注:在實(shí)用SPOOL輸出內(nèi)容到本地文件時(shí),需注意編碼格式,否則會(huì)出現(xiàn)亂碼的問題
[oracle@leonarding1 sql_loader]$ ll
total 28468
-rw-r--r-- 1 oracle oinstall 3246601 Jun 22 14:06 leo3.txt 已經(jīng)生成平面文件leo3.txt
[oracle@leonarding1 sql_loader]$ cat leo3.txt| wc -l 文件中有72678行記錄
72678
第三 創(chuàng)建裝入的表leo3_loader
LEO1@LEO1>create table leo3_loader
(
owner varchar2(30),
object_name varchar2(130),
object_id number,
object_type varchar2(20)
);
2 3 4 5 6 7
第四 創(chuàng)建sql*loader的控制文件leo3_loader.ctl
[oracle@leonarding1 sql_loader]$ vim leo3_loader.ctl
load data
infile '/home/oracle/sql_loader/leo3.txt' 待加載的數(shù)據(jù)文件
badfile '/home/oracle/sql_loader/leo3_bad.txt' 格式不匹配寫入壞文件
discardfile'/home/oracle/sql_loader/leo3_discard.txt' 條件不匹配寫入丟棄文件
append into table leo3_loader 追加的方式插入數(shù)據(jù)
fields terminated by "," 字段與字段之間的分隔符
trailing nullcols 這句的意思是將沒有對(duì)應(yīng)值的列都置為null
(owner,object_name,object_id,object_type) 數(shù)據(jù)插入的對(duì)應(yīng)字段
第五 執(zhí)行sqlldr直接加載命令
[oracle@leonarding1 sql_loader]$ sqlldrleo1/leo1 control=leo3_loader.ctl log=leo3_loader.log direct=true
SQL*Loader: Release 11.2.0.1.0 - Productionon Sat Jun 22 14:08:31 2013
Copyright (c) 1982, 2009, Oracle and/or itsaffiliates. All rights reserved.
Load completed - logical record count72678.
已經(jīng)加載了72678行,條件不匹配有72行,實(shí)際加載入72606行
LEO1@LEO1>select count(*) fromleo3_loader; 表中也是
72606
我們在看一下sql*loader日志
。。。省略前部份。。。
Total logical records skipped: 0
Total logical recordsread: 72678
Total logical records rejected: 0
Total logical records discarded: 72 條件不匹配有72行
Total stream buffers loaded by SQL*Loadermain thread: 17
Total stream buffers loaded by SQL*Loaderload thread: 6
Run began on Sat Jun 22 14:08:31 2013
Run ended on Sat Jun 22 14:08:34 2013
Elapsed time was: 00:00:02.60 所用耗時(shí)2.6秒
CPU time was: 00:00:00.13
使用conventional傳統(tǒng)加載方式寫入數(shù)據(jù)
LEO1@LEO1>truncate table leo3_loader; 清空表在加載一次
[oracle@leonarding1 sql_loader]$ sqlldrleo1/leo1 control=leo3_loader.ctl log=leo3_loader.log
LEO1@LEO1>select count(*) fromleo3_loader; 表中也是
72606
我們在看一下sql*loader日志
。。。省略前部份。。。
Total logical records skipped: 0
Total logical recordsread: 72678
Total logical records rejected: 0
Total logical records discarded: 72 條件不匹配有72行
Run began on Sat Jun 22 15:25:45 2013
Run ended on Sat Jun 22 15:26:05 2013
Elapsed time was: 00:00:20.79 所用耗時(shí)2.6秒
CPU time was: 00:00:00.48
小結(jié):經(jīng)過比對(duì)direct比conventional要提高了20倍效率,為什么direct會(huì)這么高效呢,下面我們來說說這兩種的區(qū)別。
Direct 特點(diǎn)
(1)數(shù)據(jù)繞過SGA直接寫入磁盤的數(shù)據(jù)文件
(2)數(shù)據(jù)直接寫入高水位線HWM之后的新塊,不會(huì)掃描HWM之前的空閑塊
(3)commit之后移動(dòng)HWM他人才能看到
(4)不對(duì)已用空間進(jìn)行掃描
(5)使用direct幾乎不產(chǎn)生redo log,不是完全不產(chǎn)生(安全性差),但會(huì)產(chǎn)生undo數(shù)據(jù)
(6)適用OLAP在線分析場景,增 刪 改不頻繁的場景
Conventional傳統(tǒng)加載特點(diǎn)
(1)數(shù)據(jù)先加載 -> SGA -> 磁盤的數(shù)據(jù)文件
(2)會(huì)掃描高水位線HWM之前的數(shù)據(jù)塊,如果有空閑塊(碎片經(jīng)常DML導(dǎo)致)就會(huì)利用,如果沒有再插入新塊
(3)高水位線HWM之前的數(shù)據(jù)塊是放在SGA區(qū)的
(4)會(huì)產(chǎn)生redo log和undo數(shù)據(jù)
(5)安全性高,可恢復(fù)數(shù)據(jù)
(6)傳統(tǒng)加載與SQL語句insert插入沒區(qū)別
三 比較數(shù)據(jù)泵和exp/imp對(duì)相同數(shù)據(jù)導(dǎo)出/導(dǎo)入的性能差異,給出演示過程
EXP/IMP 是數(shù)據(jù)庫傳統(tǒng)的導(dǎo)出導(dǎo)入工具,它有使用方便,在服務(wù)端和客戶端都可用的優(yōu)點(diǎn),速度沒有expdp快,功能沒有expdp強(qiáng)大
EXP 工具幫助文檔,我們可以參考幫助文檔進(jìn)行命令行配置
[oracle@leonarding1 ~]$ exp –help 或者 exp help=y
Export: Release 11.2.0.1.0 - Production onThu Jun 20 07:28:33 2013
Copyright (c) 1982, 2009, Oracle and/or itsaffiliates. All rights reserved.
You can let Export prompt you forparameters by entering the EXP
command followed by your username/password:
Example: EXP SCOTT/TIGER
Or, you can control how Export runs byentering the EXP command followed
by various arguments. To specifyparameters, you use keywords:
Format: EXP KEYWORD=value orKEYWORD=(value1,value2,...,valueN)
Example: EXP SCOTT/TIGER GRANTS=Y TABLES=(EMP,DEPT,MGR)
or TABLES=(T1:P1,T1:P2), if T1is partitioned table
USERID must be the first parameter on thecommand line.
Keyword Description (Default) Keyword Description (Default)
--------------------------------------------------------------------------
USERID username/password FULL export entire file(N)
BUFFER size of data buffer OWNER list of ownerusernames
FILE output files (EXPDAT.DMP) TABLES list of table names
COMPRESS import into one extent (Y) RECORDLENGTH length of IO record
GRANTS export grants (Y) INCTYPE incremental exporttype
INDEXES export indexes (Y) RECORD track incr. export(Y)
DIRECT direct path (N) TRIGGERS export triggers (Y)
LOG log file of screen output STATISTICS analyze objects(ESTIMATE)
ROWS export data rows (Y) PARFILE parameter filename
CONSISTENT cross-table consistency(N)CONSTRAINTS export constraints (Y)
OBJECT_CONSISTENT transaction set to read only during objectexport (N)
FEEDBACK display progress every x rows (0)
FILESIZE maximum size of each dump file
FLASHBACK_SCN SCN used to set session snapshot backto
FLASHBACK_TIME time used to get the SCN closest to thespecified time
QUERY select clause used to export asubset of a table
RESUMABLE suspend when a space related erroris encountered(N)
RESUMABLE_NAME text string used to identify resumablestatement
RESUMABLE_TIMEOUT wait time for RESUMABLE
TTS_FULL_CHECK perform full or partial dependency checkfor TTS
VOLSIZE number of bytes to write to eachtape volume
TABLESPACES list of tablespaces to export
TRANSPORT_TABLESPACE export transportabletablespace metadata (N)
TEMPLATE template name which invokes iASmode export
Export terminated successfully withoutwarnings.
#########################################################################
創(chuàng)建測試表leo1
LEO1@LEO1>set linesize 400 pagesize 999 格式化
LEO1@LEO1>drop table leo1 purge; 清空環(huán)境
Table dropped.
LEO1@LEO1>create table leo1 as select *from dba_objects; 創(chuàng)建測試表leo1
Table created.
LEO1@LEO1>insert into leo1 select * fromleo1;
72543 rows created.
LEO1@LEO1>insert into leo1 select * fromleo1;
145086 rows created.
LEO1@LEO1>insert into leo1 select * fromleo1;
290172 rows created.
LEO1@LEO1>insert into leo1 select * fromleo1;
580344 rows created.
LEO1@LEO1>commit;
Commit complete.
LEO1@LEO1>insert into leo1 select * fromleo1;
1160688 rows created.
LEO1@LEO1>commit;
Commit complete.
LEO1@LEO1>select count(*) from leo1; 現(xiàn)在是232萬
COUNT(*)
----------
2321376
[oracle@leonarding1 ~]$ mkdir exp_dump 創(chuàng)建一個(gè)導(dǎo)出文件的文件夾
[oracle@leonarding1 ~]$ exp leo1/leo1file='/home/oracle/exp_dump/leo1.dmp' tables=leo1 rows=y
Export: Release11.2.0.1.0 - Production on Fri Jun 21 06:38:28 2013
Copyright (c) 1982, 2009, Oracle and/or itsaffiliates. All rights reserved.
Connected to: Oracle Database 11gEnterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Miningand Real Application Testing options
Export done in US7ASCII character set andAL16UTF16 NCHAR character set
server uses ZHS16GBK character set(possible charset conversion)
About to export specified tables viaConventional Path ...
. . exporting table LEO1 2321376 rows exported
Export terminated successfully withoutwarnings. 成功終止導(dǎo)出,沒有出現(xiàn)告警
[oracle@leonarding1 ~]$ date
Fri Jun 21 06:39:40 CST 2013
導(dǎo)出從06:38:28開始到06:39:40結(jié)束,一共用時(shí)72秒
LEO1@LEO1>truncate table leo1; 清空表,我再導(dǎo)入一下
Table truncated.
[oracle@leonarding1 ~]$ cd exp_dump/
[oracle@leonarding1 exp_dump]$ ll
total 261356
-rw-r--r-- 1 oracle oinstall 267362304 Jun21 06:39 leo1.dmp 這個(gè)就是我們導(dǎo)出的文件
[oracle@leonarding1 exp_dump]$ impleo1/leo1 file='/home/oracle/exp_dump/leo1.dmp' full=y ignore=y
Import: Release11.2.0.1.0 - Production on Fri Jun 21 06:45:04 2013
Copyright (c) 1982, 2009, Oracle and/or itsaffiliates. All rights reserved.
Connected to: Oracle Database 11gEnterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Miningand Real Application Testing options
Export file created by EXPORT:V11.02.00 viaconventional path
import done in US7ASCII character set andAL16UTF16 NCHAR character set
import server uses ZHS16GBK character set(possible charset conversion)
. importing LEO1's objects into LEO1
. importing LEO1's objects into LEO1
. . importing table "LEO1" 2321376 rows imported
Import terminated successfully withoutwarnings. 成功終止導(dǎo)入,沒有出現(xiàn)告警
[oracle@leonarding1 exp_dump]$ date
Fri Jun 21 06:48:26 CST 2013
導(dǎo)入從06:45:04開始到06:48:26結(jié)束,一共用時(shí)202秒
EXPDP/IMPDP 是Oracle推崇的數(shù)據(jù)泵導(dǎo)入導(dǎo)出工具,用于代替?zhèn)鹘y(tǒng)的EXP/IMP,只能在服務(wù)端使用,效率比EXP/IMP快幾十倍,有續(xù)傳功能和并行功能。這個(gè)工具始于Oracle10g,從Oracle11g開始不再提供老的EXP/IMP的咨詢但工具還可以使用。
EXPDP工具的效率差不多比EXP快幾倍
IMPDP工具的效率差不多比IMP快幾十倍
所以說這兩個(gè)工具適用于大數(shù)據(jù)導(dǎo)入導(dǎo)出的場景
導(dǎo)出文件的格式更接近于數(shù)據(jù)庫本身的文件格式,避免了數(shù)據(jù)寫入文件時(shí)的轉(zhuǎn)換
直接路徑加載,跳過SGA內(nèi)存區(qū),直接加載到高水位線之后
元數(shù)據(jù)metadata和數(shù)據(jù)data在導(dǎo)出的過程中可以重疊進(jìn)行,提高導(dǎo)出的效率。
我們再來看看EXPDP/IMPDP的導(dǎo)出導(dǎo)入效果
在使用EXPDP/IMPDP之前我們需要定義一個(gè)目錄對(duì)象,告知EXPDP/IMPDP工具導(dǎo)出的文件和導(dǎo)入的文件的存放目錄
LEO1@LEO1>create directory exp_dump as'/home/oracle/exp_dump';
Directory created.
[oracle@leonarding1 exp_dump]$ expdpleo1/leo1 directory=exp_dump dumpfile=expdp_leo1.dmp tables=leo1;
Export: Release11.2.0.1.0 - Production on Fri Jun 21 07:07:09 2013
Copyright (c) 1982, 2009, Oracle and/or itsaffiliates. All rights reserved.
Connected to: Oracle Database 11gEnterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Miningand Real Application Testing options
Starting"LEO1"."SYS_EXPORT_TABLE_01": leo1/******** directory=exp_dumpdumpfile=expdp_leo1.dmp tables=leo1
Estimate in progress using BLOCKS method...
Processing object typeTABLE_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 264MB
Processing object typeTABLE_EXPORT/TABLE/TABLE
. . exported"LEO1"."LEO1" 223.9 MB 2321376 rows
Master table"LEO1"."SYS_EXPORT_TABLE_01" successfully loaded/unloaded
******************************************************************************
Dump file set for LEO1.SYS_EXPORT_TABLE_01is:
/home/oracle/exp_dump/expdp_leo1.dmp
Job"LEO1"."SYS_EXPORT_TABLE_01" successfully completed at07:08:22
導(dǎo)出從07:07:09開始到07:08:22結(jié)束,一共用時(shí)73秒,導(dǎo)出文件大小223.9MB 行數(shù)2321376 rows,與exp時(shí)間相差1秒,expdp在數(shù)據(jù)量比較大的時(shí)候會(huì)體現(xiàn)出高效率。
現(xiàn)在我們impdp導(dǎo)入
LEO1@LEO1>drop table leo1; 先把表刪除,因?yàn)槿绻斫Y(jié)構(gòu)存在的話,impdp會(huì)認(rèn)為元數(shù)據(jù)已存在報(bào)錯(cuò),不可導(dǎo)入
Table dropped.
[oracle@leonarding1 exp_dump]$ impdpleo1/leo1 directory=exp_dump dumpfile=expdp_leo1.dmp
Import: Release11.2.0.1.0 - Production on Fri Jun 21 07:24:07 2013
Copyright (c) 1982, 2009, Oracle and/or itsaffiliates. All rights reserved.
Connected to: Oracle Database 11gEnterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Miningand Real Application Testing options
Master table"LEO1"."SYS_IMPORT_FULL_01" successfully loaded/unloaded
Starting"LEO1"."SYS_IMPORT_FULL_01": leo1/******** directory=exp_dumpdumpfile=expdp_leo1.dmp
Processing object typeTABLE_EXPORT/TABLE/TABLE
Processing object typeTABLE_EXPORT/TABLE/TABLE_DATA
. . imported"LEO1"."LEO1" 223.9 MB 2321376rows
Job"LEO1"."SYS_IMPORT_FULL_01" successfully completed at07:27:27
導(dǎo)入從07:24:07開始到07:27:27結(jié)束,一共用時(shí)200秒,導(dǎo)入文件大小223.9MB 行數(shù)2321376 rows,比imp時(shí)間節(jié)約了2秒,impdp也是在數(shù)據(jù)量比較大的時(shí)候會(huì)體現(xiàn)出高效率。
四 用外部表的方式查詢當(dāng)天數(shù)據(jù)庫alert日志文件中當(dāng)天所有的ora-錯(cuò)誤信息,給出演示過程。
外部表:表中的數(shù)據(jù)以操作系統(tǒng)文件的方式來存放,現(xiàn)在表中的數(shù)據(jù)不是放在數(shù)據(jù)庫中了而是放在操作系統(tǒng)上面,Oracle提供了一種直接讀取外部數(shù)據(jù)的機(jī)制。
外部表好處:1.數(shù)據(jù)二次開發(fā)
2.大數(shù)據(jù)量遷移
3.充分利用操作系統(tǒng)空間
4.不占用數(shù)據(jù)庫空間
5.支持標(biāo)準(zhǔn)SQL條件檢索
外部表也需要目錄對(duì)象的支持,通過目錄對(duì)象可以知道從哪個(gè)目錄讀取文本數(shù)據(jù)
LEO1@LEO1>create directory alert as'/u02/app/oracle/diag/rdbms/leo1/LEO1/trace';
Directory created.
這是Oracle 11g 告警日志目錄
grant read,write on directory alert topublic; 對(duì)這個(gè)目錄對(duì)象授予讀/寫權(quán)限,并授予所有用戶
LEO1@LEO1>select * from dba_directories;
OWNER DIRECTORY_NAME DIRECTORY_PATH
--------------------------------------------------------------------------------------------------------------
SYS EXP_DUMP /home/oracle/exp_dump
SYS XMLDIR /u02/app/oracle/product/11.2.0/db_1/rdbms/xml
SYS ALERT /u02/app/oracle/diag/rdbms/leo1/LEO1/trace
SYS DATA_PUMP_DIR /u02/app/oracle/admin/LEO1/dpdump/
SYS ORACLE_OCM_CONFIG_DIR /u02/app/oracle/product/11.2.0/db_1/ccr/state
我們下面就是Oracle告警日志文件當(dāng)作數(shù)據(jù)庫的一個(gè)外部數(shù)據(jù)源來訪問,我們使用外部表的方式抽取alert日志數(shù)據(jù),然后使用標(biāo)準(zhǔn)SQL語句來檢索“ora-錯(cuò)誤信息”。
下面我們就來創(chuàng)建一個(gè)外部表
LEO1@LEO1>create table leo_alert(content varchar2(4000)) alert日志數(shù)據(jù)量多因此字符串設(shè)置的大一點(diǎn)
organization external
(
type oracle_loader 如果你設(shè)置的是oracle_datapump請修改為loader
default directory alert
access parameters (
records delimited by newline 每條記錄用換行區(qū)分
nobadfile 沒有壞文件,丟棄文件,日志文件
nodiscardfile
nologfile
)
location ('alert_LEO1.log') 加載告警日志文件內(nèi)容
); 2 3 4 5 6 7 8 9 10 11 12 13
LEO1@LEO1>select count(*) fromleo_alert; 一共7198條
COUNT(*)
----------------
7198
我們抽取其中10條ORA-開頭的錯(cuò)誤記錄顯示出來
LEO1@LEO1>select * from leo_alert wherecontent like '%ORA-%' and rownum<=10;
CONTENT
------------------------------------------------------------------------------------------------------------------------------------------------------------------
ORA-210 signalled during: create tablespacetest datafile '/u02/app/oracle/oradata/LEO1/test01.dbf' size 10m autoextendoff...
ORA-00210: cannot open the specifiedcontrol file
ORA-00202: control file:'/u02/app/oracle/oradata/LEO1/control01.ctl'
ORA-27041: unable to open file
ORA-00210: cannot open the specifiedcontrol file
ORA-00202: control file:'/u02/app/oracle/oradata/LEO1/control01.ctl'
ORA-27041: unable to open file
ORA-00210: cannot open the specifiedcontrol file
ORA-00202: control file:'/u02/app/oracle/oradata/LEO1/control01.ctl'
ORA-27037: unable to obtain file status
10 rows selected.
小結(jié):這里需要注意幾個(gè)問題,我們在創(chuàng)建外部表的時(shí)候需要設(shè)置沒有壞文件,丟棄文件,日志文件參數(shù)否則會(huì)報(bào)錯(cuò)ORA-29913: error in executing ODCIEXTTABLEOPEN callout。
sql*loader exp/imp expdp/impdp organization_external direct
Leonarding2013.6.22
北京&summer
分享技術(shù)~成就夢想
Blog:
www.leonarding.com