CentOS或RHEL7下图形化界面安装Oracle 11gR2 RAC
1、环境规划
以下环境在VMware Workstation 15 Pro 下验证实验
\1. 主机环境规划:
设置项 | 版本 |
---|---|
操作系统 | CentOS Linux release 7.7.1908 (Core) |
GRID 版本 | GI 11.2.0.4 |
数据库版本 | Oracle EE 11.2.0.4 |
db_name | xkdb |
节点数 | 2 Nodes RAC |
内存 | 4G |
SWAP | 4G |
\2. 共享存储规划:
磁盘组 | 冗余方式 | 用途 | 大小 |
---|---|---|---|
OCR | Normal | OCR,Votedisk表决磁盘等 | 1Gx3 |
DATA | External | 数据文件、控制文件等 | 20G |
FRA | External | 存放归档、闪回文件等 | 5G |
\3. IP地址规划:
节点 hostname | xk1 | xk2 |
---|---|---|
public ip | 192.168.10.60 | 192.168.10.61 |
vip | 10.10.10.10 | 10.10.10.11 |
private ip | 192.168.10.62 | 192.168.10.63 |
scan ip | 192.168.10.65 | 192.168.10.65 |
\4. 虚拟机设置:
- 每个虚拟机2块网卡,节点之间网卡名称要一致,例如都为ens33 ens34
- 一块网卡用于业务访问, 一块用于心跳网络
- 用于心跳的网卡设置仅主机模式,另一块NAT
注意:
- 主机名不要使用大写字母
- 主机名不能用_线,不建议用-线
- 主机名长度不要超过8个字符,但是vip、scan ip可以
- Oracle RAC数据库节点间通信网络需要使用单独的交换机
- 实际的存储规划需要根据业务情况确定
2、系统参数配置
这部分是软件安装前的操作系统参数配置,分别在两个节点操作
1.依赖包安装
1 | yum -y install binutils compat-libstdc++-33 gcc gcc-c++ glibc glibc-common glibc-devel ksh libaio libaio-devel libgcc libstdc++ libstdc++-devel make sysstat openssh-clients compat-libcap1 xorg-x11-utils xorg-x11-xauth elfutils unixODBC unixODBC-devel libXp elfutils-libelf elfutils-libelf-devel smartmontools |
2.关闭防火墙
1 2 3 4 | --关闭防火墙: systemctl stop firewalld --禁止开机启动: systemctl disable firewalld.service |
3.关闭SELINUX
参考文档:How to Disable or set SELinux to Permissive mode (Doc ID 457458.1)
1 2 3 4 5 | setenforce 0 vi /etc/selinux/config --更改以下内容为disabled,重启操作系统才生效 SELINUX=disabled |
3.关闭透明大页
rac环境下会导致节点重启及性能问题
查看设置前情况
1 2 3 4 5 | cat /sys/kernel/mm/transparent_hugepage/defrag [always] madvise never cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise never |
1 2 3 4 5 6 7 8 9 10 | vi /etc/rc.d/rc.local if test -f /sys/kernel/mm/transparent_hugepage/enabled;then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag;then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi chmod +x /etc/rc.d/rc.local |
4.创建目录、组和用户
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | groupadd oinstall -g 1001 groupadd dba -g 1031 groupadd oper -g 1032 groupadd asmadmin -g 1020 groupadd asmdba -g 1021 groupadd asmoper -g 1022 useradd -g oinstall -G asmadmin,asmdba,asmoper,dba,oper -u 1100 grid useradd -g oinstall -G dba,oper,asmadmin,asmdba -u 1101 oracle mkdir -p /oracle/app/11.2.0/grid mkdir -p /oracle/app/grid mkdir -p /oracle/app mkdir -p /oracle/app/oracle/product/11.2.0/dbhome_1 chown -R oracle:oinstall /oracle chown -R grid:oinstall /oracle/app/11.2.0/grid chown -R grid:oinstall /oracle/app/grid chown -R oracle:oinstall /oracle/app/oracle chmod 771 /oracle/ chmod 771 /oracle/app --修改grid、oracle用户密码 passwd grid passwd oracle |
5.内核参数配置
内核参数的说明参照【安装】CentOS7.7下图形化安装Oracle11gR2 2.9小节说明
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | --以下参数按照4G物理内存情况下计算 --vi /etc/sysctl.conf # Oracle install config kernel.shmall = 1048576 kernel.shmmax = 2147483648 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 6815744 fs.aio-max-nr = 1048576 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576 ##Huge page vm.nr_hugepages = 1300 --生效命令 sysctl -p |
注意:
- kernel.sem这个参数
四个数字:第1个数字应约大于Oracle进程数,第2个数字建议是第1和第4个数字的乘积。这个参数能够满足大部分使用,但对于连接数较高(比如单节点8000个连接)可以设置为:
10000 1280000 512 1024 - kernel.shmall = physical RAM size / pagesize For most systems, this will be the value 2097152. See Note 301830.1 for more information.前为官方文档说明,但是有些教程是shmmax/pagesize
- kernel.shmmax = 1/2 of physical RAM. This would be the value 2147483648 for a system with 4GB of physical RAM. See Note:567506.1 for more information.
- kernel.shmmax定义了单个共享内存段的最大值,要放下整个数据库SGA内存的大小。要大于sga_mas_size,在后面的安装过程中我们设置sga为1.92G小于此处设置
- limits文件里配置的memlock要大于或等于kernel.shmmax
- vm.nr_hugepages一定要计算正确,错误的设置会导致虚拟机启动失败
6.修改系统profile
1 2 3 4 5 6 7 8 9 10 | --vi /etc/profile,加到unset i上方 if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -u 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi |
7.Limit 配置
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | --vi /etc/security/limits.conf #SETTING for ORACLE oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 4096 oracle hard nofile 65536 oracle soft stack 10240 oracle hard stack 32768 grid soft nproc 2047 grid hard nproc 16384 grid soft nofile 4096 grid hard nofile 65536 grid soft stack 10240 grid hard stack 32768 oracle hard memlock unlimited oracle soft memlock unlimited |
8.配置/etc/hosts
hosts文件是Linux系统上一个负责ip地址与域名快速解析的文件,hosts文件包含了ip地址与主机名之间的映射
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | --vi /etc/hosts # Public 192.168.10.60 xk1 192.168.10.61 xk2 # Virtual 192.168.10.62 xk1-vip 192.168.10.63 xk2-vip # Private 10.10.10.10 xk1-priv 10.10.10.11 xk2-priv # Scan-ip 192.168.10.65 rac-scan |
9.设置环境变量
grid用户增加:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | --vi ~/.bash_profile export ORACLE_BASE=/oracle/app/grid export ORACLE_HOME=/oracle/app/11.2.0/grid export ORACLE_SID=+ASM1 --节点2是 +ASM2 export NLS_LANG=american_america.ZHS16GBK export NLS_DATE_FORMAT="yyyy-mm-dd hh24:mi:ss" export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin export ORACLE_TERM=xterm export LD_LIBRARY_PATH=$ORACLE_HOME/lib export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib export CLASSPATH=$ORACLE_HOME/JRE export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib export THREADS_FLAG=native export TEMP=/tmp export TMPDIR=/tmp umask 022 export TMOUT=0 |
oracle用户增加:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | export ORACLE_BASE=/oracle/app/oracle export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1 export ORACLE_SID=xkdb1 --节点2是 xkdb2 export LANG=en_US.UTF-8 export NLS_LANG=american_america.ZHS16GBK export NLS_DATE_FORMAT="yyyy-mm-dd hh24:mi:ss" export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin:/home/oracle/run export ORACLE_TERM=xterm export LD_LIBRARY_PATH=$ORACLE_HOME/lib export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib export CLASSPATH=$ORACLE_HOME/JRE export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib export THREADS_FLAG=native export TEMP=/tmp export TMPDIR=/tmp export GI_HOME=/oracle/app/11.2.0/grid export PATH=${PATH}:$GI_HOME/bin export ORA_NLS10=$GI_HOME/nls/data umask 022 export TMOUT=0 |
10.共享内存段
1 2 3 4 5 6 | vi /etc/fstab --没有的话加入下一行,size=物理内存大小 none /dev/shm tmpfs defaults,size=4G 0 0 --生效 mount -o remount /dev/shm |
11.关闭numa功能
1 2 3 4 5 6 | vi /etc/default/grub 在GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet numa=off" 这一行加上numa=off --重新编译 grub2-mkconfig -o /etc/grub2.cfg |
- https://www.thegeekdiary.com/how-to-disable-numa-in-centos-rhel-67/
- https://www.cnblogs.com/wjoyxt/p/4804081.html
12.pam配置
1 2 3 | --/etc/pam.d/login echo "session required pam_limits.so" >> /etc/pam.d/login cat /etc/pam.d/login |
13.禁用或删除虚拟网卡virbr0
为防止私有网路原因引起network HB丢失,导致root.sh无法成功执行,需要禁用linux上虚拟网卡,此处操作后在grid安装中就看不到virbr0了
参考:
How to Remove virbr0 and lxcbr0 Interfaces on Oracle Linux (Doc ID 2197674.1)
1 2 3 | /bin/systemctl stop libvirtd.service /bin/systemctl disable libvirtd.service /bin/systemctl status libvirtd.service |
14.配置NOZEROCONF
1 2 3 | vi /etc/sysconfig/network --增加如下条目 NOZEROCONF=yes |
参考:
How To Prevent Bogus Entry 169.254.0.0/255.255.0.0 Automatically Added To Routing Table (Doc ID 1161144.1)
15.配置nsswitch.conf
nsswitch.conf保存了域名的检测顺序。某些情况下,NIS(Network Informaion System)可能会导致SCAN域名解析异常,所以需要编辑如下文件设置顺序
1 2 3 | vi /etc/nsswitch.conf --将hosts行修改如下 hosts: files dns myhostname nis |
16.avahi-daemon
avahi-daemon该守护进程配合缓存用户程序的答复,以帮助减少因答复而产生的网络流量。
根据官方一些文档已经列出一些BUG,建议关闭。
只要上面NOZEROCONF配置为打开,则avahi-daemon就为其服务
1 2 | systemctl stop avahi-daemon.socket avahi-daemon.service systemctl disable avahi-daemon.socket avahi-daemon.service |
17.上传安装包到节点1
p13390677_112040_Linux-x86-64_1of7.zip
p13390677_112040_Linux-x86-64_2of7.zip
p13390677_112040_Linux-x86-64_3of7.zip
前两个是DB安装包,第三个是grid安装包
cd /opt
mkdir soft/
chmod 777 /opt/soft/
chmod 777 p13390677_112040_Linux-x86-64_3of7.zip
unzip p13390677_112040_Linux-x86-64_3of7.zip
3、配置共享存储及UDEV
在VM虚拟机实现共享存储有两种方式
- 使用Openfiler等存储管理软件
- 使用VM虚拟机本身添加共享存储
1.方式一:Openfiler
为模拟存储多路径功能,本篇采用Openfiler方式,请参考前序文章
【多路径】CentOS7.7下使用Openfiler+Multipath+UDEV
2.方式二:使用VM虚拟机本身添加共享存储
这种方式比较方便,平常自己搭建测试环境就用这种就好,UDEV同上
- 关闭rac01、02;
- rac01-【编辑虚拟机设置】-【添加】-【硬盘】-【SCSI】-【创建新的虚拟磁盘】-设置磁盘大小,立即分配所有磁盘空间,将虚拟磁盘存储为单个文件-下一步完成-创建完成-确定;
- 再选中刚才创建的磁盘-【高级】-虚拟设备节点选择SCSI1:0(增加多块盘按顺序选择),勾选独立,永久;
- 如果【永久】选项是灰色的,就需要修改配置文件找到rac01的文件夹-编辑rac01.vmx这个文件,将下方的全部配置加进去:
1 2 3 4 5 6 7 8 9 | scsi1:0.deviceType = "disk"(多个磁盘的就加多条) disk.locking="false" diskLib.dataCacheMaxSize="0" diskLib.dataCacheMaxReadAheadSize="0" diskLib.DataCacheMinReadAheadSize="0" diskLib.dataCachePageSize="4096" diskLib.maxUnsyncedWrites="0" scsi1.sharedBus="virtual" scsi1:0.mode = "independent-persistent"(设置为独立模式,如果永久选项是灰色的话添加) |
- 启用UUID
还是在虚拟机关闭状态下编辑rac01.vmx文件,增加下面配置
1 | disk.EnableUUID = "TRUE"(启用UUID) |
- rac02设置-添加-硬盘-选择“使用现有虚拟磁盘”-下一步-选择rac01下的rac01-0.vmdk-完成-高级-选择SCSI1:0,勾选独立-确定;
- 完成上述操作以后,开启虚拟机,使用fdisk -l命令分别在rac01、02查看,注意查询到磁盘盘符一定要相同;
4、grid安装
为解决grid安装BUG需要打两个补丁
参考:Installation walk-through - Oracle Grid/RAC 11.2.0.4 on Oracle Linux 7 (文档 ID 1951613.1)
将两个补丁上传到/opt/soft路径下
1.p19404309_112040_Linux-x86-64.zip补丁安装
1 2 3 4 5 6 | unzip /opt/soft/p19404309_112040_Linux-x86-64.zip cp b19404309/grid/cvu_prereq.xml /soft/grid/stage/cvu/ --装一个包 cd /opt/soft/grid/rpm rpm -ivh cvuqdisk-1.0.9-1.rpm |
2.grid软件安装
1 2 3 4 | cd /opt/soft/grid --执行命令(根据自己主机IP设置 export DISPLAY=192.168.10.1:0.0) ./runInstaller -jreLoc /etc/alternatives/jre_1.8.0 |
说明:
ASM磁盘组使用的是默认的1M AU大小,对于大型数据库,这会造成较多的内存占用,同时对性能略微有些影响,建议对于新增的用于放置数据文件的ASM磁盘组,适当调大AU大小,比如4M或8M(2的幂值)。根据实际经验,建议设置AU 为4m
第一个脚本先在两个节点分别执行
第二个脚本执行前需要先打补丁,因为在7下执行脚本会报错,需要先打p18370031_112040_Linux-x86-64.zip这个补丁
1 2 3 4 5 6 7 8 9 10 | cd /opt/soft unzip p18370031_112040_Linux-x86-64.zip --节点1: cd $ORACLE_HOME/OPatch ./opatch napply -local /opt/soft/18370031 --节点2: scp /opt/soft/p18370031_112040_Linux-x86-64.zip xk2:/opt/ 解压安装同上,安装完成用opatch lsinventory命令检查两节点是否打上 |
下面继续执行第二个脚本,这里放一下执行第二个脚本的内容
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | [root@xk1 OPatch]# /oracle/app/11.2.0/grid/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /oracle/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /oracle/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory User ignored Prerequisites during installation Installing Trace File Analyzer OLR initialization - successful root wallet root wallet cert root cert export peer wallet profile reader wallet pa wallet peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert peer root cert TP profile reader root cert TP pa root cert TP peer pa cert TP pa peer cert TP profile reader pa cert TP profile reader peer cert TP peer user cert pa user cert Adding Clusterware entries to oracle-ohasd.service CRS-2672: Attempting to start 'ora.mdnsd' on 'xk1' CRS-2676: Start of 'ora.mdnsd' on 'xk1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'xk1' CRS-2676: Start of 'ora.gpnpd' on 'xk1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'xk1' CRS-2672: Attempting to start 'ora.gipcd' on 'xk1' CRS-2676: Start of 'ora.cssdmonitor' on 'xk1' succeeded CRS-2676: Start of 'ora.gipcd' on 'xk1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'xk1' CRS-2672: Attempting to start 'ora.diskmon' on 'xk1' CRS-2676: Start of 'ora.diskmon' on 'xk1' succeeded CRS-2676: Start of 'ora.cssd' on 'xk1' succeeded ASM created and started successfully. Disk Group OCR created successfully. clscfg: -install mode specified Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. CRS-4256: Updating the profile Successful addition of voting disk 8d4334c64c694f49bfc095b9ca9a79df. Successful addition of voting disk 1e4141dd3eab4febbf98bc9a277e4b8f. Successful addition of voting disk 3d6746c493b54f79bf437ae9308aed30. Successfully replaced voting disk group with +OCR. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 8d4334c64c694f49bfc095b9ca9a79df (/dev/asmdisk1) [OCR] 2. ONLINE 1e4141dd3eab4febbf98bc9a277e4b8f (/dev/asmdisk2) [OCR] 3. ONLINE 3d6746c493b54f79bf437ae9308aed30 (/dev/asmdisk3) [OCR] Located 3 voting disk(s). CRS-2672: Attempting to start 'ora.OCR.dg' on 'xk1' CRS-2676: Start of 'ora.OCR.dg' on 'xk1' succeeded Configure Oracle Grid Infrastructure for a Cluster ... succeeded |
点击OK继续
到此grid软件安装完成
5、创建ASM磁盘
最后按照规划创建完成所有磁盘组并退出
将grid用户的ORACLE_HOME写入两个节点root用户的.bash_profile文件中,方便操作
1 2 3 4 5 6 7 | su - root vi .bash_profile --添加到PATH行,修改后如下 PATH=$PATH:/oracle/app/11.2.0/grid/bin:$HOME/bin --使生效 source .bash_profile |
关于ASM的知识单独写一篇,到这里磁盘组创建完毕
6、database软件安装
之前已经将db软件上传到/opt/soft目录下,分别解压缩
1 2 3 4 | --使用oracle用户解压 su - oracle unzip p13390677_112040_Linux-x86-64_1of7.zip unzip p13390677_112040_Linux-x86-64_2of7.zip |
执行安装:
1 2 | cd /opt/soft/database ./runInstaller -jreLoc /etc/alternatives/jre_1.8.0 |
安装到56%报错,这里为CentOS7下安装11gR2的bug,处理方式:
/oracle/app/oracle/product/11.2.0/db_1/sysman/lib/ins_emagent.mk
先备份,再修改,只改节点1
搜索NMECTL这一行 在括号后面加上 -lnnz11 修改保存
继续执行,点击retry
完成
7、dbca建库
1 2 | su - oracle dbca |
SGA+PGA要小于物理内存的80%,推荐范围在40-60%
本环境物理内存4G,留给grid集群1G,余3G用于分配SGA+PGA
3Gx80%=2.4G
SGA=2.4x80%=1920M
PGA=2.4.20%=480M
最后2.4/4=60%,在推荐范围内
8、参考
https://mp.weixin.qq.com/s/olIRhxh0gHECQ9h8_yETJQ
- Requirements for Installing Oracle 11.2.0.4 RDBMS on OL7 or RHEL7 64-bit (x86-64) (Doc ID 1962100.1)
- Installation walk-through - Oracle Grid/RAC 11.2.0.4 on Oracle Linux 7 (Doc ID 1951613.1)
- 【安装】CentOS7.7下图形化安装Oracle11gR2