# crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
检查crs进程：
# ps -ef | grep css
root 6929 1 0 19:56 ? 00:00:00 /bin/sh /etc/init.d/init.cssd fatal
root 6960 6928 0 19:56 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root 6963 6929 0 19:56 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root 7064 6935 0 19:56 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
从上可以看出init.cssd停在startcheck中，并没有运行ocssd.bin daemon。
检查crsd.log及ocssd.log，没有发现有用的信息，而在$ORA_CRS_HOME/log/rac1/client中：
[root@rac1 client]# more css339.log
Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, . All rights reserved.
-06-09 20:00:28.799: [ CSSCLNT][2541220896]clsssInitNative: connect failed, rc 9
[root@rac1 client]# more clsc790.log
Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, . All rights reserved.
-06-09 20:39:27.940: [ COMMCRS][2541220896]clsc_connect: (0×66c3d0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))
-06-09 20:39:27.941: [ COMMCRS][2541220896]clsc_connect: (0×6116e0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth))
-06-09 20:39:27.941: [ default][2541220896]Terminating clsd session
检查系统日志，在客户报故障的时间范围内发现如下信息：
Jun 9 17:19:39 rac1 logger: Cluster Ready Services starting up automatically.
Jun 9 17:19:39 rac1 init.crs: Startup will be queued to init within 90 seconds.
Jun 9 17:19:39 rac1 rc: Starting init.crs: succeeded
Jun 9 17:19:39 rac1 readahead: Starting background readahead:
Jun 9 17:19:39 rac1 rc: Starting readahead: succeeded
Jun 9 17:19:41 rac1 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.4528.
Jun 9 17:19:41 rac1 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.4531.
Jun 9 17:19:41 rac1 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.4603.
转到/tmp去查看crsctl.*文件，却发现没有，奇怪！那这些信息又是在哪里产生呢？于是去查看/etc/init.d/init.cssd脚本，该信息在脚本中的如下地方产生：
# Wait for additional filesystems and objects to become available
# crsctl should print out a message indicating cause of failure.
$SU $ORACLE_USER -c “$CRSCTL check boot > $CRSCTLOUT”
RC=$?
while [ "$RC" != "0" ]
do
$LOGMSG Cluster Ready Services waiting on dependencies. Diagnostics in $CRSCTLOUT.
$SLEEP $DEP_CHECK_WAIT
$SU $ORACLE_USER -c “$CRSCTL check boot > $CRSCTLOUT”
RC=$?
done
该段脚本正好是在startcheck的选项中执行的，再看看变量$CRSCTLOUT的定义：
# Temp file for crsctl output
CRSCTLOUT=/tmp/crsctl.$$
正是/tmp目录。难道是/tmp目录有问题？手工执行：
[root@rac1 /]# /etc/init.d/init.cssd startcheck
-bash: /tmp/crsctl.31969: No such file or directory
果然是/tmp有问题，检查/tmp的权限：
drwxr-xr-x 7 1003 dba 4096 Jun 10 04:02 tmp
晕，权限被改了，正确的应该是：
drwxrwxrwt 15 root root 4096 Jun 10 09:51 tmp
于是修改/tmp权限：
＃chown -R root:root /tmp
#chmod -R 1777 /tmp
然后执行：
[root@rac1 tmp]# /etc/init.d/init.crs stop
Shutting down Cluster Ready Services (CRS):
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
[root@rac1 tmp]# /etc/init.d/init.crs start
Startup will be queued to init within 90 seconds.
过一会儿查看crs进程，crs成功运行，至此问题解决！
客户环境：RHEL AS 4 + 10g RAC
