DBA Hub

📋Steps in this guide1/26

Add Node - Bright DBA

Add Node to 11gR2 Oracle RAC Cluster (11.2.0.3) 0. Environment 1. Pre-installation tasks for GI for a cluster 0.i) Backup OCR i) Install the cvuqdisk Package for Linux ii) Verify New Node (HWOS) iii) Verify Peer (REFNODE) iv) Verify New Node (New Pre-Node) v) Run fixup scripts 2. Cluster Node Addition for GI Home. i) … Continue reading Add Node →

oracle installationintermediate
by OracleDba
13 views
1

Overview

Add Node to 11gR2 Oracle RAC Cluster (11.2.0.3) Let start !!! 0. Environment One node RAC 11.2.0.3 (Not RAC ONE), single node RAC. Earlier it was two node RAC setup, recently i have deleted 2nd node from cluster for testing. Node name: RAC1 OS: RHEL 5 DATABASE: nike, instance: nike1 Node name:
2

Section 2

OS: DATABASE: Task: We are going to add node “RAC2” to our existing cluster. Task: Current status
3

Section 3

1. Pre-installation tasks for GI for a cluster A) Install and Configure the Linux Operating System on the New Node << This is step already done by SA B) Configure Access to the Shared Storage << This is step already done by SA C) Install and Configure ASMLib << This is step already done by SA D) SSH configure << This is step already done by SA 0.i) Backup OCR

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
[root@rac1 ~]#
crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
ora.DATA1.dg
               ONLINE  ONLINE       rac1
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
ora.asm
               ONLINE  ONLINE       rac1                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
ora.net1.network
               ONLINE  ONLINE       rac1
ora.ons
               ONLINE  ONLINE       rac1
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.scan1.vip
      1        ONLINE  ONLINE       rac1
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[root@rac1 ~]#
4

Section 4

From: Node RAC1 As root i) Install the cvuqdisk Package for Linux Note:

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@rac1 ~]#
ocrconfig -manualbackup
rac1     2015/06/19 23:38:03     /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150619_233803.ocr
[root@rac1 ~]#

[root@rac2 oracle]#
rpm -ivh cvuqdisk-1.0.9-1.rpm
Preparing...                ########################################### [100%]
Using default group oinstall to install package
   1:cvuqdisk               ########################################### [100%]
[root@rac2 oracle]#
Note:
Without cvuqdisk package, CVU cannot discover shared disks and you will receive the error message "Package cvuqdisk not installed" when the CVU is run.
Example below:
==============
Checking shared storage accessibility...

WARNING:
rac2:PRVF-7017 : Package cvuqdisk not installed
        rac2
No shared storage found
Shared storage check failed on nodes "rac2"
5

Section 5

ii) Verify New Node (HWOS) iii) Verify Peer (REFNODE) From active node RAC1 As GI Home owner

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
As GI Home owner
From active node RAC1.
[oracle@rac1 ~]$
cluvfy stage -post hwos -n rac2
Performing post-checks for hardware and operating system setup

Checking node reachability...
Node reachability check passed from node "rac1"


Checking user equivalence...
User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Node connectivity passed for subnet "192.168.2.0" with node(s) rac2
TCP connectivity check passed for subnet "192.168.2.0"

Node connectivity passed for subnet "192.168.0.0" with node(s) rac2
TCP connectivity check passed for subnet "192.168.0.0"

Node connectivity passed for subnet "10.0.4.0" with node(s) rac2
TCP connectivity check passed for subnet "10.0.4.0"


Interfaces found on subnet "10.0.4.0" that are likely candidates for VIP are:
rac2 eth2:10.0.4.15

Interfaces found on subnet "192.168.2.0" that are likely candidates for a private interconnect are:
rac2 eth0:192.168.2.102

Interfaces found on subnet "192.168.0.0" that are likely candidates for a private interconnect are:
rac2 eth1:192.168.0.102

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "10.0.4.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "10.0.4.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.
Check for multiple users with UID value 0 passed
Time zone consistency check passed

Checking shared storage accessibility...

  Disk                                  Sharing Nodes (1 in count)
  ------------------------------------  ------------------------
  /dev/sda                              rac2

  Disk                                  Sharing Nodes (1 in count)
  ------------------------------------  ------------------------
  /dev/sdb                              rac2


Shared storage check was successful on nodes "rac2"
Post-check for hardware and operating system setup was successful.
[oracle@rac1 ~]$
6

Section 6

Verification of peer compatibility was unsuccessful. Checks did not pass for the following node(s): rac2 iv) Verify New Node (New Pre-Node) As GI Home owner From Node RAC1

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
[oracle@rac1 ~]$
cluvfy comp peer -refnode rac1 -n rac2 -orainv oinstall -osdba dba -verbose
Verifying peer compatibility

Checking peer compatibility...

Compatibility check: Physical memory [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          3.8633GB (4050940.0KB)    3.8633GB (4050940.0KB)    matched
Physical memory check passed
Compatibility check: Available memory [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          3.6215GB (3797424.0KB)    2.6165GB (2743628.0KB)    mismatched
Available memory check failed
Compatibility check: Swap space [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          1.9994GB (2096472.0KB)    5.5385GB (5807488.0KB)    mismatched
Swap space check failed
Compatibility check: Free disk space for "/u01/app/11.2.0/grid" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          17.165GB (1.7998848E7KB)  21.2725GB (2.2305792E7KB)  mismatched
Free disk space check failed
Compatibility check: Free disk space for "/tmp" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          3.707GB (3887104.0KB)     4.4717GB (4688896.0KB)    mismatched
Free disk space check failed

Compatibility check: User existence for "oracle" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          oracle(1100)              oracle(1100)              matched
User existence for "oracle" check passed

Compatibility check: Group existence for "oinstall" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          oinstall(1000)            oinstall(1000)            matched
Group existence for "oinstall" check passed

Compatibility check: Group existence for "dba" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          dba(1200)                 dba(1200)                 matched
Group existence for "dba" check passed

Compatibility check: Group membership for "oracle" in "oinstall (Primary)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          yes                       yes                       matched
Group membership for "oracle" in "oinstall (Primary)" check passed

Compatibility check: Group membership for "oracle" in "dba" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          yes                       yes                       matched
Group membership for "oracle" in "dba" check passed

Compatibility check: Run level [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          5                         5                         matched
Run level check passed

Compatibility check: System architecture [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          x86_64                    x86_64                    matched
System architecture check passed

Compatibility check: Kernel version [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          2.6.32-200.13.1.el5uek    2.6.32-200.13.1.el5uek    matched
Kernel version check passed

Compatibility check: Kernel param "semmsl" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          250                       250                       matched
Kernel param "semmsl" check passed

Compatibility check: Kernel param "semmns" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          32000                     32000                     matched
Kernel param "semmns" check passed

Compatibility check: Kernel param "semopm" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          100                       100                       matched
Kernel param "semopm" check passed

Compatibility check: Kernel param "semmni" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          128                       128                       matched
Kernel param "semmni" check passed
Compatibility check: Kernel param "shmmax" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          2074081280                1054504960                mismatched
Kernel param "shmmax" check failed
Compatibility check: Kernel param "shmmni" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          4096                      4096                      matched
Kernel param "shmmni" check passed

Compatibility check: Kernel param "shmall" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          2097152                   2097152                   matched
Kernel param "shmall" check passed

Compatibility check: Kernel param "file-max" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          6815744                   6815744                   matched
Kernel param "file-max" check passed

Compatibility check: Kernel param "ip_local_port_range" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          between 9000.0 & 65500.0  between 9000.0 & 65500.0  matched
Kernel param "ip_local_port_range" check passed

Compatibility check: Kernel param "rmem_default" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          262144                    262144                    matched
Kernel param "rmem_default" check passed

Compatibility check: Kernel param "rmem_max" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          4194304                   4194304                   matched
Kernel param "rmem_max" check passed

Compatibility check: Kernel param "wmem_default" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          262144                    262144                    matched
Kernel param "wmem_default" check passed

Compatibility check: Kernel param "wmem_max" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          1048586                   1048586                   matched
Kernel param "wmem_max" check passed

Compatibility check: Kernel param "aio-max-nr" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          1048576                   1048576                   matched
Kernel param "aio-max-nr" check passed

Compatibility check: Package existence for "make" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          make-3.81-3.el5           make-3.81-3.el5           matched
Package existence for "make" check passed

Compatibility check: Package existence for "binutils" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6-14.el5  matched
Package existence for "binutils" check passed

Compatibility check: Package existence for "gcc (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc-4.1.2-51.el5 (x86_64)  gcc-4.1.2-51.el5 (x86_64)  matched
Package existence for "gcc (x86_64)" check passed

Compatibility check: Package existence for "libaio (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio-0.3.106-5 (x86_64),libaio-0.3.106-5 (i386)  libaio-0.3.106-5 (x86_64),libaio-0.3.106-5 (i386)  matched
Package existence for "libaio (x86_64)" check passed

Compatibility check: Package existence for "glibc (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-2.5-65 (x86_64),glibc-2.5-65 (i686)  glibc-2.5-65 (x86_64),glibc-2.5-65 (i686)  matched
Package existence for "glibc (x86_64)" check passed

Compatibility check: Package existence for "compat-libstdc++-33 (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          compat-libstdc++-33-3.2.3-61 (x86_64),compat-libstdc++-33-3.2.3-61 (i386)  compat-libstdc++-33-3.2.3-61 (x86_64),compat-libstdc++-33-3.2.3-61 (i386)  matched
Package existence for "compat-libstdc++-33 (x86_64)" check passed

Compatibility check: Package existence for "elfutils-libelf (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          elfutils-libelf-0.137-3.el5 (x86_64),elfutils-libelf-0.137-3.el5 (i386)  elfutils-libelf-0.137-3.el5 (x86_64),elfutils-libelf-0.137-3.el5 (i386)  matched
Package existence for "elfutils-libelf (x86_64)" check passed

Compatibility check: Package existence for "elfutils-libelf-devel" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.137-3.el5  matched
Package existence for "elfutils-libelf-devel" check passed

Compatibility check: Package existence for "glibc-common" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-common-2.5-65       glibc-common-2.5-65       matched
Package existence for "glibc-common" check passed

Compatibility check: Package existence for "glibc-devel (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-devel-2.5-65 (x86_64),glibc-devel-2.5-65 (i386)  glibc-devel-2.5-65 (x86_64),glibc-devel-2.5-65 (i386)  matched
Package existence for "glibc-devel (x86_64)" check passed

Compatibility check: Package existence for "glibc-headers" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-headers-2.5-65      glibc-headers-2.5-65      matched
Package existence for "glibc-headers" check passed

Compatibility check: Package existence for "gcc-c++ (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc-c++-4.1.2-51.el5 (x86_64)  gcc-c++-4.1.2-51.el5 (x86_64)  matched
Package existence for "gcc-c++ (x86_64)" check passed

Compatibility check: Package existence for "libaio-devel (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio-devel-0.3.106-5 (i386),libaio-devel-0.3.106-5 (x86_64)  libaio-devel-0.3.106-5 (i386),libaio-devel-0.3.106-5 (x86_64)  matched
Package existence for "libaio-devel (x86_64)" check passed

Compatibility check: Package existence for "libgcc (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libgcc-4.1.2-51.el5 (x86_64),libgcc-4.1.2-51.el5 (i386)  libgcc-4.1.2-51.el5 (x86_64),libgcc-4.1.2-51.el5 (i386)  matched
Package existence for "libgcc (x86_64)" check passed

Compatibility check: Package existence for "libstdc++ (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++-4.1.2-51.el5 (x86_64),libstdc++-4.1.2-51.el5 (i386)  libstdc++-4.1.2-51.el5 (x86_64),libstdc++-4.1.2-51.el5 (i386)  matched
Package existence for "libstdc++ (x86_64)" check passed

Compatibility check: Package existence for "libstdc++-devel (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++-devel-4.1.2-51.el5 (x86_64)  libstdc++-devel-4.1.2-51.el5 (x86_64)  matched
Package existence for "libstdc++-devel (x86_64)" check passed

Compatibility check: Package existence for "sysstat" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          sysstat-7.0.2-11.el5      sysstat-7.0.2-11.el5      matched
Package existence for "sysstat" check passed

Compatibility check: Package existence for "ksh" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          ksh-20100202-1.el5_6.6    ksh-20100202-1.el5_6.6    matched
Package existence for "ksh" check passed
Verification of peer compatibility was unsuccessful.
Checks did not pass for the following node(s):
        rac2
[oracle@rac1 ~]$

[oracle@rac1 ~]$
cluvfy stage -pre nodeadd -n rac2 -fixup -verbose
Performing pre-checks for node addition

Checking node reachability...

Check: Node reachability from node "rac1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rac2                                  yes
Result: Node reachability check passed from node "rac1"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac1                                  passed
  rac2                                  passed

Verification of the hosts config file successful


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.101   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.106   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.107   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.103   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.105   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth1   192.168.0.101   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth1   169.254.6.127   169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:D3:B8:9F 1500


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.102   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth1   192.168.0.102   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:4C:B3:01 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1[192.168.2.101]             rac1[192.168.2.106]             yes
  rac1[192.168.2.101]             rac1[192.168.2.107]             yes
  rac1[192.168.2.101]             rac1[192.168.2.103]             yes
  rac1[192.168.2.101]             rac1[192.168.2.105]             yes
  rac1[192.168.2.101]             rac2[192.168.2.102]             yes
  rac1[192.168.2.106]             rac1[192.168.2.107]             yes
  rac1[192.168.2.106]             rac1[192.168.2.103]             yes
  rac1[192.168.2.106]             rac1[192.168.2.105]             yes
  rac1[192.168.2.106]             rac2[192.168.2.102]             yes
  rac1[192.168.2.107]             rac1[192.168.2.103]             yes
  rac1[192.168.2.107]             rac1[192.168.2.105]             yes
  rac1[192.168.2.107]             rac2[192.168.2.102]             yes
  rac1[192.168.2.103]             rac1[192.168.2.105]             yes
  rac1[192.168.2.103]             rac2[192.168.2.102]             yes
  rac1[192.168.2.105]             rac2[192.168.2.102]             yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.2.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.2.101              rac1:192.168.2.106              passed
  rac1:192.168.2.101              rac1:192.168.2.107              passed
  rac1:192.168.2.101              rac1:192.168.2.103              passed
  rac1:192.168.2.101              rac1:192.168.2.105              passed
  rac1:192.168.2.101              rac2:192.168.2.102              passed
Result: TCP connectivity check passed for subnet "192.168.2.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking CRS integrity...

Clusterware version consistency passed
The Oracle Clusterware is healthy on node "rac1"

CRS integrity check passed

Checking shared resources...

Checking CRS home location...
"/u01/app/11.2.0/grid" is shared
Result: Shared resources check for node addition passed


Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac1                                  passed
  rac2                                  passed

Verification of the hosts config file successful


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.101   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.106   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.107   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.103   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.105   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth1   192.168.0.101   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth1   169.254.6.127   169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:D3:B8:9F 1500


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.102   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth1   192.168.0.102   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:4C:B3:01 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1[192.168.2.101]             rac1[192.168.2.106]             yes
  rac1[192.168.2.101]             rac1[192.168.2.107]             yes
  rac1[192.168.2.101]             rac1[192.168.2.103]             yes
  rac1[192.168.2.101]             rac1[192.168.2.105]             yes
  rac1[192.168.2.101]             rac2[192.168.2.102]             yes
  rac1[192.168.2.106]             rac1[192.168.2.107]             yes
  rac1[192.168.2.106]             rac1[192.168.2.103]             yes
  rac1[192.168.2.106]             rac1[192.168.2.105]             yes
  rac1[192.168.2.106]             rac2[192.168.2.102]             yes
  rac1[192.168.2.107]             rac1[192.168.2.103]             yes
  rac1[192.168.2.107]             rac1[192.168.2.105]             yes
  rac1[192.168.2.107]             rac2[192.168.2.102]             yes
  rac1[192.168.2.103]             rac1[192.168.2.105]             yes
  rac1[192.168.2.103]             rac2[192.168.2.102]             yes
  rac1[192.168.2.105]             rac2[192.168.2.102]             yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.2.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.2.101              rac1:192.168.2.106              passed
  rac1:192.168.2.101              rac1:192.168.2.107              passed
  rac1:192.168.2.101              rac1:192.168.2.103              passed
  rac1:192.168.2.101              rac1:192.168.2.105              passed
  rac1:192.168.2.101              rac2:192.168.2.102              passed
Result: TCP connectivity check passed for subnet "192.168.2.0"


Check: Node connectivity for interface "eth1"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1[192.168.0.101]             rac2[192.168.0.102]             yes
Result: Node connectivity passed for interface "eth1"


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.0.101              rac2:192.168.0.102              passed
Result: TCP connectivity check passed for subnet "192.168.0.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          3.8633GB (4050940.0KB)    1.5GB (1572864.0KB)       passed
  rac1          3.8633GB (4050940.0KB)    1.5GB (1572864.0KB)       passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          3.6216GB (3797500.0KB)    50MB (51200.0KB)          passed
  rac1          2.4853GB (2606016.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed
Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          1.9994GB (2096472.0KB)    3.8633GB (4050940.0KB)    failed  <<<<<
  rac1          5.5385GB (5807488.0KB)    3.8633GB (4050940.0KB)    passed
Result: Swap space check failed
Check: Free disk space for "rac2:/u01/app/11.2.0/grid"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /u01/app/11.2.0/grid  rac2          /u01          17.165GB      5.5GB         passed
Result: Free disk space check passed for "rac2:/u01/app/11.2.0/grid"

Check: Free disk space for "rac1:/u01/app/11.2.0/grid"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /u01/app/11.2.0/grid  rac1          /u01          21.2705GB     5.5GB         passed
Result: Free disk space check passed for "rac1:/u01/app/11.2.0/grid"

Check: Free disk space for "rac2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rac2          /             3.707GB       1GB           passed
Result: Free disk space check passed for "rac2:/tmp"

Check: Free disk space for "rac1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rac1          /             4.4717GB      1GB           passed
Result: Free disk space check passed for "rac1:/tmp"

Check: User existence for "oracle"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac2          passed                    exists(1100)
  rac1          passed                    exists(1100)

Checking for multiple users with UID value 1100
Result: Check for multiple users with UID value 1100 passed
Result: User existence check passed for "oracle"

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          5                         3,5                       passed
  rac1          5                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac1              hard          65536         65536         passed
  rac2              hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac1              soft          1024          1024          passed
  rac2              soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac1              hard          16384         16384         passed
  rac2              hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac1              soft          2047          2047          passed
  rac2              soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          x86_64                    x86_64                    passed
  rac1          x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          2.6.32-200.13.1.el5uek    2.6.18                    passed
  rac1          2.6.32-200.13.1.el5uek    2.6.18                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              250           250           250           passed
  rac2              250           250           250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              32000         32000         32000         passed
  rac2              32000         32000         32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              100           100           100           passed
  rac2              100           100           100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              128           128           128           passed
  rac2              128           128           128           passed
Result: Kernel parameter check passed for "semmni"
Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              1054504960    1054504960    2074081280    failed        Current value too low. Configured value too low.  <<< 
  rac2              2074081280    1054504960    2074081280    failed        Configured value too low.  <<<
Result: Kernel parameter check failed for "shmmax"
Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              4096          4096          4096          passed
  rac2              4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              2097152       2097152       2097152       passed
  rac2              2097152       2097152       2097152       passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              6815744       6815744       6815744       passed
  rac2              6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
  rac2              between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              262144        262144        262144        passed
  rac2              262144        262144        262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              4194304       4194304       4194304       passed
  rac2              4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              262144        262144        262144        passed
  rac2              262144        262144        262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              1048586       1048586       1048576       passed
  rac2              1048586       1048586       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              1048576       1048576       1048576       passed
  rac2              1048576       1048576       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          make-3.81-3.el5           make-3.81                 passed
  rac1          make-3.81-3.el5           make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6      passed
  rac1          binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "gcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc(x86_64)-4.1.2-51.el5  gcc(x86_64)-4.1.2         passed
  rac1          gcc(x86_64)-4.1.2-51.el5  gcc(x86_64)-4.1.2         passed
Result: Package existence check passed for "gcc(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio(x86_64)-0.3.106-5  libaio(x86_64)-0.3.106    passed
  rac1          libaio(x86_64)-0.3.106-5  libaio(x86_64)-0.3.106    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc(x86_64)-2.5-65      glibc(x86_64)-2.5-24      passed
  rac1          glibc(x86_64)-2.5-65      glibc(x86_64)-2.5-24      passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          compat-libstdc++-33(x86_64)-3.2.3-61  compat-libstdc++-33(x86_64)-3.2.3  passed
  rac1          compat-libstdc++-33(x86_64)-3.2.3-61  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "elfutils-libelf(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          elfutils-libelf(x86_64)-0.137-3.el5  elfutils-libelf(x86_64)-0.125  passed
  rac1          elfutils-libelf(x86_64)-0.137-3.el5  elfutils-libelf(x86_64)-0.125  passed
Result: Package existence check passed for "elfutils-libelf(x86_64)"

Check: Package existence for "elfutils-libelf-devel"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125  passed
  rac1          elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125  passed
Result: Package existence check passed for "elfutils-libelf-devel"

Check: Package existence for "glibc-common"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-common-2.5-65       glibc-common-2.5          passed
  rac1          glibc-common-2.5-65       glibc-common-2.5          passed
Result: Package existence check passed for "glibc-common"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-devel(x86_64)-2.5-65  glibc-devel(x86_64)-2.5   passed
  rac1          glibc-devel(x86_64)-2.5-65  glibc-devel(x86_64)-2.5   passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "glibc-headers"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-headers-2.5-65      glibc-headers-2.5         passed
  rac1          glibc-headers-2.5-65      glibc-headers-2.5         passed
Result: Package existence check passed for "glibc-headers"

Check: Package existence for "gcc-c++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc-c++(x86_64)-4.1.2-51.el5  gcc-c++(x86_64)-4.1.2     passed
  rac1          gcc-c++(x86_64)-4.1.2-51.el5  gcc-c++(x86_64)-4.1.2     passed
Result: Package existence check passed for "gcc-c++(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio-devel(x86_64)-0.3.106-5  libaio-devel(x86_64)-0.3.106  passed
  rac1          libaio-devel(x86_64)-0.3.106-5  libaio-devel(x86_64)-0.3.106  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libgcc(x86_64)-4.1.2-51.el5  libgcc(x86_64)-4.1.2      passed
  rac1          libgcc(x86_64)-4.1.2-51.el5  libgcc(x86_64)-4.1.2      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++(x86_64)-4.1.2-51.el5  libstdc++(x86_64)-4.1.2   passed
  rac1          libstdc++(x86_64)-4.1.2-51.el5  libstdc++(x86_64)-4.1.2   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++-devel(x86_64)-4.1.2-51.el5  libstdc++-devel(x86_64)-4.1.2  passed
  rac1          libstdc++-devel(x86_64)-4.1.2-51.el5  libstdc++-devel(x86_64)-4.1.2  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          sysstat-7.0.2-11.el5      sysstat-7.0.2             passed
  rac1          sysstat-7.0.2-11.el5      sysstat-7.0.2             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          ksh-20100202-1.el5_6.6    ksh-20060214              passed
  rac1          ksh-20100202-1.el5_6.6    ksh-20060214              passed
Result: Package existence check passed for "ksh"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
  rac1                                  passed

Check for consistency of root user's primary group passed

Checking OCR integrity...

OCR integrity check passed

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed
Check: Time zone consistency
Result: Time zone consistency check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed


Checking to make sure user "oracle" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac2          passed                    does not exist
  rac1          passed                    does not exist
Result: User "oracle" is not part of "root" group. Check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "rajasekhar.com" as found on node "rac1"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rac1                                  failed
  rac2                                  failed
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: rac1,rac2

File "/etc/resolv.conf" is not consistent across nodes
Fixup information has been generated for following node(s):
rac2,rac1
Please run the following script on each node as "root" user to execute the fixups:
'/tmp/CVU_11.2.0.3.0_oracle/runfixup.sh'

Pre-check for node addition was unsuccessful on all the nodes.
[oracle@rac1 ~]$
7

Section 7

v) Run fixup scripts On RAC 1, as root user On RAC2, as root user

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[root@rac1 ~]#
/tmp/CVU_11.2.0.3.0_oracle/runfixup.sh
Response file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.response
Enable file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.enable
Log file location: /tmp/CVU_11.2.0.3.0_oracle/orarun.log
Setting Kernel Parameters...
kernel.shmmax = 68719476736
kernel.shmmax = 1054504960
/tmp/CVU_11.2.0.3.0_oracle/orarun.sh: line 230: [: 68719476736kernel.shmmax: integer expression expected
The value for shmmax in response file is not greater than value for shmmax in /etc/sysctl.conf file. Hence not changing it.
kernel.shmmax = 2074081280
[root@rac1 ~]#

[root@rac2 ~]#
/tmp/CVU_11.2.0.3.0_oracle/runfixup.sh
Response file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.response
Enable file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.enable
Log file location: /tmp/CVU_11.2.0.3.0_oracle/orarun.log
Setting Kernel Parameters...
kernel.shmmax = 68719476736
kernel.shmmax = 1054504960
/tmp/CVU_11.2.0.3.0_oracle/orarun.sh: line 230: [: 68719476736kernel.shmmax: integer expression expected
The value for shmmax in response file is not greater than value for shmmax in /etc/sysctl.conf file. Hence not changing it.
The value for shmmax in response file is not greater than value of shmmax for current session. Hence not changing it.
[root@rac2 ~]#
8

Section 8

2. Cluster Node Addition for GI Home. Note: Pre-node check failed. Hence need to set the environment variable IGNORE_PREADDNODE_CHECKS=Y before running addNode.sh otherwise, the silent node addition will fail without showing any errors to the console. Note: i) Run addnode.sh script
9

Section 9

As GI Home owner From active node RAC1 ./addNode.sh -silent "CLUSTER_NEW_NODES={rac2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac2-vip}" The Cluster Node Addition of /u01/app/11.2.0/grid was successful. Note: If you are using GNS: cd $GI_HOME/oui/bin ./addNode.sh -silent “CLUSTER_NEW_NODES={rac2}” Note:

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
[oracle@rac1 ~]$ cd /u01/app/11.2.0/grid/oui/bin/
[oracle@rac1 bin]$
IGNORE_PREADDNODE_CHECKS=Y
[oracle@rac1 bin]$
export IGNORE_PREADDNODE_CHECKS
[oracle@rac1 bin]$
./addNode.sh -silent "CLUSTER_NEW_NODES={rac2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac2-vip}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5671 MB    Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.


Performing tests to see whether nodes rac2 are available
............................................................... 100% Done.

.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/app/11.2.0/grid
   New Nodes
Space Requirements
   New Nodes
      rac2
         /u01: Required 6.99GB : Available 15.98GB
Installed Products
   Product Names
      Oracle Grid Infrastructure 11.2.0.3.0
      Sun JDK 1.5.0.30.03
      Installer SDK Component 11.2.0.3.0
      Oracle One-Off Patch Installer 11.2.0.1.7
      Oracle Universal Installer 11.2.0.3.0
      Oracle USM Deconfiguration 11.2.0.3.0
      Oracle Configuration Manager Deconfiguration 10.3.1.0.0
      Enterprise Manager Common Core Files 10.2.0.4.4
      Oracle DBCA Deconfiguration 11.2.0.3.0
      Oracle RAC Deconfiguration 11.2.0.3.0
      Oracle Quality of Service Management (Server) 11.2.0.3.0
      Installation Plugin Files 11.2.0.3.0
      Universal Storage Manager Files 11.2.0.3.0
      Oracle Text Required Support Files 11.2.0.3.0
      Automatic Storage Management Assistant 11.2.0.3.0
      Oracle Database 11g Multimedia Files 11.2.0.3.0
      Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
      Oracle Core Required Support Files 11.2.0.3.0
      Bali Share 1.1.18.0.0
      Oracle Database Deconfiguration 11.2.0.3.0
      Oracle Quality of Service Management (Client) 11.2.0.3.0
      Expat libraries 2.0.1.0.1
      Oracle Containers for Java 11.2.0.3.0
      Perl Modules 5.10.0.0.1
      Secure Socket Layer 11.2.0.3.0
      Oracle JDBC/OCI Instant Client 11.2.0.3.0
      Oracle Multimedia Client Option 11.2.0.3.0
      LDAP Required Support Files 11.2.0.3.0
      Character Set Migration Utility 11.2.0.3.0
      Perl Interpreter 5.10.0.0.2
      PL/SQL Embedded Gateway 11.2.0.3.0
      OLAP SQL Scripts 11.2.0.3.0
      Database SQL Scripts 11.2.0.3.0
      Oracle Extended Windowing Toolkit 3.4.47.0.0
      SSL Required Support Files for InstantClient 11.2.0.3.0
      SQL*Plus Files for Instant Client 11.2.0.3.0
      Oracle Net Required Support Files 11.2.0.3.0
      Oracle Database User Interface 2.2.13.0.0
      RDBMS Required Support Files for Instant Client 11.2.0.3.0
      RDBMS Required Support Files Runtime 11.2.0.3.0
      XML Parser for Java 11.2.0.3.0
      Oracle Security Developer Tools 11.2.0.3.0
      Oracle Wallet Manager 11.2.0.3.0
      Enterprise Manager plugin Common Files 11.2.0.3.0
      Platform Required Support Files 11.2.0.3.0
      Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
      RDBMS Required Support Files 11.2.0.3.0
      Oracle Ice Browser 5.2.3.6.0
      Oracle Help For Java 4.2.9.0.0
      Enterprise Manager Common Files 10.2.0.4.3
      Deinstallation Tool 11.2.0.3.0
      Oracle Java Client 11.2.0.3.0
      Cluster Verification Utility Files 11.2.0.3.0
      Oracle Notification Service (eONS) 11.2.0.3.0
      Oracle LDAP administration 11.2.0.3.0
      Cluster Verification Utility Common Files 11.2.0.3.0
      Oracle Clusterware RDBMS Files 11.2.0.3.0
      Oracle Locale Builder 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Buildtools Common Files 11.2.0.3.0
      Oracle RAC Required Support Files-HAS 11.2.0.3.0
      SQL*Plus Required Support Files 11.2.0.3.0
      XDK Required Support Files 11.2.0.3.0
      Agent Required Support Files 10.2.0.4.3
      Parser Generator Required Support Files 11.2.0.3.0
      Precompiler Required Support Files 11.2.0.3.0
      Installation Common Files 11.2.0.3.0
      Required Support Files 11.2.0.3.0
      Oracle JDBC/THIN Interfaces 11.2.0.3.0
      Oracle Multimedia Locator 11.2.0.3.0
      Oracle Multimedia 11.2.0.3.0
      HAS Common Files 11.2.0.3.0
      Assistant Common Files 11.2.0.3.0
      PL/SQL 11.2.0.3.0
      HAS Files for DB 11.2.0.3.0
      Oracle Recovery Manager 11.2.0.3.0
      Oracle Database Utilities 11.2.0.3.0
      Oracle Notification Service 11.2.0.3.0
      SQL*Plus 11.2.0.3.0
      Oracle Netca Client 11.2.0.3.0
      Oracle Net 11.2.0.3.0
      Oracle JVM 11.2.0.3.0
      Oracle Internet Directory Client 11.2.0.3.0
      Oracle Net Listener 11.2.0.3.0
      Cluster Ready Services Files 11.2.0.3.0
      Oracle Database 11g 11.2.0.3.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Sunday, June 21, 2015 12:35:19 PM IST)
.                                                                 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Sunday, June 21, 2015 12:35:22 PM IST)
...............................................................................................                                 96% Done.
Home copied to new nodes

Saving inventory on nodes (Sunday, June 21, 2015 12:45:29 PM IST)
.                                                               100% Done.
Save inventory complete
WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.
To register the new inventory please run the script at '/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes 'rac2'.
If you do not register the inventory, you may not be able to update or patch the products you installed.
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/oraInventory/orainstRoot.sh #On nodes rac2
/u01/app/11.2.0/grid/root.sh #On nodes rac2
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/app/11.2.0/grid was successful.
Please check '/tmp/silentInstall.log' for more details.
[oracle@rac1 bin]$
10

Section 10

ii) Run orainstRoot.sh #On nodes rac2 On node RAC2 As root /u01/app/oraInventory/orainstRoot.sh The execution of the script is complete.

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
[root@rac2 ~]#
/u01/app/oraInventory/orainstRoot.sh
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac2 ~]#
11

Section 11

iii) Run root.sh #On nodes rac2 On node RAC2 As root /u01/app/11.2.0/grid/root.sh Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@rac2 ~]#
/u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac2 ~]#
12

Section 12

Note: The root.sh script will configuring Grid Infrastructure on the new node and includes adding High Availability Services to the /etc/inittab so that CRS starts up when the machine starts. Note: iv) Check Clusterware Resources after ran root.sh crsctl stat res -t

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
[root@rac2 ~]#
crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.DATA1.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.asm
               ONLINE  ONLINE       rac1                     Started
               ONLINE  ONLINE       rac2                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
               OFFLINE OFFLINE      rac2
ora.net1.network
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.ons
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.rac2.vip
      1        ONLINE  ONLINE       rac2
ora.scan1.vip
      1        ONLINE  ONLINE       rac2
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[root@rac2 ~]#

[root@rac2 ~]#
crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@rac2 ~]#
13

Section 13

crsctl check crs v) Run cluvfy post-node add script As GI Home owner From node RAC1 as best practice, because initially we ran pre-node cluvfy from RAC 1 onl y cluvfy stage -post nodeadd -n rac2 -verbose

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
[oracle@rac1 ~]$
cluvfy stage -post nodeadd -n rac2 -verbose
Performing post-checks for node addition

Checking node reachability...

Check: Node reachability from node "rac1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rac2                                  yes
Result: Node reachability check passed from node "rac1"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
  rac1                                  passed

Verification of the hosts config file successful


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.102   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth0   192.168.2.107   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth0   192.168.2.104   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth1   192.168.0.102   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth1   169.254.215.111 169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:4C:B3:01 1500


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.101   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.106   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.103   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.105   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth1   192.168.0.101   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth1   169.254.6.127   169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:D3:B8:9F 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac2[192.168.2.102]             rac2[192.168.2.107]             yes
  rac2[192.168.2.102]             rac2[192.168.2.104]             yes
  rac2[192.168.2.102]             rac1[192.168.2.101]             yes
  rac2[192.168.2.102]             rac1[192.168.2.106]             yes
  rac2[192.168.2.102]             rac1[192.168.2.103]             yes
  rac2[192.168.2.102]             rac1[192.168.2.105]             yes
  rac2[192.168.2.107]             rac2[192.168.2.104]             yes
  rac2[192.168.2.107]             rac1[192.168.2.101]             yes
  rac2[192.168.2.107]             rac1[192.168.2.106]             yes
  rac2[192.168.2.107]             rac1[192.168.2.103]             yes
  rac2[192.168.2.107]             rac1[192.168.2.105]             yes
  rac2[192.168.2.104]             rac1[192.168.2.101]             yes
  rac2[192.168.2.104]             rac1[192.168.2.106]             yes
  rac2[192.168.2.104]             rac1[192.168.2.103]             yes
  rac2[192.168.2.104]             rac1[192.168.2.105]             yes
  rac1[192.168.2.101]             rac1[192.168.2.106]             yes
  rac1[192.168.2.101]             rac1[192.168.2.103]             yes
  rac1[192.168.2.101]             rac1[192.168.2.105]             yes
  rac1[192.168.2.106]             rac1[192.168.2.103]             yes
  rac1[192.168.2.106]             rac1[192.168.2.105]             yes
  rac1[192.168.2.103]             rac1[192.168.2.105]             yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.2.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.2.101              rac2:192.168.2.102              passed
  rac1:192.168.2.101              rac2:192.168.2.107              passed
  rac1:192.168.2.101              rac2:192.168.2.104              passed
  rac1:192.168.2.101              rac1:192.168.2.106              passed
  rac1:192.168.2.101              rac1:192.168.2.103              passed
  rac1:192.168.2.101              rac1:192.168.2.105              passed
Result: TCP connectivity check passed for subnet "192.168.2.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking cluster integrity...

  Node Name
  ------------------------------------
  rac1
  rac2

Cluster integrity check passed


Checking CRS integrity...

Clusterware version consistency passed
The Oracle Clusterware is healthy on node "rac2"
The Oracle Clusterware is healthy on node "rac1"

CRS integrity check passed

Checking shared resources...

Checking CRS home location...
"/u01/app/11.2.0/grid" is not shared
Result: Shared resources check for node addition passed


Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
  rac1                                  passed

Verification of the hosts config file successful


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.102   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth0   192.168.2.107   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth0   192.168.2.104   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth1   192.168.0.102   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth1   169.254.215.111 169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:4C:B3:01 1500


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.101   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.106   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.103   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.105   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth1   192.168.0.101   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth1   169.254.6.127   169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:D3:B8:9F 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac2[192.168.2.102]             rac2[192.168.2.107]             yes
  rac2[192.168.2.102]             rac2[192.168.2.104]             yes
  rac2[192.168.2.102]             rac1[192.168.2.101]             yes
  rac2[192.168.2.102]             rac1[192.168.2.106]             yes
  rac2[192.168.2.102]             rac1[192.168.2.103]             yes
  rac2[192.168.2.102]             rac1[192.168.2.105]             yes
  rac2[192.168.2.107]             rac2[192.168.2.104]             yes
  rac2[192.168.2.107]             rac1[192.168.2.101]             yes
  rac2[192.168.2.107]             rac1[192.168.2.106]             yes
  rac2[192.168.2.107]             rac1[192.168.2.103]             yes
  rac2[192.168.2.107]             rac1[192.168.2.105]             yes
  rac2[192.168.2.104]             rac1[192.168.2.101]             yes
  rac2[192.168.2.104]             rac1[192.168.2.106]             yes
  rac2[192.168.2.104]             rac1[192.168.2.103]             yes
  rac2[192.168.2.104]             rac1[192.168.2.105]             yes
  rac1[192.168.2.101]             rac1[192.168.2.106]             yes
  rac1[192.168.2.101]             rac1[192.168.2.103]             yes
  rac1[192.168.2.101]             rac1[192.168.2.105]             yes
  rac1[192.168.2.106]             rac1[192.168.2.103]             yes
  rac1[192.168.2.106]             rac1[192.168.2.105]             yes
  rac1[192.168.2.103]             rac1[192.168.2.105]             yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.2.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.2.101              rac2:192.168.2.102              passed
  rac1:192.168.2.101              rac2:192.168.2.107              passed
  rac1:192.168.2.101              rac2:192.168.2.104              passed
  rac1:192.168.2.101              rac1:192.168.2.106              passed
  rac1:192.168.2.101              rac1:192.168.2.103              passed
  rac1:192.168.2.101              rac1:192.168.2.105              passed
Result: TCP connectivity check passed for subnet "192.168.2.0"


Check: Node connectivity for interface "eth1"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac2[192.168.0.102]             rac1[192.168.0.101]             yes
Result: Node connectivity passed for interface "eth1"


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.0.101              rac2:192.168.0.102              passed
Result: TCP connectivity check passed for subnet "192.168.0.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking node application existence...

Checking existence of VIP node application (required)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          yes                       yes                       passed
  rac1          yes                       yes                       passed
VIP node application check passed

Checking existence of NETWORK node application (required)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          yes                       yes                       passed
  rac1          yes                       yes                       passed
NETWORK node application check passed

Checking existence of GSD node application (optional)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          no                        no                        exists
  rac1          no                        no                        exists
GSD node application is offline on nodes "rac2,rac1"

Checking existence of ONS node application (optional)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          no                        yes                       passed
  rac1          no                        yes                       passed
ONS node application check passed


Checking Single Client Access Name (SCAN)...
  SCAN Name         Node          Running?      ListenerName  Port          Running?
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac-scan.rajasekhar.com  rac2          true          LISTENER_SCAN1  1521          true
  rac-scan.rajasekhar.com  rac1          true          LISTENER_SCAN2  1521          true
  rac-scan.rajasekhar.com  rac1          true          LISTENER_SCAN3  1521          true

Checking TCP connectivity to SCAN Listeners...
  Node          ListenerName              TCP connectivity?
  ------------  ------------------------  ------------------------
  rac1          LISTENER_SCAN1            yes
  rac1          LISTENER_SCAN2            yes
  rac1          LISTENER_SCAN3            yes
TCP connectivity to SCAN Listeners exists on all cluster nodes

Checking name resolution setup for "rac-scan.rajasekhar.com"...
  SCAN Name     IP Address                Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  rac-scan.rajasekhar.com  192.168.2.107             passed
  rac-scan.rajasekhar.com  192.168.2.105             passed
  rac-scan.rajasekhar.com  192.168.2.106             passed

Verification of SCAN VIP and Listener setup passed

Checking to make sure user "oracle" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac2          passed                    does not exist
Result: User "oracle" is not part of "root" group. Check passed

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
Result: CTSS resource check passed


Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State
  ------------------------------------  ------------------------
  rac2                                  Active
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  rac2          0.0                       passed

Time offset is within the specified limits on the following set of nodes:
"[rac2]"
Result: Check of clock time offsets passed


Oracle Cluster Time Synchronization Services check passed
Post-check for node addition was successful. <<<
[oracle@rac1 ~]$
14

Section 14

Post-check for node addition was successful. <<< vi) Check Cluster Nodes vii) Check TNS Listener

Code/Command (click line numbers to comment):

1
2
3
4
5
[oracle@rac2 ~]$
olsnodes -n
rac1    1
rac2    2
[oracle@rac2 ~]$
15

Section 15

On node RAC2 viii) Check ASM Status

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
[oracle@rac2 ~]$
ps -ef | grep tns
root        13     2  0 Jun20 ?        00:00:00 [netns]
oracle   24168     1  0 12:57 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
oracle   24639     1  0 12:57 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
oracle   28355 28292  0 13:19 pts/1    00:00:00 grep tns
[oracle@rac2 ~]$

[oracle@rac2 ~]$
srvctl status asm -a
ASM is running on rac2,rac1
ASM is enabled.
[oracle@rac2 ~]$
16

Section 16

ix) Check OCR On node RAC2 x) Check Vote disk

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[oracle@rac2 ~]$
ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       4076
         Available space (kbytes) :     258044
         ID                       : 1037097601
         Device/File Name         :      +DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

[oracle@rac2 ~]$
17

Section 17

On node RAC2 3. Cluster Node Addition for RDBMS Home. i) Run addnode.sh script

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
[oracle@rac2 ~]$
crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a785661b81264f8ebfa8538128d4e1fe (/dev/oracleasm/disks/DISK1) [DATA]
 2. ONLINE   7a41922f13254f61bf4cee3f53b9aa74 (/dev/oracleasm/disks/DISK2) [DATA]
 3. ONLINE   492244b5021f4fc7bf7d75b74cfe841a (/dev/oracleasm/disks/DISK3) [DATA]
Located 3 voting disk(s).
[oracle@rac2 ~]$
18

Section 18

From Node 1 RAC1 As RDBMS HOME owner ./addNode.sh -silent "CLUSTER_NEW_NODES={rac2}" /u01/app/oracle/product/11.2.0/db_1/root.sh #On nodes rac2 The Cluster Node Addition of /u01/app/oracle/product/11.2.0/db_1 was successful. ii) root.sh #On nodes rac2 from RDBMS home

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
[oracle@rac1 ~]$
cd /u01/app/oracle/product/11.2.0/db_1/oui/bin/
<<< This is RDBMS home

[oracle@rac1 bin]$
./addNode.sh -silent "CLUSTER_NEW_NODES={rac2}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5670 MB    Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.


Performing tests to see whether nodes rac2 are available
............................................................... 100% Done.

........
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/app/oracle/product/11.2.0/db_1
   New Nodes
Space Requirements
   New Nodes
      rac2
         /u01: Required 5.04GB : Available 10.57GB
Installed Products
   Product Names
      Oracle Database 11g 11.2.0.3.0
      Sun JDK 1.5.0.30.03
      Installer SDK Component 11.2.0.3.0
      Oracle One-Off Patch Installer 11.2.0.1.7
      Oracle Universal Installer 11.2.0.3.0
      Oracle USM Deconfiguration 11.2.0.3.0
      Oracle Configuration Manager Deconfiguration 10.3.1.0.0
      Oracle DBCA Deconfiguration 11.2.0.3.0
      Oracle RAC Deconfiguration 11.2.0.3.0
      Oracle Database Deconfiguration 11.2.0.3.0
      Oracle Configuration Manager Client 10.3.2.1.0
      Oracle Configuration Manager 10.3.5.0.1
      Oracle ODBC Driverfor Instant Client 11.2.0.3.0
      LDAP Required Support Files 11.2.0.3.0
      SSL Required Support Files for InstantClient 11.2.0.3.0
      Bali Share 1.1.18.0.0
      Oracle Extended Windowing Toolkit 3.4.47.0.0
      Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
      Oracle Real Application Testing 11.2.0.3.0
      Oracle Database Vault J2EE Application 11.2.0.3.0
      Oracle Label Security 11.2.0.3.0
      Oracle Data Mining RDBMS Files 11.2.0.3.0
      Oracle OLAP RDBMS Files 11.2.0.3.0
      Oracle OLAP API 11.2.0.3.0
      Platform Required Support Files 11.2.0.3.0
      Oracle Database Vault option 11.2.0.3.0
      Oracle RAC Required Support Files-HAS 11.2.0.3.0
      SQL*Plus Required Support Files 11.2.0.3.0
      Oracle Display Fonts 9.0.2.0.0
      Oracle Ice Browser 5.2.3.6.0
      Oracle JDBC Server Support Package 11.2.0.3.0
      Oracle SQL Developer 11.2.0.3.0
      Oracle Application Express 11.2.0.3.0
      XDK Required Support Files 11.2.0.3.0
      RDBMS Required Support Files for Instant Client 11.2.0.3.0
      SQLJ Runtime 11.2.0.3.0
      Database Workspace Manager 11.2.0.3.0
      RDBMS Required Support Files Runtime 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Exadata Storage Server 11.2.0.1.0
      Provisioning Advisor Framework 10.2.0.4.3
      Enterprise Manager Database Plugin -- Repository Support 11.2.0.3.0
      Enterprise Manager Repository Core Files 10.2.0.4.4
      Enterprise Manager Database Plugin -- Agent Support 11.2.0.3.0
      Enterprise Manager Grid Control Core Files 10.2.0.4.4
      Enterprise Manager Common Core Files 10.2.0.4.4
      Enterprise Manager Agent Core Files 10.2.0.4.4
      RDBMS Required Support Files 11.2.0.3.0
      regexp 2.1.9.0.0
      Agent Required Support Files 10.2.0.4.3
      Oracle 11g Warehouse Builder Required Files 11.2.0.3.0
      Oracle Notification Service (eONS) 11.2.0.3.0
      Oracle Text Required Support Files 11.2.0.3.0
      Parser Generator Required Support Files 11.2.0.3.0
      Oracle Database 11g Multimedia Files 11.2.0.3.0
      Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
      Oracle Multimedia Annotator 11.2.0.3.0
      Oracle JDBC/OCI Instant Client 11.2.0.3.0
      Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
      Precompiler Required Support Files 11.2.0.3.0
      Oracle Core Required Support Files 11.2.0.3.0
      Sample Schema Data 11.2.0.3.0
      Oracle Starter Database 11.2.0.3.0
      Oracle Message Gateway Common Files 11.2.0.3.0
      Oracle XML Query 11.2.0.3.0
      XML Parser for Oracle JVM 11.2.0.3.0
      Oracle Help For Java 4.2.9.0.0
      Installation Plugin Files 11.2.0.3.0
      Enterprise Manager Common Files 10.2.0.4.3
      Expat libraries 2.0.1.0.1
      Deinstallation Tool 11.2.0.3.0
      Oracle Quality of Service Management (Client) 11.2.0.3.0
      Perl Modules 5.10.0.0.1
      JAccelerator (COMPANION) 11.2.0.3.0
      Oracle Containers for Java 11.2.0.3.0
      Perl Interpreter 5.10.0.0.2
      Oracle Net Required Support Files 11.2.0.3.0
      Secure Socket Layer 11.2.0.3.0
      Oracle Universal Connection Pool 11.2.0.3.0
      Oracle JDBC/THIN Interfaces 11.2.0.3.0
      Oracle Multimedia Client Option 11.2.0.3.0
      Oracle Java Client 11.2.0.3.0
      Character Set Migration Utility 11.2.0.3.0
      Oracle Code Editor 1.2.1.0.0I
      PL/SQL Embedded Gateway 11.2.0.3.0
      OLAP SQL Scripts 11.2.0.3.0
      Database SQL Scripts 11.2.0.3.0
      Oracle Locale Builder 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      SQL*Plus Files for Instant Client 11.2.0.3.0
      Required Support Files 11.2.0.3.0
      Oracle Database User Interface 2.2.13.0.0
      Oracle ODBC Driver 11.2.0.3.0
      Oracle Notification Service 11.2.0.3.0
      XML Parser for Java 11.2.0.3.0
      Oracle Security Developer Tools 11.2.0.3.0
      Oracle Wallet Manager 11.2.0.3.0
      Cluster Verification Utility Common Files 11.2.0.3.0
      Oracle Clusterware RDBMS Files 11.2.0.3.0
      Oracle UIX 2.2.24.6.0
      Enterprise Manager plugin Common Files 11.2.0.3.0
      HAS Common Files 11.2.0.3.0
      Precompiler Common Files 11.2.0.3.0
      Installation Common Files 11.2.0.3.0
      Oracle Help for the  Web 2.0.14.0.0
      Oracle LDAP administration 11.2.0.3.0
      Buildtools Common Files 11.2.0.3.0
      Assistant Common Files 11.2.0.3.0
      Oracle Recovery Manager 11.2.0.3.0
      PL/SQL 11.2.0.3.0
      Generic Connectivity Common Files 11.2.0.3.0
      Oracle Database Gateway for ODBC 11.2.0.3.0
      Oracle Programmer 11.2.0.3.0
      Oracle Database Utilities 11.2.0.3.0
      Enterprise Manager Agent 10.2.0.4.3
      SQL*Plus 11.2.0.3.0
      Oracle Netca Client 11.2.0.3.0
      Oracle Multimedia Locator 11.2.0.3.0
      Oracle Call Interface (OCI) 11.2.0.3.0
      Oracle Multimedia 11.2.0.3.0
      Oracle Net 11.2.0.3.0
      Oracle XML Development Kit 11.2.0.3.0
      Database Configuration and Upgrade Assistants 11.2.0.3.0
      Oracle JVM 11.2.0.3.0
      Oracle Advanced Security 11.2.0.3.0
      Oracle Internet Directory Client 11.2.0.3.0
      Oracle Enterprise Manager Console DB 11.2.0.3.0
      HAS Files for DB 11.2.0.3.0
      Oracle Net Listener 11.2.0.3.0
      Oracle Text 11.2.0.3.0
      Oracle Net Services 11.2.0.3.0
      Oracle Database 11g 11.2.0.3.0
      Oracle OLAP 11.2.0.3.0
      Oracle Spatial 11.2.0.3.0
      Oracle Partitioning 11.2.0.3.0
      Enterprise Edition Options 11.2.0.3.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Sunday, June 21, 2015 2:06:51 PM IST)
.                                                                 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Sunday, June 21, 2015 2:08:47 PM IST)
...............................................................................................                                 96% Done.
Home copied to new nodes

Saving inventory on nodes (Sunday, June 21, 2015 8:10:31 PM IST)
.                                                               100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/oracle/product/11.2.0/db_1/root.sh #On nodes rac2
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/app/oracle/product/11.2.0/db_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
[oracle@rac1 bin]$
19

Section 19

ii) root.sh #On nodes rac2 from RDBMS home /u01/app/oracle/product/11.2.0/db_1/root.sh 4. Add Instance to Database through Command-Line or you can add via dbca. i) Pre-task

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[root@rac2 ~]#
/u01/app/oracle/product/11.2.0/db_1/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/11.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@rac2 ~]#
20

Section 20

i) Pre-task On RAC2 As RDBMS Home owner mv initnike1.ora initnike2.ora mv orapwnike1 orapwnike2 cat initnike2.ora

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
[oracle@rac2 ~]$
cd /u01/app/oracle/product/11.2.0/db_1/dbs
[oracle@rac2 dbs]$ ls -ltr
total 28
-rw-r----- 1 oracle oinstall 1536 Jun 21 19:56
orapwnike1 <<
-rw-r----- 1 oracle oinstall  161 Jun 21 19:56 initDBUA5216639.ora
-rw-r----- 1 oracle oinstall   36 Jun 21 19:56
initnike1.ora <<
-rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_DBUA5216639.dat 
-rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_nike1.dat 
-rw-r--r-- 1 oracle oinstall 2851 Jun 21 19:56 init.ora 
-rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwDBUA5216639 [oracle@rac2 dbs]$ 
[oracle@rac2 dbs]$
mv initnike1.ora initnike2.ora
[oracle@rac2 dbs]$
mv orapwnike1 orapwnike2
[oracle@rac2 dbs]$ ls -ltr 
total 28 
-rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwnike2 
-rw-r----- 1 oracle oinstall  161 Jun 21 19:56 initDBUA5216639.ora 
-rw-r----- 1 oracle oinstall   36 Jun 21 19:56 initnike2.ora 
-rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_DBUA5216639.dat 
-rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_nike1.dat 
-rw-r--r-- 1 oracle oinstall 2851 Jun 21 19:56 init.ora 
-rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwDBUA5216639 [oracle@rac2 dbs]$ 
[oracle@rac2 dbs]$
cat initnike2.ora
SPFILE='+DATA1/nike/spfilenike.ora' 
[oracle@rac2 dbs]$ 
[oracle@rac2 dbs]$
echo "nike2:/u01/app/oracle/product/11.2.0/db_1:N" >> /etc/oratab
[oracle@rac2 dbs]$
echo "nike:/u01/app/oracle/product/11.2.0/db_1:N" >> /etc/oratab
cat /etc/oratab
..
#
+ASM2:/u01/app/11.2.0/grid:N            # line added by Agent
nike2:/u01/app/oracle/product/11.2.0/db_1:N
nike:/u01/app/oracle/product/11.2.0/db_1:N
[oracle@rac2 ~]$
mkdir -p /u01/app/oracle/admin/nike/adump
[oracle@rac2 ~]$
mkdir -p /u01/app/oracle/admin/nike/dpdump
[oracle@rac2 ~]$
mkdir -p /u01/app/oracle/admin/nike/hdump
[oracle@rac2 ~]$
mkdir -p /u01/app/oracle/admin/nike/pfile
21

Section 21

echo "nike2:/u01/app/oracle/product/11.2.0/db_1:N" >> /etc/oratab echo "nike:/u01/app/oracle/product/11.2.0/db_1:N" >> /etc/oratab ii) Add redo thread On RAC1, As ORACLE HOME owner

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
SQL> set lines 180
SQL> col MEMBER for a60
SQL>
select b.thread#, a.group#, a.member, b.bytes FROM v$logfile a, v$log b WHERE a.group# = b.group#;
THREAD#     GROUP# MEMBER                                                            BYTES
---------- ---------- ------------------------------------------------------------ ----------
         1          2 +DATA1/nike/onlinelog/group_2.290.847761035                    52428800
         1          1 +DATA1/nike/onlinelog/group_1.289.847761031                    52428800

SQL>

SQL>
alter database add logfile thread 2 group 3 ('+DATA1') size 52428800, group 4 ('+DATA1') size 52428800;
Database altered.

SQL>

SQL> select b.thread#, a.group#, a.member, b.bytes FROM v$logfile a, v$log b WHERE a.group# = b.group#;

   THREAD#     GROUP# MEMBER                                                            BYTES
---------- ---------- ------------------------------------------------------------ ----------
         1          2 +DATA1/nike/onlinelog/group_2.290.847761035                    52428800
         1          1 +DATA1/nike/onlinelog/group_1.289.847761031                    52428800
2          3 +DATA1/nike/onlinelog/group_3.271.883012175                    52428800
         2          4 +DATA1/nike/onlinelog/group_4.270.883012181                    52428800
SQL>

SQL>
alter database enable public thread 2;
Database altered.

SQL>
22

Section 22

select b.thread#, a.group#, a.member, b.bytes FROM v$logfile a, v$log b WHERE a.group# = b.group#; alter database add logfile thread 2 group 3 ('+DATA1') size 52428800, group 4 ('+DATA1') size 52428800; alter database enable public thread 2; iii) Add undo tablespace On node RAC1
23

Section 23

iv) Add instance to OCR From node RAC1 as ORACLE_HOME owner srvctl add instance -d nike -i nike2 -n rac2

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
SQL> set pages 0
SQL> set long 9999999
SQL>
select dbms_metadata.get_ddl('TABLESPACE','UNDOTBS1') from dual;
CREATE UNDO TABLESPACE "UNDOTBS1" DATAFILE
  SIZE 26214400
  AUTOEXTEND ON NEXT 5242880 MAXSIZE 32767M
  BLOCKSIZE 8192
  EXTENT MANAGEMENT LOCAL AUTOALLOCATE
   ALTER DATABASE DATAFILE
  '+DATA1/nike/datafile/undotbs1.357.847760839' RESIZE 41943040


SQL>
create undo tablespace undotbs2 datafile '+DATA1' size 25M autoextend on next 5m maxsize 40M;
Tablespace created.

SQL>

SQL>
alter system set undo_tablespace=undotbs2 scope=spfile sid='nike2';
System altered.

SQL>
alter system set instance_number=2 scope=spfile sid='nike2';
System altered.

SQL>
alter system set thread=2 scope=spfile sid='nike2';
System altered.

SQL>

SQL>
alter system set cluster_database_instances=2 scope=spfile sid='*';
System altered.

SQL>

SQL>
select inst_id,name,value from gv$parameter where name like 'undo_table%';
INST_ID NAME                 VALUE
------- -------------------- ---------------
2 undo_tablespace      UNDOTBS2
1 undo_tablespace      UNDOTBS1

SQL>

[oracle@rac1 ~]$ which srvctl
/u01/app/oracle/product/11.2.0/db_1/bin/srvctl
<<< You should run from RDBMS Home
[oracle@rac1 ~]$
[oracle@rac1 ~]$
srvctl add instance -d nike -i nike2 -n rac2
[oracle@rac1 ~]$
[oracle@rac1 ~]$
srvctl status database -d nike -v
Instance nike1 is running on node rac1 with online services nike_srv. Instance status: Open.
Instance nike2 is
not running
on node rac2 <<<
[oracle@rac1 ~]$ 
[oracle@rac1 ~]$
srvctl start instance -d nike -i nike2
[oracle@rac1 ~]$
[oracle@rac1 ~]$
srvctl status database -d nike -v
Instance nike1 is running on node rac1 with online services nike_srv. Instance status: Open.
Instance nike2
is running on
node rac2. Instance status: Open. <<<
[oracle@rac1 ~]$
[oracle@rac1 ~]$
srvctl config database -d nike
Database unique name: nike
Database name: nike
Oracle home: /u01/app/oracle/product/11.2.0/db_1 <<<
Oracle user: oracle
Spfile: +DATA1/nike/spfilenike.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: nike
Database instances: nike1,nike2  <<<<
Disk Groups: DATA1
Mount point paths:
Services: nike_srv <<
Type: RAC
Database is administrator managed <<<
[oracle@rac1 ~]$ SQL> col host_name format a22
SQL> set lines 180
SQL>
select host_name, inst_id, instance_name, status, to_char(startup_time, 'DD-MON-YYYY HH24:MI:SS') as "START_TIME" from gv$instance order by inst_id;
HOST_NAME                 INST_ID INSTANCE_NAME    STATUS       START_TIME
---------------------- ---------- ---------------- ------------ --------------------
rac1.rajasekhar.com             1 nike1            OPEN         21-JUN-2015 11:38:48
rac2.rajasekhar.com             2 nike2            OPEN         22-JUN-2015 01:25:13

SQL>
24

Section 24

not running srvctl start instance -d nike -i nike2 is running on v) Add New Instance to service via srvctl or you can add via dbca From node RAC1, as ORACLE_HOME owner
25

Section 25

Note: If you want add instance as preferred then follow below Note: Note: Modify tnsnames.ora file if required. vi) Check the cluster stack

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
[oracle@rac1 ~]$ which srvctl
/u01/app/oracle/product/11.2.0/db_1/bin/srvctl
<<< You should run from RDBMS Home
[oracle@rac1 ~]$
[oracle@rac1 ~]$
srvctl add service -d nike -s nike_srv -a nike2 -u
<<<< -a means available instance.
[oracle@rac1 ~]$
[oracle@rac1 ~]$
srvctl config service -d nike -s nike_srv
Service name: nike_srv <<<
Service is enabled
Server pool: nike_nike_srv
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Failover type: NONE
Failover method: NONE
TAF failover retries: 0
TAF failover delay: 0
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Preferred instances: nike1
Available instances:
nike2
<<<<
[oracle@rac1 ~]$

[oracle@rac1 ~]$ which srvctl
/u01/app/oracle/product/11.2.0/db_1/bin/srvctl
<<< You should run from RDBMS Home
[oracle@rac1 ~]$
[oracle@rac1 ~]$
srvctl add service -d nike -s nike_srv -r nike2 -u
[oracle@rac1 ~]$
srvctl config service -d nike -s nike_srv
Service name: nike_srv
Service is enabled
Server pool: nike_nike_srv
Cardinality: 2
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Failover type: NONE
Failover method: NONE
TAF failover retries: 0
TAF failover delay: 0
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Preferred instances: nike2,nike1 <<<
Available instances:
[oracle@rac1 ~]$
srvctl status service -d nike
Service nike_srv is running on instance(s) nike1 <<<<
[oracle@rac1 ~]$
srvctl start service -d nike
[oracle@rac1 ~]$ srvctl status service -d nike -v
Service nike_srv is running on instance(s) nike1,nike2 <<<<
[oracle@rac1 ~]$
26

Section 26

vi) Check the cluster stack Caution: Your use of any information or materials on this website is entirely at your own ris k. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using. Caution: Your use of any information or materials on this website is entirely at your own ris k. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using. Still page under construction !!! 🙂

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
[oracle@rac1 ~]$
crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.DATA1.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.asm
               ONLINE  ONLINE       rac1                     Started
               ONLINE  ONLINE       rac2                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
               OFFLINE OFFLINE      rac2
ora.net1.network
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.ons
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
      2        ONLINE  ONLINE       rac2                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.rac2.vip
      1        ONLINE  ONLINE       rac2
ora.scan1.vip
      1        ONLINE  ONLINE       rac2
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[oracle@rac1 ~]$

Comments (0)

Please to add comments

No comments yet. Be the first to comment!