DBA Hub

📋Steps in this guide1/2

SYSRESV Utility and "ORA-01041: internal error. hostdef extension doesn't exist"

This article provides an example of using the SYSRESV utility to identify the shared memory segments associated with an Oracle instance.

oracle miscconfigurationintermediate
by OracleDba
14 views
1

The Problem

A couple of times we've had problems with instances not responding. When we try to connect to the database from the server we get something like this. When you check on the OS you can see the processes running, but there is no way to connect. Even a preliminary connection fails. There is nothing you can do with the instance, so the obvious thing to try and do is kill it, but you can’t do this with a "shutdown" or "shutdown abort" because you can't connect. Instead you try to kill it at the OS level. After this you see all the processes gone, but you still can't connect to the instance.

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ export ORACLE_SID=orcl
$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 13 08:41:23 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

ERROR:
ORA-01041: internal error. hostdef extension doesn't exist

$

$ kill -9 `ps -ef | grep orcl | awk '{print $2}'`

$ export ORACLE_SID=orcl
$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 13 08:41:23 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

ERROR:
ORA-01041: internal error. hostdef extension doesn't exist

$
2

The Solution (SYSRESV Utility)

The last time this happened we guessed it might be about shared memory segments hanging around after the instance has died. You can check the shared memory directly, but it can be difficult to relate this back to the instances. One way is by comparing the creation time of the shared memory segment with the column of other running instances on the server and use a process of elimiation. The type of output you can expect from "ipcs -mpt" is shown below. Instead, you can use the "$ORACLE_HOME/bin/sysresv" command to give you the shared memory segments associated with a named instance. So we can see there are shared memory segments for an instance that is now dead. It worked as expected, but if for some reason it is not able to link the dead instance to the shared memory segments, you can always work backwards. Check the segments for all the running instances. If there are any shared memory segments for "oracle oinstall" from the "ipcs -mpt" command that are not seen in the following output, by a process of elimination, they must belong to the dead instance. Once we know the dodgy shared memory segments (directly or indirectly) we can we can remove them using "ipcrm". Once that is done, you should be able to connect to, and start the instance normally. It's possible once you remove the first segment (98338) the others will no longer be seen as linked to the instance, but you will still need to remove them or you will be able to connect, but the instance will not start. For more information see: - SYSRESV Utility (Doc ID 123322.1) Hope this helps. Regards Tim...

Code/Command (click line numbers to comment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
$ ipcs -mpt
IPC status from /dev/kmem as of Wed Jul 13 10:04:09 2016
T         ID     KEY        MODE        OWNER     GROUP  CPID  LPID   ATIME    DTIME    CTIME
Shared Memory:
m          0 0x411c0123 --rw-rw-rw-      root      root   957 29670  7:14:00  7:14:00  9:47:13
m          1 0x4e0c0002 --rw-rw-rw-      root      root   957  4731  9:06:51  9:15:11  9:47:13
m          2 0x41200006 --rw-rw-rw-      root      root   957 29670  7:14:00  7:14:00  9:47:13
m          3 0x00a5c581 --rw-------     sfmdb     users  2065  2102  9:48:14  9:48:14  9:48:14
m          4 0x06347849 --rw-rw-rw-      root      root  2623  2629  9:48:50  9:48:47  9:48:47
m          5 0x0c6629c9 --rw-r-----      root      root  2644  4295 10:33:52 10:34:17  9:48:47
m      32774 0xa90c08cc -----------      root      root  2954  2955  9:49:03 no-entry  9:49:03
m          7 0x012009d0 --rw-rw-r--      root      root  2660  2660  9:48:48  9:48:48  9:48:48
m          8 0x012009cc --rw-rw-r--      root      root  2662  2662  9:48:48  9:48:48  9:48:48
m          9 0x012009c4 --rw-rw-r--      root      root  2666  2666  9:48:48 no-entry  9:48:48
m         10 0x49143b35 --rw-r--r--      root      root  2629  2629 10:04:00 10:04:00  9:48:49
m         11 0x012009ce --rw-rw-r--      root      root  2709  2709  9:48:49 no-entry  9:48:49
m      32780 0x00000000 --rw-r-----    oracle  oinstall  6230 11639 10:03:44 10:03:44 12:00:36
m         13 0x00000000 --rw-r-----    oracle  oinstall  6230 11639 10:03:44 10:03:44 12:00:36
m         14 0x00000000 --rw-r-----    oracle  oinstall  6230 11639 10:03:44 10:03:44 12:00:36
m         15 0x588c2d20 --rw-r-----    oracle  oinstall  6230 11639 10:03:44 10:03:44 12:00:36
m      32784 0x00000000 --rw-r-----    oracle  oinstall  6560 11683 10:04:03 10:04:03 12:01:02
m         17 0x00000000 --rw-r-----    oracle  oinstall  6560 11683 10:04:03 10:04:03 12:01:02
m         18 0x375d46a0 --rw-r-----    oracle  oinstall  6560 11683 10:04:03 10:04:03 12:01:02
m      32787 0x00000000 --rw-r-----    oracle  oinstall  7117 11663 10:04:00 10:03:46 12:04:33
m         20 0x00000000 --rw-r-----    oracle  oinstall  7117 11663 10:04:00 10:03:46 12:04:33
m         21 0x00000000 --rw-r-----    oracle  oinstall  7117 11663 10:04:00 10:03:46 12:04:33
m         22 0x00000000 --rw-r-----    oracle  oinstall  7117 11663 10:04:00 10:03:46 12:04:33
m         23 0x10badea0 --rw-r-----    oracle  oinstall  7117 11663 10:04:00 10:04:00 12:04:33
m      98328 0x00000000 --rw-r-----    oracle  oinstall  7969 11681 10:04:02 10:04:02 12:06:39
m         25 0x00000000 --rw-r-----    oracle  oinstall  7969 11681 10:04:02 10:04:02 12:06:39
m         26 0x00000000 --rw-r-----    oracle  oinstall  7969 11681 10:04:02 10:04:02 12:06:39
m         27 0x00000000 --rw-r-----    oracle  oinstall  7969 11681 10:04:02 10:04:02 12:06:39
m         28 0x09c1865c --rw-r-----    oracle  oinstall  7969 11681 10:04:02 10:04:02 12:06:39
m     196637 0x00000000 --rw-r-----    oracle  oinstall  8741 11665 10:04:00 10:03:49 12:08:13
m         30 0x00000000 --rw-r-----    oracle  oinstall  8741 11665 10:04:00 10:03:49 12:08:13
m         31 0x00000000 --rw-r-----    oracle  oinstall  8741 11665 10:04:00 10:03:49 12:08:13
m         32 0x00000000 --rw-r-----    oracle  oinstall  8741 11665 10:04:00 10:03:49 12:08:13
m         33 0xb747ec2c --rw-r-----    oracle  oinstall  8741 11665 10:04:00 10:04:00 12:08:13
m     426018 0x00000000 --rw-r-----    oracle  oinstall  2869  9042 10:03:02 10:04:00  8:55:00
m      32803 0x00000000 --rw-r-----    oracle  oinstall  2869  9042 10:03:02 10:04:00  8:55:00
m      32804 0x00000000 --rw-r-----    oracle  oinstall  2869  9042 10:03:02 10:04:00  8:55:00
m      32805 0x00000000 --rw-r-----    oracle  oinstall  2869  9042 10:03:02 10:04:00  8:55:00
m      32806 0x472f4a20 --rw-r-----    oracle  oinstall  2869  9042 10:03:02 10:04:00  8:55:00
$

$ sysresv -l orcl

IPC Resources for ORACLE_SID "orcl" :
Shared Memory:
ID              KEY
98338           0x00000000
35              0x00000000
36              0x00000000
37              0x00000000
38              0x472f4a20
Semaphores:
ID              KEY
12312           0xdd03b17c
Oracle Instance not alive for sid "orcl"
$

$ sysresv -l instance1 instance2 instance3

$ ipcrm -m 98338
$ ipcrm -m 35
$ ipcrm -m 36
$ ipcrm -m 37
$ ipcrm -m 38

Comments (0)

Please to add comments

No comments yet. Be the first to comment!