%deffont "standard" xfont "goth-regular-r"
%deffont "oops" xfont "courier-bold-r"
%deffont "title" xfont "goth-regular-r"
%deffont "title-italic" xfont "goth-regular-i"
%deffont "quote" xfont "avantgarde-book-o"
%%
%default 1 leftfill, size 2, fore "orange", vgap 20, back "black", font "title"
%default 2 size 7, vgap 10, prefix " "
%default 3 size 2, bar "blue", vgap 10
%default 4 size 5, fore "white", vgap 30, prefix " ", font "standard"
%%
%tab 1 size 5, vgap 40, prefix "  ", icon arc "white" 50
%tab 2 size 4, vgap 40, prefix "            ", icon delta3 "orange" 40
%tab 3 size 3, vgap 40, prefix "            ", icon box "white" 40
%tab 4 size 3, vgap 40, prefix "                  ", icon dia "orange" 40
%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%nodefault
%size 19, font "title", fore "white", vgap 20, back "black"
%bquality 10

%center
OOPS!
%size 6, font "title-italic", fore "white", vgap 20, back "black"


An Introduction to Linux Kernel Debugging



%size 3
Simon Trimmer <simon@veritas.com>

%image "EMBEDDIR/vrts-logo.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Overview



	types of crashes

	oopsen
		capturing
		decoding
		debugging...

	lockups

	bug reports

	further information

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Not Covering...


Would have liked to have covered...

	OS principles

	kernel debuggers

	crash dump generators

	hardware faults

%size 3
If there is the interest we could run another talk to cover them \
some other time.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Different types of crashes



	the common oops

	lockup types
		temporary and permanent

	spontaneous reboot (not covered in detail)



Q? (what sort of audience is this?)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page






%size 10, font "title-italic", fore "white", vgap 20, back "black"
%bquality 10

%center
part 1a



%size 7
oops basics

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

The Oops


An oops is triggered by some exception and is a dump of the CPU state and kernel stack at that instant.


oopsen can get sent to:

	the console
		possibly a serial port

	the kernel ring buffer
		klogd pulls it out and sends it to syslogd

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Example

%font "oops"
%size 3

Unable to handle kernel NULL pointer dereference at virtual address  00000014
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c017d558>]
EFLAGS: 00210213
eax: 00000000   ebx: c6155c6c   ecx: 00000038   edx: 00000000
esi: c672f000   edi: c672f07c   ebp: 00000004   esp: c6155b0c
ds: 0018   es: 0018   ss: 0018
Process tar (pid: 2293, stackpage=c6155000)
Stack: c672f000 c672f07c 00000000 00000038 00000060 00000000 c6d7d2a0 c6c79018 
       00000001 c6155c6c 00000000 c6d7d2a0 c017eb4f c6155c6c 00000000 00000098 
       c017fc44 c672f000 00000084 00001020 00001000 c7129028 00000038 00000069 
Call Trace: [<c017eb4f>] [<c017fc44>] [<c0180115>] [<c018a1c8>] [<c017bb3a>] [<c018738f>] [<c0177a13>] 
       [<d0871044>] [<c0178274>] [<c0142e36>] [<c013c75f>] [<c013c7f8>] [<c0108f77>] [<c010002b>] 

Code: 8b 40 14 ff d0 89 c2 8b 06 83 c4 10 01 c2 89 16 8b 83 8c 01 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Safety




%center
%font "title-italic", size 7
You can't trust an oopsed kernel

%font "standard", size 5

%left
The running kernel component was killed along with any userspace process without releasing locks or cleaning up structures

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Decoding oopsen

a closer look at the oops dump...
%font "oops"
%size 3

Unable to handle kernel NULL pointer dereference at virtual address  00000014
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c017d558>]
EFLAGS: 00210213
eax: 00000000   ebx: c6155c6c   ecx: 00000038   edx: 00000000
esi: c672f000   edi: c672f07c   ebp: 00000004   esp: c6155b0c
ds: 0018   es: 0018   ss: 0018
Process tar (pid: 2293, stackpage=c6155000)
Stack: c672f000 c672f07c 00000000 00000038 00000060 00000000 c6d7d2a0 c6c79018 
       00000001 c6155c6c 00000000 c6d7d2a0 c017eb4f c6155c6c 00000000 00000098 
       c017fc44 c672f000 00000084 00001020 00001000 c7129028 00000038 00000069 
Call Trace: [<c017eb4f>] [<c017fc44>] [<c0180115>] [<c018a1c8>] [<c017bb3a>] [<c018738f>] [<c0177a13>] 
       [<d0871044>] [<c0178274>] [<c0142e36>] [<c013c75f>] [<c013c7f8>] [<c0108f77>] [<c010002b>] 

Code: 8b 40 14 ff d0 89 c2 8b 06 83 c4 10 01 c2 89 16 8b 83 8c 01 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Decoding oopsen: fault

%font "oops", size 3, fore "green"
Unable to handle kernel NULL pointer dereference at virtual address  00000014
%fore "white"
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c017d558>]
EFLAGS: 00210213
eax: 00000000   ebx: c6155c6c   ecx: 00000038   edx: 00000000
esi: c672f000   edi: c672f07c   ebp: 00000004   esp: c6155b0c
ds: 0018   es: 0018   ss: 0018
Process tar (pid: 2293, stackpage=c6155000)
Stack: c672f000 c672f07c 00000000 00000038 00000060 00000000 c6d7d2a0 c6c79018 
       00000001 c6155c6c 00000000 c6d7d2a0 c017eb4f c6155c6c 00000000 00000098 
       c017fc44 c672f000 00000084 00001020 00001000 c7129028 00000038 00000069 
Call Trace: [<c017eb4f>] [<c017fc44>] [<c0180115>] [<c018a1c8>] [<c017bb3a>] [<c018738f>] [<c0177a13>] 
       [<d0871044>] [<c0178274>] [<c0142e36>] [<c013c75f>] [<c013c7f8>] [<c0108f77>] [<c010002b>] 

Code: 8b 40 14 ff d0 89 c2 8b 06 83 c4 10 01 c2 89 16 8b 83 8c 01 

%font "standard", fore "yellow", size 5
(low address implies accessing a structure member)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Decoding oopsen: oops counter

%font "oops", size 3
Unable to handle kernel NULL pointer dereference at virtual address  00000014
*pde = 00000000
%fore "green"
Oops: 0000
%fore "white"
CPU:    0
EIP:    0010:[<c017d558>]
EFLAGS: 00210213
eax: 00000000   ebx: c6155c6c   ecx: 00000038   edx: 00000000
esi: c672f000   edi: c672f07c   ebp: 00000004   esp: c6155b0c
ds: 0018   es: 0018   ss: 0018
Process tar (pid: 2293, stackpage=c6155000)
Stack: c672f000 c672f07c 00000000 00000038 00000060 00000000 c6d7d2a0 c6c79018 
       00000001 c6155c6c 00000000 c6d7d2a0 c017eb4f c6155c6c 00000000 00000098 
       c017fc44 c672f000 00000084 00001020 00001000 c7129028 00000038 00000069 
Call Trace: [<c017eb4f>] [<c017fc44>] [<c0180115>] [<c018a1c8>] [<c017bb3a>] [<c018738f>] [<c0177a13>] 
       [<d0871044>] [<c0178274>] [<c0142e36>] [<c013c75f>] [<c013c7f8>] [<c0108f77>] [<c010002b>] 

Code: 8b 40 14 ff d0 89 c2 8b 06 83 c4 10 01 c2 89 16 8b 83 8c 01 

%fore "yellow", font "standard", size 5
(often one oops triggers more, only the first is reliable)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Decoding oopsen: EIP

%font "oops"
%size 3
Unable to handle kernel NULL pointer dereference at virtual address  00000014
*pde = 00000000
Oops: 0000
CPU:    0
%fore "green"
EIP:    0010:[<c017d558>]
%fore "white"
EFLAGS: 00210213
eax: 00000000   ebx: c6155c6c   ecx: 00000038   edx: 00000000
esi: c672f000   edi: c672f07c   ebp: 00000004   esp: c6155b0c
ds: 0018   es: 0018   ss: 0018
Process tar (pid: 2293, stackpage=c6155000)
Stack: c672f000 c672f07c 00000000 00000038 00000060 00000000 c6d7d2a0 c6c79018 
       00000001 c6155c6c 00000000 c6d7d2a0 c017eb4f c6155c6c 00000000 00000098 
       c017fc44 c672f000 00000084 00001020 00001000 c7129028 00000038 00000069 
Call Trace: [<c017eb4f>] [<c017fc44>] [<c0180115>] [<c018a1c8>] [<c017bb3a>] [<c018738f>] [<c0177a13>] 
       [<d0871044>] [<c0178274>] [<c0142e36>] [<c013c75f>] [<c013c7f8>] [<c0108f77>] [<c010002b>] 

Code: 8b 40 14 ff d0 89 c2 8b 06 83 c4 10 01 c2 89 16 8b 83 8c 01 

%fore "yellow", font "standard", size 5
(EIP shows the code segment and instruction address)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Decoding oopsen: EFLAGS + Registers

%font "oops"
%size 3
Unable to handle kernel NULL pointer dereference at virtual address  00000014
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c017d558>]
%fore "green"
EFLAGS: 00210213
eax: 00000000   ebx: c6155c6c   ecx: 00000038   edx: 00000000
esi: c672f000   edi: c672f07c   ebp: 00000004   esp: c6155b0c
ds: 0018   es: 0018   ss: 0018
%fore "white"
Process tar (pid: 2293, stackpage=c6155000)
Stack: c672f000 c672f07c 00000000 00000038 00000060 00000000 c6d7d2a0 c6c79018 
       00000001 c6155c6c 00000000 c6d7d2a0 c017eb4f c6155c6c 00000000 00000098 
       c017fc44 c672f000 00000084 00001020 00001000 c7129028 00000038 00000069 
Call Trace: [<c017eb4f>] [<c017fc44>] [<c0180115>] [<c018a1c8>] [<c017bb3a>] [<c018738f>] [<c0177a13>] 
       [<d0871044>] [<c0178274>] [<c0142e36>] [<c013c75f>] [<c013c7f8>] [<c0108f77>] [<c010002b>] 

Code: 8b 40 14 ff d0 89 c2 8b 06 83 c4 10 01 c2 89 16 8b 83 8c 01 
%fore "yellow", font "standard", size 5
(program status and control register, the general purpose registers and more segment registers)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Decoding oopsen: stack

%font "oops"
%size 3
Unable to handle kernel NULL pointer dereference at virtual address  00000014
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c017d558>]
EFLAGS: 00210213
eax: 00000000   ebx: c6155c6c   ecx: 00000038   edx: 00000000
esi: c672f000   edi: c672f07c   ebp: 00000004   esp: c6155b0c
ds: 0018   es: 0018   ss: 0018
%fore "green"
Process tar (pid: 2293, stackpage=c6155000)
Stack: c672f000 c672f07c 00000000 00000038 00000060 00000000 c6d7d2a0 c6c79018 
       00000001 c6155c6c 00000000 c6d7d2a0 c017eb4f c6155c6c 00000000 00000098 
       c017fc44 c672f000 00000084 00001020 00001000 c7129028 00000038 00000069 
%fore "white"
Call Trace: [<c017eb4f>] [<c017fc44>] [<c0180115>] [<c018a1c8>] [<c017bb3a>] [<c018738f>] [<c0177a13>] 
       [<d0871044>] [<c0178274>] [<c0142e36>] [<c013c75f>] [<c013c7f8>] [<c0108f77>] [<c010002b>] 

Code: 8b 40 14 ff d0 89 c2 8b 06 83 c4 10 01 c2 89 16 8b 83 8c 01 

%font "standard", fore "yellow", size 5
(stack, full of stuff, half run ops, return addresses...)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Decoding oopsen: call trace

%font "oops"
%size 3
Unable to handle kernel NULL pointer dereference at virtual address  00000014
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c017d558>]
EFLAGS: 00210213
eax: 00000000   ebx: c6155c6c   ecx: 00000038   edx: 00000000
esi: c672f000   edi: c672f07c   ebp: 00000004   esp: c6155b0c
ds: 0018   es: 0018   ss: 0018
Process tar (pid: 2293, stackpage=c6155000)
Stack: c672f000 c672f07c 00000000 00000038 00000060 00000000 c6d7d2a0 c6c79018 
       00000001 c6155c6c 00000000 c6d7d2a0 c017eb4f c6155c6c 00000000 00000098 
       c017fc44 c672f000 00000084 00001020 00001000 c7129028 00000038 00000069 
%fore "green"
Call Trace: [<c017eb4f>] [<c017fc44>] [<c0180115>] [<c018a1c8>] [<c017bb3a>] [<c018738f>] [<c0177a13>] 
       [<d0871044>] [<c0178274>] [<c0142e36>] [<c013c75f>] [<c013c7f8>] [<c0108f77>] [<c010002b>] 
%fore "white"

Code: 8b 40 14 ff d0 89 c2 8b 06 83 c4 10 01 c2 89 16 8b 83 8c 01 

%font "standard", fore "yellow", size 5
(claim to be return addresses, useless on their own)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Resolving addresses


%font "standard"
(from oops output)
%font "oops"
EIP:    0010:[<c017d558>]


%font "standard"
%pause
%lcutin
(from System.map)

%font "oops"
....
c017cdf0 T reiserfs_dir_fsync
c017ce80 t reiserfs_readdir
%fore "green"
c017d2f0 t create_virtual_node
%fore "white"
c017d780 t check_left
c017d8d0 t check_right
....

%font "title-italic"
EIP = function base address + instruction offset

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Decoding with ksymoops

%font "oops"
%size 3

...
Process tar (pid: 2293, stackpage=c6155000)
...

>>EIP; c017d558 <create_virtual_node+298/490>   <=====
Trace; c017eb4f <ip_check_balance+34f/ae0>
Trace; c017fc44 <reiserfs_kfree+14/50>
Trace; c0180115 <fix_nodes+115/450>
Trace; c018a1c8 <reiserfs_insert_item+88/110>
Trace; c017bb3a <reiserfs_new_inode+3da/500>
Trace; c018738f <pathrelse+1f/30>
Trace; c0177a13 <reiserfs_lookup+73/d0>
Trace; d0871044 <END_OF_CODE+9a77/???
Trace; c0178274 <reiserfs_mkdir+d4/1d0>
Trace; c0142e36 <d_alloc+16/160>
Trace; c013c75f <vfs_mkdir+7f/b0>
Trace; c013c7f8 <sys_mkdir+68/b0>
Trace; c0108f77 <system_call+33/38>
Trace; c010002b <startup_32+2b/139>

Code;  c017d558 <create_virtual_node+298/490>
00000000 <_EIP>:
Code;  c017d558 <create_virtual_node+298/490>   <=====
   0:   8b 40 14                  mov    0x14(%eax),%eax   <=====
Code;  c017d55b <create_virtual_node+29b/490>
   3:   ff d0                     call   *%eax

(cut)


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Decoding with klogd

it may also be saved in the system log, klogd also performs some symbol lookups

%font "oops", size 2
Jan 23 19:44:00 localhost kernel:  printing eip:
Jan 23 19:44:00 localhost kernel: c017d558
Jan 23 19:44:00 localhost kernel: *pde = 00000000
Jan 23 19:44:00 localhost kernel: Oops: 0000
Jan 23 19:44:00 localhost kernel: CPU:    0
Jan 23 19:44:00 localhost kernel: EIP:    0010:[create_virtual_node+664/1168]
Jan 23 19:44:00 localhost kernel: EFLAGS: 00210213
Jan 23 19:44:00 localhost kernel: eax: 00000000   ebx: c6155c6c   ecx: 00000038   edx: 00000000
Jan 23 19:44:00 localhost kernel: esi: c672f000   edi: c672f07c   ebp: 00000004   esp: c6155b0c
Jan 23 19:44:00 localhost kernel: ds: 0018   es: 0018   ss: 0018
Jan 23 19:44:00 localhost kernel: Process tar (pid: 2293, stackpage=c6155000)
Jan 23 19:44:00 localhost kernel: Stack: c672f000 c672f07c 00000000 00000038 00000060 00000000 c6d7d2a0 c6c79018 
Jan 23 19:44:00 localhost kernel:        00000001 c6155c6c 00000000 c6d7d2a0 c017eb4f c6155c6c 00000000 00000098 
Jan 23 19:44:00 localhost kernel:        c017fc44 c672f000 00000084 00001020 00001000 c7129028 00000038 00000069 
Jan 23 19:44:00 localhost kernel: Call Trace: [ip_check_balance+847/2784] [reiserfs_kfree+20/80] [fix_nodes+277/1104] [reiserfs_insert_item+136/272] [reiserfs_new_inode+986/1280] [pathrelse+31/48] [reiserfs_lookup+115/208] 
Jan 23 19:44:00 localhost kernel:        [<d0871044>] [reiserfs_mkdir+212/464] [d_alloc+22/352] [vfs_mkdir+127/176] [sys_mkdir+104/176] [system_call+51/56] [stext+43/313] 
Jan 23 19:44:00 localhost kernel: 
Jan 23 19:44:00 localhost kernel: Code: 8b 40 14 ff d0 89 c2 8b 06 83 c4 10 01 c2 89 16 8b 83 8c 01 



%font "title-italic", size 4
Note: these offsets are in base10, not base16 like everywhere else!

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page






%size 10, font "title-italic", fore "white", vgap 20, back "black"
%bquality 10

%center
part 1b



%size 7
Basic Debugging

%size 3
one step beyond!


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Disassembler basics


With the function name and offset we can work out where the oops happened



	match the address with the assembler function

	match the function offset to the instruction

	match the assembler instructions to the C source

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

objdump

%font "oops", size 2
$ objdump -d fix_node.o
fix_node.o:     file format elf32-i386

Disassembly of section .text:

00000000 <create_virtual_node>:
       0:       55                      push   %ebp
       1:       57                      push   %edi
       2:       56                      push   %esi
       3:       53                      push   %ebx
       4:       83 ec 10                sub    $0x10,%esp
       7:       8b 44 24 24             mov    0x24(%esp,1),%eax
       b:       8b 5c 24 28             mov    0x28(%esp,1),%ebx
       f:       8b 50 10                mov    0x10(%eax),%edx
      12:       8b b0 90 01 00 00       mov    0x190(%eax),%esi

%font "title", size 3
..... remember offset 298?
%font "oops", size 2
     25d:       b8 02 00 00 00          mov    $0x2,%eax
     262:       74 05                   je     269 <create_virtual_node+0x269>
     264:       b8 0f 00 00 00          mov    $0xf,%eax
     269:       89 c1                   mov    %eax,%ecx
     26b:       eb 10                   jmp    27d <create_virtual_node+0x27d>
     26d:       8d 76 00                lea    0x0(%esi),%esi
     270:       0f b6 4a 0f             movzbl 0xf(%edx),%ecx
     274:       c0 e9 04                shr    $0x4,%cl
     277:       0f b6 c1                movzbl %cl,%eax
     27a:       0f b7 c8                movzwl %ax,%ecx
     27d:       8b 5c 24 24             mov    0x24(%esp,1),%ebx
     281:       89 c8                   mov    %ecx,%eax
     283:       8b 04 85 00 00 00 00    mov    0x0(,%eax,4),%eax
     28a:       8b 8b 24 01 00 00       mov    0x124(%ebx),%ecx
     290:       51                      push   %ecx
     291:       8b 54 24 08             mov    0x8(%esp,1),%edx
     295:       52                      push   %edx
     296:       57                      push   %edi
     297:       56                      push   %esi
     298:       8b 40 14                mov    0x14(%eax),%eax
     29b:       ff d0                   call   *%eax
     29d:       89 c2                   mov    %eax,%edx
     29f:       8b 06                   mov    (%esi),%eax
     2a1:       83 c4 10                add    $0x10,%esp
     2a4:       01 c2                   add    %eax,%edx

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Matching assembler to C


Things to look for:

		flow control... ifs, cases, whiles equate to tests, cmps and jumps
		returns equate to big jumps to near the end of the functions
		calling functions... pushing arguments onto the stack and making a call
		constants, ors, locks all stand out (asm/spinlock.h)
		use of pointers, arithmetic, memory manipulation

More than one way of doing things, compilers select different methods in different cases

It's a bit of an art, you get used to it....
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

objdump (again)

%font "oops", size 2
$ objdump -d fix_node.o
fix_node.o:     file format elf32-i386

Disassembly of section .text:

00000000 <create_virtual_node>:
       0:       55                      push   %ebp
       1:       57                      push   %edi
       2:       56                      push   %esi
       3:       53                      push   %ebx
       4:       83 ec 10                sub    $0x10,%esp
       7:       8b 44 24 24             mov    0x24(%esp,1),%eax
       b:       8b 5c 24 28             mov    0x28(%esp,1),%ebx
       f:       8b 50 10                mov    0x10(%eax),%edx
      12:       8b b0 90 01 00 00       mov    0x190(%eax),%esi

%font "title", size 3
..... remember offset 298?
%font "oops", size 2
     25d:       b8 02 00 00 00          mov    $0x2,%eax
     262:       74 05                   je     269 <create_virtual_node+0x269>
     264:       b8 0f 00 00 00          mov    $0xf,%eax
     269:       89 c1                   mov    %eax,%ecx
     26b:       eb 10                   jmp    27d <create_virtual_node+0x27d>
     26d:       8d 76 00                lea    0x0(%esi),%esi
     270:       0f b6 4a 0f             movzbl 0xf(%edx),%ecx
     274:       c0 e9 04                shr    $0x4,%cl
     277:       0f b6 c1                movzbl %cl,%eax
     27a:       0f b7 c8                movzwl %ax,%ecx
     27d:       8b 5c 24 24             mov    0x24(%esp,1),%ebx
     281:       89 c8                   mov    %ecx,%eax
     283:       8b 04 85 00 00 00 00    mov    0x0(,%eax,4),%eax
     28a:       8b 8b 24 01 00 00       mov    0x124(%ebx),%ecx
     290:       51                      push   %ecx
     291:       8b 54 24 08             mov    0x8(%esp,1),%edx
     295:       52                      push   %edx
     296:       57                      push   %edi
     297:       56                      push   %esi
     298:       8b 40 14                mov    0x14(%eax),%eax
     29b:       ff d0                   call   *%eax
     29d:       89 c2                   mov    %eax,%edx
     29f:       8b 06                   mov    (%esi),%eax
     2a1:       83 c4 10                add    $0x10,%esp
     2a4:       01 c2                   add    %eax,%edx

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
fs/reiserfs/fix_node.c
create_virtual_node()



%font "oops", size 2
    /* go through all items those remain in the virtual node (except for the new (inserted) one) */
    for (new_num = 0; new_num < vn->vn_nr_item; new_num ++) {
        int j;
        struct virtual_item * vi = vn->vn_vi + new_num;
        int is_affected = ((new_num != vn->vn_affected_item_num) ? 0 : 1);


        if (is_affected && vn->vn_mode == M_INSERT)
            continue;

        /* get item number in source node */
        j = old_item_num (new_num, vn->vn_affected_item_num, vn->vn_mode);

        vi->vi_item_len += ih[j].ih_item_len + IH_SIZE;
        vi->vi_ih = ih + j;
        vi->vi_item = B_I_PITEM (Sh, ih + j);
        vi->vi_uarea = vn->vn_free_ptr;

%fore "yellow"
        // FIXME: there is no check, that item operation did not
        // consume too much memory
        vn->vn_free_ptr += op_create_vi (vn, vi, is_affected, tb->insert_size [0]);
%fore "white"
        if (tb->vn_buf + tb->vn_buf_size < vn->vn_free_ptr)
            reiserfs_panic (tb->tb_sb, "vs-8030: create_virtual_node: "
                            "virtual node space consumed");

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

objdump for cheats :)

%size 3
(with -g compiled kernel objects)

%font "oops", size 2
$ objdump --source -d fix_node.o
fix_node.o:     file format elf32-i386

Disassembly of section .text:

00000000 <create_virtual_node>:
  return new_num + 1;
}

static void create_virtual_node (struct tree_balance * tb, int h)
{
       0:       55                      push   %ebp
       1:       57                      push   %edi
       2:       56                      push   %esi
       3:       53                      push   %ebx
       4:       83 ec 10                sub    $0x10,%esp
    struct item_head * ih;
    struct virtual_node * vn = tb->tb_vn;
       7:       8b 44 24 24             mov    0x24(%esp,1),%eax
       b:       8b 5c 24 28             mov    0x28(%esp,1),%ebx

...

        // FIXME: there is no check, that item operation did not
        // consume too much memory
        vn->vn_free_ptr += op_create_vi (vn, vi, is_affected, tb->insert_size [0]);
     27d:       8b 5c 24 24             mov    0x24(%esp,1),%ebx
     281:       89 c8                   mov    %ecx,%eax
     283:       8b 04 85 00 00 00 00    mov    0x0(,%eax,4),%eax
     28a:       8b 8b 24 01 00 00       mov    0x124(%ebx),%ecx
     290:       51                      push   %ecx
     291:       8b 54 24 08             mov    0x8(%esp,1),%edx
     295:       52                      push   %edx
     296:       57                      push   %edi
     297:       56                      push   %esi
     298:       8b 40 14                mov    0x14(%eax),%eax
     29b:       ff d0                   call   *%eax
     29d:       89 c2                   mov    %eax,%edx
     29f:       8b 06                   mov    (%esi),%eax



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

GDB

%font "title", size 4
what to expect...

%font "oops", size 2
(gdb) gdb vmlinux
GNU gdb 5.0
Copyright 2000 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux"...
(gdb) 
(gdb) disassemble create_virtual_node
Dump of assembler code for function create_virtual_node:
0xc017d2f0 <create_virtual_node>:       push   %ebp
0xc017d2f1 <create_virtual_node+1>:     push   %edi
0xc017d2f2 <create_virtual_node+2>:     push   %esi
0xc017d2f3 <create_virtual_node+3>:     push   %ebx

......and so on

(gdb) info scope create_virtual_node
Scope for create_virtual_node:
Symbol tb is an argument at stack/frame offset 36, length 4.
Symbol h is an argument at stack/frame offset 40, length 4.
Symbol h is a local variable in register $ebx, length 4.
Symbol ih is a local variable at frame offset 12, length 4.
Symbol vn is a local variable in register $esi, length 4.
Symbol new_num is a local variable in register $ebp, length 4.
Symbol Sh is a local variable at frame offset 8, length 4.
(gdb) 

%font "title", size 4
Can run gdb on a running kernel
(include debugging symbols, that's -g in CFLAGS)

%font "oops", size 2
> gdb -q vmlinux /proc/kcore

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Hang on,


So, what went wrong?

%font "standard",size 4
it faulted on the mov (298)
%font "oops",size 3
     290:       51                      push   %ecx
     291:       8b 54 24 08             mov    0x8(%esp,1),%edx
     295:       52                      push   %edx
     296:       57                      push   %edi
     297:       56                      push   %esi
     298:       8b 40 14                mov    0x14(%eax),%eax
     29b:       ff d0                   call   *%eax

%pause

%font "standard",size 4
but...the call was:
%font "oops",size 3
op_create_vi(vn, vi, is_affected, tb->insert_size [0]);

%font "title-italic",size 4
why is it loading EAX rather than calling the address directly?

%pause
The answer is: it's another macro...

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

So what went wrong? Revisited


%font "standard",size 4
it faulted on the mov (298)
%font "oops",size 3
     290:       51                      push   %ecx
     291:       8b 54 24 08             mov    0x8(%esp,1),%edx
     295:       52                      push   %edx
     296:       57                      push   %edi
     297:       56                      push   %esi
     298:       8b 40 14                mov    0x14(%eax),%eax
     29b:       ff d0                   call   *%eax


%font "oops",size 3
 #define op_create_vi(vn,vi,is_affected,insert_size)
%fore "yellow"
      item_ops[le_ih_k_type ((vi)->vi_ih)]
%cont, fore "white"
->create_vi (vn,vi,is_affected,insert_size)


%font "standard",size 4
It's getting the address of the handler function from an array called item_ops, \
but it's basing the offset into that array on a pointer which could have failed \
without checking it.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Enough, please!


#%font "title-italic"
# you can't solve the problem just by putting in a check for the null pointer, that's covering up the bug rather than fixing it
%font "quote"
%size 4
...

I'm afraid that I've seen too many people fix bugs by looking at debugger output, and that almost inevitably leads to fixing the symptoms rather than the underlying problems. 

...

"Use the Source, Luke, use the Source.  Be one with the code.".  Think of Luke Skywalker discarding the automatic firing system when closing on the deathstar, and firing the proton torpedo (or whatever) manually.  _Then_ do you have the right mindset for fixing kernel bugs.

...
%right
Linus Torvalds <torvalds@transmeta.com>

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page






%size 10, font "title-italic", fore "white", vgap 20, back "black"
%bquality 10

%center
part 2


%size 7

lockups
%size 4
(all down hill from here!)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Lockups


Lockups are when the system just stops, no messages and doesn't respond

Kinds of lockups:

	hardware lock ups

	lockups with interrupts enabled

	lockups without interrupts enabled


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Hardware lockup



Tricky little area,


	Andrea Arcangeli's print-EIP patch (part of IKD)

	Hardware monitoring tool (expensive!)


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Lockups with IRQs enabled




Spinning in a loop? waiting on a lock?

Toggle keyboard lights, like caps-lock

The kernel is half alive,

On console, try the sysinfo keys
		shift-scroll lock	show memory
		control-scroll lock	show process state
		rightalt-scroll lock	show registers

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Magic sysrq key


		usually ALT-PRINTSCREEN on x86
		BREAK on serial console
		often disabled on distributions
		echo "1" > /proc/sys/kernel/sysrq

It can:
			sync data to disks					s
			remount partitions read only				u
			reboot the machine					b
			power the machine off				o

			turns off keyboard raw mode and puts it in XLATE	r
			dump the cpu registers and flags			p
			list tasks and some useful information about them	t
			kills of a process on the current console		k
			send SIGTERM to everything except init			e
			send SIGKILL to everything except init			i
			send SIGKILL to everything including init		l
			change the console loglevel				0-9

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Using the magic sysrq key


	use SYSRQ-P to dump cpu information a few times

%font "oops", size 3
SysRq: Show Regs

EIP: 0010:[<c011251f>] CPU: 0 EFLAGS: 00200296
EAX: 00005305 EBX: 00000000 ECX: 00000000 EDX: c147bfa0
ESI: 00000002 EDI: 00005305 EBP: 00000001 DS: 0018 ES: 0018
CR0: 8005003b CR2: 4001a000 CR3: 0cc66000 CR4: 00000690
Call Trace: [<c0112604>] [<c0112eb7>] [<c011389e>] [<c0113600>] 
[<c010757b>]

%font "title"
	look to see how much the EIP changes
		if it's in a loop you can spot the small changes in the address
		resolve the EIP like in an oops to find out where
		deadlocks, lock ordering inversion

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Lockups with IRQs disabled



	The NMI Watchdog

		NMI = non-maskable interrupt
		delivered whatever (usually...)
		can detect when a cpu is locked up
		NMI watchdog ticks every few seconds
		when the system locks up it will automatically generate an oops

	Print EIP patch might help

	X Windows

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Making bug reports

See REPORTING-BUGS file in kernel souce for a suggested bug-report format

Information required
		summarise the bug into a line or so
		use the short summary as the subject of the email
		have a more terse description of the bug
		decoded oops
		kernel version
		kernel configuration
		if you can describe how to reproduce the bug
		any other relevant information

Email the report to the maintainer of that system
		MAINTAINERS file
		if you don't know who to send it to, sent it to the Linux Kernel Mailing list

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Summary


		different lockups and oopsen
		collecting information
		decoding the raw data into something useful
		what to do with useful information


%font "title-italic", size 5, center
being able to apply this makes you useful!
%left

%font "title", size 4
If you want to go forward and learn more...

		understand the kernel
		learn C and Intel assembler
		experiment, play around, break things :)
		be patient, it takes time

%font "title-italic", size 3, right
and have some of that black magic! 8)


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page

Questions?

%font "title", size 5
Further reading, references & links...

%font "standard", size 4
These slides will be published on:
		http://www.urbanmyth.org/linux

Intel CPU Documentation
		http://developer.intel.com/design/Pentium4/manuals/

The Linux Kernel Mailing list	http://www.tux.org/lkml

Andrea Arcangeli's IKD and Talk on Kernel Debugging
		ftp://ftp.kernel.org/pub/linux/people/andrea/ikd
		ftp://ftp.suse.com/pub/people/andrea/talks/

Kernel Debuggers
		KDB		http://oss.sgi.com/projects/kdb
		KGDB		http://kgdb.sourceforge.net
		DProbes	(has a URL so long it doesn't fit on a line)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% The END!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%embed "vrts-logo.gif"
M1TE&.#EAOP`T`.<```````D%`!$1"1$1$1$5"0D)"0D-"0`%``D)`#(M&5M5
M.F->0F-:0EM1.DM%,DM!,E-),B$A&3HU(5--,FM>0F-50BHI&8Q^6]_+E>?/
ME=_'E=_'C.?+E7QR4QD=$:V:=(R"6QD9$;ZB=,:J?,ZSA.?/G<ZWA+ZF?+6B
M=(1V4[6>=+ZJ?+6B?%--.IV.8]_+C-;#C#(M(2$=&1D5$3(Q(8QZ6]:_C-_#
MC'QN4SHQ(<Z[A'1F2X1Z6Y6&8ZV::R$=$:62:\:SA$(],L:OA+ZO?$(Y*IV*
M8SHY*JV>=)6"6WQN2]:[C"$E&4(]*F-5.J66=)6*8\:O?'1J2S(I&=:_A'1J
M4U-).GQJ2ZVB=-_/E>?3E1$-";6F?"$A$2HA&4L]*BHE&=_#E<Z[C%-1.FMB
M0I6&6TM!*BHI(6M>2UM-.CHU*F-:.M:[A,:WA!$5$5M50DM),J66:WQJ4VMF
M0IV2:^?+G:6::[6F=(1^6VMF2X1R4S(I(5-%,@D1":V6:\:S?)6"8YV.:X1V
M6Z6.8X1Z4T)!*FMB2W1N2[ZF=*V6=(QZ4QD5"<Z_A$)!,LZ_C!D1$:6.:YV&
M8WQV4R$9$0`)```%"71B2TM%*EM10CHY(1D9&3(U,BHM*D)%0CHY.AD=&3(Q
M,BHI*B$A(4M-2ZVOK;6SM:VJK82&A)V:G962E4)!0H2"A+Z_OG1R=%M>6Z6B
MI8R*C%-54SH].I66E?____?S]];7UN_O[^_K[]_?W];3UO_[_VMN:XR.C.?C
MY\[/SL;'QN?GY\;#QGQ^?$M)2]_;WUM:6U-14\[+SGQZ?/?W]Z6FI6MJ:[6W
MM9V>G;Z[OB$E(6-B8V-F8W1V=/__________________________________
M____________________________________________________________
M____________________________________________________________
M_____________________RP`````OP`T```(_@`!"!Q(L*#!@P@3*ES(L*'#
MAQ`C2IQ(L:)%BP<^F-!AHJ/'CR!#ZH@BX*+)DRA3JERX14>&ES!CRIR9@4..
ME3ASZMRY,`(,FD!G.N!)M*A1E!;04%C*M*G3I0LH-#E*M:K5A@@,%##`M:O7
MKENOBAT[5DW(LQYY'"#+MBW1-4%EGECKMJ[=E%+BOB1"]Z[?OQ-3Q!U2$K#A
MPPP/@%G,F+$%-X@C2SY(0`V-RY<M3-[,>0N1F!R&<AX=68#+EZ))JS8<XD:&
M"GU7R[X;H4/LV;ASZ][-N[?OW\"#"Q].O+AQE`$&""!0H'D!`=!O#T2P7(!6
M`P0*#PQ0W;GW`@@0_A0($4/Z0`/+!WB'3H"Y1.@"FH>OGC`'C1#QQ3?7^MW[
MEA\1.$0#%.8]A``8#E3`0`M'1&!`0A?`I,$&&VC`04T=(/3!A1A4B`$''`0X
M$!879C!AA112>.)+!"2TPH4<:"#C!B""J($)<HC(4`@?FBBC!A]RT,)!`=CP
M$@8S5GCBCTO*F$$0#OGP&D4'3/#33#>T>)`!$;3P@DPNR(``0@A$4`-,-^00
M0D$%=/&%D7IQ\"!"`G0Q@5XUV3'G0N2M$!,&$,A@W@QX#M80&"]!*9$!*,`$
M`Q<FP,0!&`!``($%`DQ1$`&NP70!0QV\M,$,"N4%DQ%C-"#%#DE$:L.8_@M%
M6,)+#X"1`PXRW1&`0P_$%(A"%KQD`A)`!%+#!8)@`),8*5QP00]Q!/'2"0V%
MF@$&:T:4R$LV@('``0?\X,*UI$+@@!0@1&&08#`-L=`!GV70PT(*Q,1`008$
M84*!!.4AJ79@G/:2'@+&Q(-"$(2V*T$(E)C!'04%$$,0U"Z$0!LPS0M1`T?2
M@"\*&$!V!`T+])""00)L`!,&.AYTQ$L:_+`0QS!18%`'*S1$P;\$@:'L2VP4
M#%.&"4F1LT$%.,S%010LO5"O,14`$2*)'E0$!BV"9]V>!'T04Q(*]?`2"@S1
M_-("!N4`@<X\$X3$RI0RM$=,.R@T0\O3.0RQ_D$T$*U0'#)-]9"1)2A:T`\:
M:*=0$S'!P"^<J24$04QH4[3S2QPH#@`#,4EP*-T3-0S3WA(AH(%,1SL$)[8&
M'7!30P=("U,,"-6;013\#@3U2S9;WO9`:L3T^D*(PE1WZ4I39&H4,<GL$",P
MI4Z1$C$A89Z?&1S_-.4575Z3YC1(V,?GQH>>_$32PA`#Z`Z=$-,'L$[DP>F8
M:TG0#,K>D.WV,%4^D?>9*PCC7B(]A13O)=J#B.A>XK2(@.%"/3B`^UZR!*XI
M9`>Y<MY$`A$3/ACD3!DX6$-VEP'_">0`EF$;YC2'P9>LC7P(-!],&@@1(+RD
M$`"PW4L$QQ`#"&Q6_C<X`D6^$),@Q.8`*LN`9D88DSCLX(D\N(#7\J#"[Q4$
M>Q]XR`&S)T,&2H02I[/!@T+@JX?<:28-F$@!(@63EH4O`W%X"`F!HH8J!G`@
M`S2!Y@S(OH@L,`,T=,CD,G`R@30J`R70P/X8@BN9<,!O$/$73*AXPD,608XQ
MB0(*N(""$\#)`U7,@!/V$((N,.!T)L`;\?JHP/,]I!)L])A`2$B&AP1`#S/A
M0`(=TH68F&!.9"R!#N+'$!(R`%S@,L`"[K@0[V&N0C"1I199:2!7.B0"B8K?
M%AQ6P'>1828:$&)$``<3SP%@`2]1`$1(V+N!S(`#4F.(,V$``_J]!`C$_F3(
M%G?ID#\&$E0O"82E!LK&E]01(@ZX$DQ,()'@P:0&`$``G/:H$!*:$``&&$+N
M!@+`$!@@!"U\B10@LL_0_0R0$-EFH30&D1`4E%81,8`];X"`-X)MG=SS7?U.
MB#&8\%"?U.SG2?^YN$)E8)@2*8#`,D"VB(0T`Q`0&P<6$1$SY/1_DK)?RF"R
MA$7RL7S(FR%$_(":KW`%!#%9PT0BD,2:;!1E#D.!D7`7D31<52(`5)S9F/I6
M``0+K!%)FEBO"2,+"H0))^TF08KP0H,00E+Y;,@$8_)3A^RUG05Y*P#M)Q!R
MOH2E"GFC2"=B@)@0%2&2S")"F'<D#1K$"'$\_L@`K[4PB.1`)@R5R%XO*A`!
M^&"CS@0E000@AIA8@2&B#2%I8T*Z=_WA)=(T2"-?`H6$Q($$D07`^H1%$3B]
M!)(/T6$&:FD0/M!T(=9ZB6MG*1/:A38FA%ANNQPR0!LHA`#VW$"!&N7>@CB@
M9A1A5P9@8-B&//6?!S#!'S;Z6)@LL2"XE-"#71:33TED"[YT"`<S4%V%O`VP
M5X3C053P$L=11`"=<@%%(A03%RC@`4UP@"'<!P2&2`DFES0(`L;EJ,@5I`5-
MG,@/8G*#+3!$`O2KK$'VR@&O"J2G0-#1#VQ8DP=81$H<Z.]#+'"$E])D5B,E
MTQD<X+`,^*`(,=#<M!8V?*H'[*$O8)#`(6&F`!I,6"$_B$$29'(!-5C`L"'8
M0QJ*BT!,)20&]GP2'&+`V2&LC`M`0$&)8)!&BQ2"`SYXCR-LL`0=>%H,F[;!
M!FXPZ@VH+R$.L($-Q.!I3ZO:!I)H70-6H(-7J]J]`B`!#%C=ZEJKV@N)^8"J
M>>UJ55.2("X8=JM5#5YD%]O6@B"(&KA`(]#8``AQN\@:A)L2^$`GN\=1B0`*
/088H4B`3X`ZWNDT2$``[
`
%endembed