Monday, November 26, 2012

A Simple Lab Guide of Using GDB in Embedded Linux System

A Simple Lab Guide of Using GDB in Embedded Linux System

by cawan (cawan[at]ieee.org or chuiyewleong[at]hotmail.com)

on 27/11/2012

GDB is a useful debugging tool for linux developer as well as linux hacker in
understanding the binary running in the system. So, it is a need for embedded linux
developer to master the skill of GDB too. However, due to the nature of embedded
system which is highly customized in term of processor platform, it is a little bit
tricky in applying the basic GDB skill in embedded platform. First, it is important
to cross compile the version of GDB which is compatible to the embedded platform.
Well, this has been discussed in the previous paper in this blog, "How To Cross
Compile GDB and Run in Embedded System". Second, it is always a necessity to
understand the architecture internal of the RISC processor, such as MIPS, ARM, PPC,
and etc. Besides, for each RISC platform, it is crucial to know the registers,
instruction sets, addressing modes, operation modes, calling conventions, status
control, and much much more in detail. Third, it is also a need to understand how
linux works such as the boot process, file system, runtime, shared library, system
call, and etc. In addition, it is an advantage to know more about the linux kernel.
So, in order use GDB for relevant jobs in proper, it is important to have some level
of fundamental knowledges in the aforementioned topics. On the other hand, the GDB is
also a precious tool to assist in understanding the system internal in further
details. Yes, it is bidirectional. Besides, it is really recommended to perform local
debugging with GDB, instead of remote debugging with GDBServer. Well, the reasons are
from a number of aspects, but we will show in examples in our discussion later. So,
without wasting any more time, let us start our demonstration of GDB in embedded
linux system right now. We are having a not responding process and we need to figure
out the root cause of the problem. Our embedded platform is MIPS.

tango3[gdb]# ./gdb
GNU gdb (GDB) 7.1
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "mips-linux-gnu".
For bug reporting instructions, please see:
.
Setting up the environment for debugging gdb.
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n])
[answered N; input not from terminal]
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n])
[answered N; input not from terminal]
/root/cawan/mips-4.3/bin/gdb-7.1/gdb/.gdbinit:8: Error in sourced command file:
No breakpoint number 0.
(gdb) attach 766
Attaching to process 766
Reading symbols from /bin/busybox...(no debugging symbols found)...done.
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/ld.so.1
0x2ac050a8 in nanosleep () from /lib/libc.so.6
(gdb)

Well, the pid of the not responding process is 766 and we attach into it now. From
the message output, it seems the program counter is stucked at nanosleep() in
/lib/libc.so.6. Let's do a backtrace.

(gdb) bt
#0  0x2ac050a8 in nanosleep () from /lib/libc.so.6
#1  0x2ac04d8c in sleep () from /lib/libc.so.6
#2  0x0045b2bc in ?? ()

Yes, the nanosleep() is being called by sleep(). How to know how long it will sleep ?
We know that sleep() will accept a single parameter in the unit of second. But, what
is the situation when the sleep() calls nanosleep() ? Let us check the function
prototype of nanosleep().

int nanosleep(const struct timespec *req, struct timespec *rem);

struct timespec {
    time_t tv_sec;        /* seconds */
    long   tv_nsec;       /* nanoseconds */
};

Well, the nanosleep() accepts 2 parameters in the format of timespec structure. Now,
we need to know a little bit about the calling convention in MIPS environment. In
general, the first 4 paramenters will be stored in register $a0 to $a3, and the rest
will be in stack. So, while nanosleep() accepts 2 parameters, then they are should be
in $a0 and $a1. Let us check now.

(gdb) info reg
          zero       at       v0       v1       a0       a1       a2       a3
 R0   00000000 fffffff8 00000204 00000000 7fb737f8 7fb737f8 00000000 00000001
            t0       t1       t2       t3       t4       t5       t6       t7
 R8   00000031 2ab67540 00000000 7fb73878 ffffffff 7fb737d0 2aab1b84 00000000
            s0       s1       s2       s3       s4       s5       s6       s7
 R16  000003e8 00020000 00000000 7fb73800 7fb737f8 7fb73880 80000000 7fffffff
            t8       t9       k0       k1       gp       sp       s8       ra
 R24  000000be 2ac05084 00000000 00000000 2acc9950 7fb737e0 2ac05084 2ac04d8c
        status       lo       hi badvaddr    cause       pc
      00001c13 0002af19 000001e5 2ac9a570 50808020 2ac050a8
          fcsr      fir  restart
      00000000 01739300 00000204

Fine, both $a0 and $a1 share a single parameter, 7fb737f8. So, we should dump the
address and it should be the timespec structure. Let's verify.

(gdb) x/16x $a0
0x7fb737f8:     0x000003d3      0x344b26ae      0x00020000      0x00000000
0x7fb73808:     0x00000000      0x00000000      0x00000000      0x00000000
0x7fb73818:     0x00000000      0x00000000      0x00000000      0x00000000
0x7fb73828:     0x00000000      0x00000000      0x00000000      0x00000000
(gdb) x/16d $a0
0x7fb737f8:     979     877340334       131072  0
0x7fb73808:     0       0       0       0
0x7fb73818:     0       0       0       0
0x7fb73828:     0       0       0       0

Yes, it seems the not responding process will sleep for 979 seconds or 877340334 in
nano seconds. However, in fact, the not responding process is created by a simple
command line.

tango3[gdb]# sleep 1000 &
766

So, we suppose to get 1000 seconds in timespec structure, but why we get 979 seconds
now ? Let's check again.

(gdb) quit
A debugging session is active.

        Inferior 1 [process 766] will be detached.

Quit anyway? (y or n) y
Detaching from program: /bin/busybox, process 766

tango3[gdb]# ps
PID   USER     TIME   COMMAND
    1 root       0:01 init      
    2 root       0:00 [kthreadd]
    3 root       0:00 [ksoftirqd/0]
    4 root       0:00 [events/0]
    5 root       0:00 [khelper]
   41 root       0:00 [kblockd/0]
   42 root       0:00 [ata/0]
   43 root       0:00 [ata_aux]
   46 root       0:00 [khubd]
   48 root       0:00 [kseriod]
   49 root       0:00 [kmmcd]
   61 root       0:00 [pdflush]
   62 root       0:00 [pdflush]
   63 root       0:00 [kswapd0]
   64 root       0:00 [kprefetchd]
   65 root       0:00 [aio/0]
  658 root       0:00 [scsi_eh_0]
  662 root       0:00 [scsi_eh_1]
  699 root       0:00 [sigmblockd]
  717 root       0:00 [sigm_write_cach]
  724 root       0:00 /sbin/hotplug2 --no-coldplug --persistent --override --ma
  731 root       0:00 /sbin/watchdog -t 20 -T 30 /dev/watchdog
  738 root       0:00 -sh
  739 root       0:00 /sbin/syslogd -n -m 0
  740 root       0:00 /sbin/klogd -n
  751 root       0:00 [rpciod/0]
  773 root       0:00 ps
tango3[gdb]#

Well, it seems the sleep timer has been expired and the process is returned. We
should take an important note here, unlike debuggers in windows environment, while
GDB is detached from a process, the process will not be closed. Let us verify now.

tango3[gdb]# sleep 1000 &
774
tango3[gdb]# ps
PID   USER     TIME   COMMAND
    1 root       0:01 init      
    2 root       0:00 [kthreadd]
    3 root       0:00 [ksoftirqd/0]
    4 root       0:00 [events/0]
    5 root       0:00 [khelper]
   41 root       0:00 [kblockd/0]
   42 root       0:00 [ata/0]
   43 root       0:00 [ata_aux]
   46 root       0:00 [khubd]
   48 root       0:00 [kseriod]
   49 root       0:00 [kmmcd]
   61 root       0:00 [pdflush]
   62 root       0:00 [pdflush]
   63 root       0:00 [kswapd0]
   64 root       0:00 [kprefetchd]
   65 root       0:00 [aio/0]
  658 root       0:00 [scsi_eh_0]
  662 root       0:00 [scsi_eh_1]
  699 root       0:00 [sigmblockd]
  717 root       0:00 [sigm_write_cach]
  724 root       0:00 /sbin/hotplug2 --no-coldplug --persistent --override --ma
  731 root       0:00 /sbin/watchdog -t 20 -T 30 /dev/watchdog
  738 root       0:00 -sh
  739 root       0:00 /sbin/syslogd -n -m 0
  740 root       0:00 /sbin/klogd -n
  751 root       0:00 [rpciod/0]
  774 root       0:00 sleep 1000
  786 root       0:00 ps
tango3[gdb]# ./gdb
GNU gdb (GDB) 7.1
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "mips-linux-gnu".
For bug reporting instructions, please see:
.
Setting up the environment for debugging gdb.
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n])
[answered N; input not from terminal]
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n])
[answered N; input not from terminal]
/root/cawan/mips-4.3/bin/gdb-7.1/gdb/.gdbinit:8: Error in sourced command file:
No breakpoint number 0.
(gdb) attach 774
Attaching to process 774
Reading symbols from /bin/busybox...(no debugging symbols found)...done.
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/ld.so.1
0x2ac050a8 in nanosleep () from /lib/libc.so.6
(gdb) detach
Detaching from program: /bin/busybox, process 774
(gdb) quit
tango3[gdb]# ps
PID   USER     TIME   COMMAND
    1 root       0:01 init      
    2 root       0:00 [kthreadd]
    3 root       0:00 [ksoftirqd/0]
    4 root       0:00 [events/0]
    5 root       0:00 [khelper]
   41 root       0:00 [kblockd/0]
   42 root       0:00 [ata/0]
   43 root       0:00 [ata_aux]
   46 root       0:00 [khubd]
   48 root       0:00 [kseriod]
   49 root       0:00 [kmmcd]
   61 root       0:00 [pdflush]
   62 root       0:00 [pdflush]
   63 root       0:00 [kswapd0]
   64 root       0:00 [kprefetchd]
   65 root       0:00 [aio/0]
  658 root       0:00 [scsi_eh_0]
  662 root       0:00 [scsi_eh_1]
  699 root       0:00 [sigmblockd]
  717 root       0:00 [sigm_write_cach]
  724 root       0:00 /sbin/hotplug2 --no-coldplug --persistent --override --ma
  731 root       0:00 /sbin/watchdog -t 20 -T 30 /dev/watchdog
  738 root       0:00 -sh
  739 root       0:00 /sbin/syslogd -n -m 0
  740 root       0:00 /sbin/klogd -n
  751 root       0:00 [rpciod/0]
  774 root       0:00 sleep 1000
  791 root       0:00 ps
tango3[gdb]#

Well, the process being attached will not exit when it is detached from GDB. So,
by attach and detach the sleeping process for several times, we should realize
something meaningful.

tango3[gdb]# sleep 1000 &
795
tango3[gdb]# ./gdb
GNU gdb (GDB) 7.1
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "mips-linux-gnu".
For bug reporting instructions, please see:
.
Setting up the environment for debugging gdb.
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n])
[answered N; input not from terminal]
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n])
[answered N; input not from terminal]
/root/cawan/mips-4.3/bin/gdb-7.1/gdb/.gdbinit:8: Error in sourced command file:
No breakpoint number 0.
(gdb) attach 795
Attaching to process 795
Reading symbols from /bin/busybox...(no debugging symbols found)...done.
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/ld.so.1
0x2ac050a8 in nanosleep () from /lib/libc.so.6
(gdb) x/16d $a0
0x7fc837f8:     985     817127037       131072  0
0x7fc83808:     0       0       0       0
0x7fc83818:     0       0       0       0
0x7fc83828:     0       0       0       0
(gdb) detach
Detaching from program: /bin/busybox, process 795
(gdb) attach 795
Attaching to program: /bin/busybox, process 795
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/ld.so.1
0x2ac050a8 in nanosleep () from /lib/libc.so.6
(gdb) x/16d $a0
0x7fc837f8:     942     6108777 131072  0
0x7fc83808:     0       0       0       0
0x7fc83818:     0       0       0       0
0x7fc83828:     0       0       0       0
(gdb) detach
Detaching from program: /bin/busybox, process 795
(gdb) attach 795
Attaching to program: /bin/busybox, process 795
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/ld.so.1
0x2ac050a8 in nanosleep () from /lib/libc.so.6
(gdb) x/16d $a0
0x7fc837f8:     911     935667926       131072  0
0x7fc83808:     0       0       0       0
0x7fc83818:     0       0       0       0
0x7fc83828:     0       0       0       0
(gdb)

Nice, it seems the values in timespec structure keep deducting even though the process
itself is being debugged. So, we can say that once the counter in timespec reach to 0,
the process will automatically exit once we detach the process from GDB. Let's verify.

(gdb) detach   
Detaching from program: /bin/busybox, process 795
(gdb) attach 795
Attaching to program: /bin/busybox, process 795
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/ld.so.1
0x2ac050a8 in nanosleep () from /lib/libc.so.6
(gdb) x/16d $a0
0x7fc837f8:     0       652534296       131072  0
0x7fc83808:     0       0       0       0
0x7fc83818:     0       0       0       0
0x7fc83828:     0       0       0       0
(gdb) detach   
Detaching from program: /bin/busybox, process 795
(gdb) attach 795
Attaching to program: /bin/busybox, process 795
ptrace: No such process.
(gdb)

Yes, we are in the right way. Now, let's have a look to source level debugging with
GDB. Before that, let us create a simple cawan.c first.

Host:# cat cawan.c
#include
main()
{
  int flag=0;
  char *string="cawan";
  if(flag==0)
    printf("hello world %s\n",string);
  else
    printf("You should not see this !\n");
}
Host:# mips-linux-gnu-gcc -EL -g -o cawan cawan.c
Host:# ls cawan*
cawan  cawan.c
Host:#

Well, we have cross compiled the cawan.c with debugging symbols, and the executable
is cawan. Let us run it in MIPS environment with GDB.

tango3[gdb]# ./gdb       
GNU gdb (GDB) 7.1
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "mips-linux-gnu".
For bug reporting instructions, please see:
.
Setting up the environment for debugging gdb.
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]
/root/cawan/mips-4.3/bin/gdb-7.1/gdb/.gdbinit:8: Error in sourced command file:
No breakpoint number 0.
(gdb) file cawan
Reading symbols from /root/cawan/mips-4.3/bin/gdb-7.1/gdb/cawan...done.
(gdb) list -
During symbol reading, DW_AT_name missing from DW_TAG_base_type.
(gdb) list 
1       #include
2       main()
3       {
4         int flag=0;
5         char *string="cawan";
6         if(flag==0)
7           printf("hello world %s\n",string);
8         else
9           printf("You should not see this !\n");
10      }
(gdb) run
Starting program: /root/cawan/mips-4.3/bin/gdb-7.1/gdb/cawan
hello world cawan

Program exited with code 022.
(gdb)

Good, the executable runs as usual. Let us set a breakpoint at line 7.

(gdb) b 7
Breakpoint 1 at 0x40068c: file cawan.c, line 7.
(gdb) run
Starting program: /root/cawan/mips-4.3/bin/gdb-7.1/gdb/cawan

Breakpoint 1, main () at cawan.c:7
7           printf("hello world %s\n",string);
(gdb) p string
During symbol reading, incomplete CFI data; unspecified registers
(e.g., zero) at 0x400670.
$1 = 0x400850 "cawan"
(gdb) n
hello world cawan
10      }
(gdb)

Nice, once the breakpoint be hit, we can check the value of "string", and it is
definitely equal to "cawan" as what we have defined in proper. Now, we make another
breakpoint at line 6, and run the executable again.

(gdb) b 6    
Breakpoint 2 at 0x400680: file cawan.c, line 6.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y

Starting program: /root/cawan/mips-4.3/bin/gdb-7.1/gdb/cawan

Breakpoint 2, main () at cawan.c:6
6         if(flag==0)
(gdb) p flag
During symbol reading, incomplete CFI data; unspecified registers
(e.g., zero) at 0x400670.
$3 = 0
(gdb) set variable flag = 1
(gdb) n
9           printf("You should not see this !\n");
(gdb) n
You should not see this !
10      }
(gdb)

Excellent, we can simply change the value of "flag" and let the executable runs in
different path, which is not the normal track. Well, that's all about our discussion
of GDB in performing local debugging. Then how about remote debugging ? Let's try and
we expect it works as usual.

tango3[gdb]# ./gdbserver/gdbserver :8888 ./cawan
Process ./cawan created; pid = 756
Listening on port 8888

Well, the gdbserver should be running at the MIPS platform now. Let's connect it from
the host machine with mips-linux-gnu-gdb.

Host:# mips-linux-gnu-gdb ./cawan
GNU gdb (Sourcery G++ Lite 4.3-51) 6.8.50.20080821-cvs
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=mips-linux-gnu".
For bug reporting instructions, please see:
...
(gdb) target remote 192.168.10.2:8888
Remote debugging using 192.168.10.2:8888
warning: Architecture rejected target-supplied description
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
0x2aaa8810 in ?? ()
(gdb) list
1    #include
2    main()
3    {
4      int flag=0;
5      char *string="cawan";
6      if(flag==0)
7        printf("hello world %s\n",string);
8      else
9        printf("You should not see this !\n");
10    }
(gdb) c
Continuing.
warning: .dynamic section for "/lib/libc.so.6" is not at the expected address
(wrong library or version mismatch?)
Error while mapping shared library sections:
/lib/ld.so.1: No such file or directory.

Program exited with code 022.
(gdb)

Nice, it seems the executable has been run in MIPS environment. Let's check the
MIPS console.

Remote debugging from host 192.168.10.1
hello world cawan

Child exited with status 18
GDBserver exiting

tango3[gdbserver]#

Good, the executable has been run once. In order to allow the target executable to
restart without restarting the GDBServer, we should use "target extended-remote"
instead of "target remote". So, let us do the same thing as what we did in local
debugging section. First, run the GDBServer at MIPS environment.

tango3[gdbserver]# ./gdbserver :8888 ./cawan
Process ./cawan created; pid = 762
Listening on port 8888

Then, at host machine, connect to MIPS environment with mips-linux-gnu-gdb.

Host:# mips-linux-gnu-gdb ./cawan
GNU gdb (Sourcery G++ Lite 4.3-51) 6.8.50.20080821-cvs
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=mips-linux-gnu".
For bug reporting instructions, please see:
...
(gdb) target extended-remote 192.168.10.2:8888
Remote debugging using 192.168.10.2:8888
warning: Architecture rejected target-supplied description
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
0x2aaa8810 in ?? ()
(gdb) list
1    #include
2    main()
3    {
4      int flag=0;
5      char *string="cawan";
6      if(flag==0)
7        printf("hello world %s\n",string);
8      else
9        printf("You should not see this !\n");
10    }
(gdb) b 7
Breakpoint 1 at 0x40068c: file cawan.c, line 7.
(gdb) c
Continuing.

Program received signal SIGINT, Interrupt.
0x2aaa8810 in ?? ()
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y

Starting program: /home/smp383/mips-4.3/bin/testgdb/gdb-7.0/gdb/gdbserver/cawan
warning: Architecture rejected target-supplied description
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
warning: .dynamic section for "/lib/libc.so.6" is not at the expected address
(wrong library or version mismatch?)
Error while mapping shared library sections:
/lib/ld.so.1: No such file or directory.
[Switching to Thread 763]

Breakpoint 1, main () at cawan.c:7
7        printf("hello world %s\n",string);
(gdb) p string
$1 = 0x400850 "cawan"
(gdb) b 6
Breakpoint 2 at 0x400680: file cawan.c, line 6.
(gdb) c
Continuing.

Program exited with code 022.
(gdb) run
Starting program: /home/smp383/mips-4.3/bin/testgdb/gdb-7.0/gdb/gdbserver/cawan
warning: Architecture rejected target-supplied description
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
warning: .dynamic section for "/lib/libc.so.6" is not at the expected address
(wrong library or version mismatch?)
Error while mapping shared library sections:
/lib/ld.so.1: No such file or directory.
[Switching to Thread 764]

Breakpoint 2, main () at cawan.c:6
6      if(flag==0)
(gdb) p flag
$2 = 0
(gdb) set variable flag = 1
(gdb) n
9        printf("You should not see this !\n");
(gdb) n
10    }
(gdb)

Let's verify the message output at MIPS console.

Remote debugging from host 192.168.10.1
Killing all inferiors
Process ./cawan created; pid = 763
hello world cawan

Child exited with status 18
Process ./cawan created; pid = 764
You should not see this !

Well, it seems we can do the same thing as in local debugging mode. However, if we
do something unknown by the GDBServer, the GDB at the host machine will get into
infinity loop... and there are quite a number of such booby traps. Let's show an
example when we try to step into printf().

(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y

Starting program: /home/smp383/mips-4.3/bin/testgdb/gdb-7.0/gdb/gdbserver/cawan
warning: Architecture rejected target-supplied description
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
warning: .dynamic section for "/lib/libc.so.6" is not at the expected address
(wrong library or version mismatch?)
Error while mapping shared library sections:
/lib/ld.so.1: No such file or directory.
[Switching to Thread 765]

Breakpoint 2, main () at cawan.c:6
6      if(flag==0)
(gdb) n

Breakpoint 1, main () at cawan.c:7
7        printf("hello world %s\n",string);
(gdb) s
warning: GDB can't find the start of the function at 0x2aabe06c.

    GDB is unable to find the start of the function at 0x2aabe06c
and thus can't determine the size of that function's stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
    This problem is most likely caused by an invalid program counter or
stack pointer.
    However, if you think GDB should simply search farther back
from 0x2aabe06c for code which looks like the beginning of a
function, you can increase the range of the search using the `set
heuristic-fence-post' command.
warning: GDB can't find the start of the function at 0x2aabe06b.
warning: GDB can't find the start of the function at 0x2aabe06c.
warning: GDB can't find the start of the function at 0x2aabe06b.
warning: GDB can't find the start of the function at 0x2aabe06c.
warning: GDB can't find the start of the function at 0x2aabe06b.
warning: GDB can't find the start of the function at 0x2aabe06c.
warning: GDB can't find the start of the function at 0x2aabe06b.
warning: GDB can't find the start of the function at 0x2aabe06c.
warning: GDB can't find the start of the function at 0x2aabe06b.
warning: GDB can't find the start of the function at 0x2aabe06c.
warning: GDB can't find the start of the function at 0x2aabe06b.
warning: GDB can't find the start of the function at 0x2aabe06c.
warning: GDB can't find the start of the function at 0x2aabe06b.
warning: GDB can't find the start of the function at 0x2aabe06c.

Yes, it is quite annoying whenever we are concentrating to troubleshoot our binary,
or designing our shellcode for embedded linux system. So, again, it is really
recommended to use local debugging method, instead of the remote debugging, under
the embedded platform. Besides, it is important to note that the version of cross
compiler being used should match to the dedicated version of GDB. For our example
here, the cross compiler that we are using is CodeSourcery v4.3, and it should work
together with GDB v7.0. Otherwise, if we use CodeSourcery v4.3 to compile GDB v7.1,
we will get the error as similar to this,

tango3[gdbserver]# ./gdbserver :8888 ./cawan
Process ./cawan created; pid = 757
Listening on port 8888

Host:# mips-linux-gnu-gdb ./cawan
GNU gdb (Sourcery G++ Lite 4.3-51) 6.8.50.20080821-cvs
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=mips-linux-gnu".
For bug reporting instructions, please see:
...
(gdb) target remote 192.168.10.2:8888
Remote debugging using 192.168.10.2:8888
warning: Architecture rejected target-supplied description
Malformed packet(b) (missing colon): ore:0;
Packet: 'T051d:30bba07f;25:1088aa2a;thread:2f5;core:0;'

(gdb)

So, the latest version of a tool doesn't mean it will always work in embedded system.
Instead, the matching of the version number among each others should always be alert.


pdf version:

http://www.scribd.com/doc/114458156/A-Simple-Lab-Guide-of-Using-GDB-in-Embedded-System

No comments:

Post a Comment