Testing debuggers

Software programs must be tested before put in use. When there is a single program, things are relatively simple. Running it on multiple platforms is more challenging because it requires testing all of them. But the real nightmare starts when there are multiple programs running on multiple platforms and going to a high abstract level is not an option.
This is the case with IDA debuggers. The current version supports five different variants: Win32/Win64/WinCE/Linux/iMac but the upcoming v5.3 adds Symbian and iPhone to the list.
We can not use a high level language for debuggers because the very nature of the information is low level: bits and bytes, registers and memory cells. The core of the debugger operates with some abstractions but this does not make it really impervious to low level details: each platform has its peculiarities. It is more of a disadvantage than a bonus: changing anything in debugger core could introduce a bug in any of the supported platforms, something hard to reproduce and fix.
Given all the above, we decided to spend some time on a test system. And here is the result: custom made, hardcore command line, user agnostic (or unfriendly?) but it does the job. It generates megabytes of logs and can check all debuggers: local or remote, 32 or 64 bit, single or multithreaded.
Below is a link to a short video. I’m sorry for the window size, it is really difficult to fit all windows into a small area.

One more thing I wanted to tell you: we have the iPhone debugger ready for beta testing. It was a hard job to create it, especially with the broken ptrace on iPhone. I’m still curious to know more about its implementation and limitations…
If you want to participate in beta testing of the iPhone debugger, just send us a message.

8 thoughts on “Testing debuggers”

  1. “””
    We can not use a high level language for debuggers because the very nature of the information is low level: bits and bytes, registers and memory cells.
    You haven’t seen Pai Mei / Pydbg? It’s just a case of using a lower level language to build a debugger API, while abstracting an interface on top.

  2. Pai Mei is an excellent framework and provides the user with a high level API but it itself is built upon low level things. When testing the debugger, one of the things we want the test is that the low level stuff works correctly.
    Second, we consider the case of testing a software running on multiple platforms. Maybe debugger APIs can be abstracted to cover different platforms without leaking low level details (I personally doubt it) but this is something to be done. Exceptions, program start and stop, threads and other stuff work quite differently on different platforms. If we want to ensure that the basic logic works correctly on all platforms, again we have to take into account all these details.
    That’s why I mentioned that going high level is not a option when testing the debugger.
    Debuggers should provide a high level API (IDA Pro has it too).

  3. Now that you are testing the debugger, I suggest that you’d better test some big program, eg, the skype and some binaries larger than 1MB, in my experience, Idapro debugger can not deal with some larger programs, it appears to be dead sometimes and not respond in 20 minutes or so.
    This may because there’re dead locks in relocation processing routine or who knows, but because I had no time to investigate it, so I switched to Ollydbg/Windbg.I think if you test with these big programs, mybe you can re-find that.

  4. Hi Ilfak,
    Just one question How can I become your student, It’ll be an honour to learn from you. I have some experience but I need more and more and more and more ………….
    I am sorry I am doing this on the blog, but I didnt have any other option.

  5. Thank you for your confidence but I’m not into tutoring, sorry about that.

  6. Hi Ilfak,
    I just read your post on the support for the iPhone for IDA and I think it would be really useful. I have been writing quite a bit of stuff of for the iPhone recently. If your offer of helping to test the debugger is open – just send me a mail.

  7. Hi Steve,
    Thank you for your interest in the debugger! Please contact me with your active ida.key and I’ll be glad to send you the beta version. Thanks.

Comments are closed.