If you were using import to import your own “currently-in-development” modules from your IDAPython scripts, you may want to use idaapi.require(), starting with IDA 6.5.

Rationale

When using IDAPython scripts, users were sometimes facing the following issue

Specifically:

• User loads script
• Script imports user’s module mymodule
• Script ends
• User modifies code of mymodule (Note: the module is modified, not the script)
• User reloads script
• Modifications to mymodule aren’t taken into consideration.

While that’s perfectly understandable (the python runtime doesn’t have to reload mymodule if it has been compiled & loaded already), this is somewhat of an annoyance for users that were importing modules that were often modified.

IDA <= 6.4: Ensuring a user-specified module gets reloaded, by destroying it.

Up until IDA 6.4, the IDAPython plugin would do some magic after you have run your user script.
(click “expand all” to reveal the diff)

The sequence becomes:

• User loads script
• Script imports user’s module mymodule
• Script ends
• [module mymodule is deleted]
• User modifies code of mymodule
• User reloads script
• Modifications to mymodule are taken into consideration, since module was deleted.

Unfortunately we have to stop doing this because:

• That prevents us from using python-based hooks to be used after the script is finished (see below).
• That goes against the rest of the python philosophy (i.e., modifications to objects are not reverted), and is therefore unexpected.

Issues with hooks.

Imagine you have the following script, dbghooks.py:

from idaapi import *
import mydbghelpers

class MyHooks(DBG_Hooks):

  def __init__(self):
    ...

  def dbg_bpt(self, tid, ea):
    mydbghelpers.do_something()
    return 0

  def dbg_step_into(self):
    ...

hooks = MyHooks()
hooks.hook()

• User loads script
• Scripts imports mydbghelpers
• Script creates instance of MyHooks, and hooks it into IDA’s debugger APIs
• Script ends
• [module mydbghelpers is deleted]
• User runs debugger, and a breakpoint is hit. Two things can happen:
  • The hook fails executing
  • IDA crashes (that can happen if the form from mydbghelpers import * was used)

IDA > 6.4: Introducing idaapi.require()

Everywhere else in python, when you modify a runtime object, those changes will remain visible.

We decided it would be better to not go against that standard behaviour anymore, and provide a helper to achieve the same results as what was achieved before with the deletion of user modules.

You can now import & re-import of a module with: idaapi.require(name)

Here is its definition:

def require(modulename):
    if modulename in sys.modules.keys():
        reload(sys.modules[modulename])
    else:
        import importlib
        import inspect
        m = importlib.import_module(modulename)
        frame_obj, filename, line_number, function_name, lines, index = inspect.stack()[1]
        importer_module = inspect.getmodule(frame_obj)
        if importer_module is None: # No importer module; called from command line
            importer_module = sys.modules['__main__']
        setattr(importer_module, modulename, m)
        sys.modules[modulename] = m

Example

The example debugger hooks script above becomes:

from idaapi import *
idaapi.require("mydbghelpers")

class MyHooks(DBG_Hooks):

  def __init__(self):
    ...

  def dbg_bpt(self, tid, ea):
    mydbghelpers.do_something()
    return 0

  def dbg_step_into(self):
    ...

hooks = MyHooks()
hooks.hook()

I.e., only the second line changes.

The issue he was facing is that his system is a 64-bit Ubuntu 12.04 VM.
Therefore using the Ubuntu-bundled PIP will just result in installing the desired package (let’s say ssdeep) for the system Python runtime, which is a 64-bit runtime and therefore not compatible with IDA.

The best (as in: cleanest) solution I have found is to:

• build a 32-bits python on the system.
• pip-install packages in that 32-bits python’s sub-directories.
• export PYTHONPATH to point to the 32-bits python’s sub-directories.

We figured we’d write it down here just in case it might help others.

Prerequisites

• Install autoconf
• Install ia32-libs

Building & installing a 32-bits python

• ..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH
• Download Python2.7.4
  • Note:You should make sure that the MD5 checksum and the size of the file you downloaded match those that are advertised on the page. That would prevent a man-in-the-middle attacker from providing you with a malicious Python bundle.
• Build it. Note that you’ll probably have to sudo-create a few symlinks. I had to do this, on the Ubuntu 12.04 64-bit VM I tested this on:
  • /lib/i386-linux-gnu/libssl.so ⇒ /lib/i386-linux-gnu/libssl.so.1.0.0
  • /lib/i386-linux-gnu/libcrypto.so ⇒ /lib/i386-linux-gnu/libcrypto.so.1.0.0
  • /lib/i386-linux-gnu/libz.so ⇒ /lib/i386-linux-gnu/libz.so.1
• For the sake of completeness, here are my build commands (don’t forget the flags, of course):

  • ..$ CFLAGS=-m32 LDFLAGS=-m32 ./configure --prefix=/opt/Python2.7.4-32bits

  • ..$ CFLAGS=-m32 LDFLAGS=-m32 make -j 8

Once the build completes

Here’s what I have as last lines of the build:

INFO: Can't locate Tcl/Tk libs and/or headers

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _curses            _curses_panel
_sqlite3           _tkinter           bsddb185
bz2                dbm                gdbm
readline           sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

If you see, below that, that it failed to build, say 'binascii', then something went wrong.

Make sure you run make -j 1 to check out what went wrong (i.e., what library it claims not being able to find)

Once you have succesfully built your 32-bits Python, it’s time to install it: sudo make install

Trying your freshly-built python

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii
>>>

No complaint so far. Good.

Checking that pkg_resources is available.

Try importing pkg_resources. If it fails, you’ll probably have to do the following:

..$ cd /tmp
..$ curl -O http://python-distribute.org/distribute_setup.py
..$ less distribute_setup.py  # (*)
..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 distribute_setup.py

That will print out quite a fair amount of info, and should succeed.

(*) Note: A careful reader has pointed out that it would be fairly easy to intercept (man-in-the-middle) such an HTTP request, and serve malicious content that would then be piped (as root) to Python.
That’s why I think it’s important to mention, as a third step (i.e., less ...), that the code that was downloaded should ideally be checked. Hopefully, http://python-distribute.org will soon provide HTTPS support, which will limit such MITM attack risks.

Trying your freshly-built python, again

We want to make sure pkg_resources can be imported.

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>>

Still no complaint. Good.

If yours complains, you’ll have to first make sure you fix whatever is causing it to fail, because the next will not work without that.

Installing PIP for your new Python build

Since using your system’s PIP will probably not work (as it would build & install things in a 64-bits python sub-directory), you’ll have to install a PIP package specifically for your freshly-built Python.

Here’s how I proceeded:

..$ cd /tmp;
..$ curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py;
..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 get-pip.py

PIP is now installed.

PIP-installing a package (i.e., ssdeep)

To download/build/install the ssdeep package I ran, as root (either that, or you’ll have to give your user the rights to write in /opt/Python2.7.4-32bits):

..$ su
Password:
root ..$ export CFLAGS=-m32
root ..$ export LDFLAGS=-m32
root ..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH
root ..$ /opt/Python2.7.4-32bits/bin/python2.7 /opt/Python2.7.4-32bits/bin/pip install ssdeep

Notice how I use my freshly-built python, with my fresly-installed PIP (and not the system one.)

Note: Don’t forget the export lines, or PIP will partially build stuff for x64, and partially for x86. That, as you can guess, won’t quite work.

If you forgot the export lines and started building anyway (and the build failed because of the mixed architecture issue I just wrote about), make sure you delete whatever is in /tmp/pip-build-*, so that there won’t be stale object files of inappropriate architecture in there.

Check out the PIP-installed package works

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssdeep
>>> ssdeep
<module 'ssdeep' from '/opt/Python2.7.4-32bits/lib/python2.7/site-packages/ssdeep.so'>
>>> dir(ssdeep)
['Error', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__test__', '__version__', 'compare', 'hash', 'hash_from_file', 'sys']
>>>

So far so good.

Testing the PIP-installed package in IDA

Since that’s still the goal (though you might have forgotten by now, given the amount of directions above.. ),
we’ll now try and make use of that PIP-installed package in IDA.

• ..$ export PYTHONPATH=/opt/Python2.7.4-32bits/lib/python2.7/site-packages:/opt/Python2.7.4-32bits/:$PYTHONPATH
• ..$ idaq

If all went well, typing import ssdeep in the Python input line should properly, silently, nicely import the package.

The videos are not up yet but here are the slides of my presentation and a few demo scripts I made for it to parse GCC’s RTTI structures and exception tables. I also added my old scripts from OpenRCE which I amended slightly for the current IDA versions (mostly changed hotkeys).

Slides
Scripts

The IDA APIs are provided by the kernel dynamic library. In Windows, it’s called ida.wll (or ida64.wll), in Linux libida[64].so and on OS X libida[64].dylib. ctypes provides a nice feature that dynamically creates a callable wrapper for a DLL export by treating it as an attribute of a special class instance. Here’s how to get that instance under the three platforms supported by IDA:

import ctypes
idaname = "ida64" if __EA64__ else "ida"
if sys.platform == "win32":
    dll = ctypes.windll[idaname + ".wll"]
elif sys.platform == "linux2":
    dll = ctypes.cdll["lib" + idaname + ".so"]
elif sys.platform == "darwin":
    dll = ctypes.cdll["lib" + idaname + ".dylib"]

We use “windll” because IDA APIs use stdcall calling convention on Windows (check the definition of idaapi in pro.h).

Now we just need to call our function just as if it was an attribute of the “dll” object. But first we need to prepare the arguments. Here’s the declaration from loader.hpp:

idaman ssize_t ida_export get_loader_name(char *buf, size_t bufsize);

ctypes provides a convenience functions for creating character buffers:

buf = ctypes.create_string_buffer(256)

And now we can call the function:

dll.get_loader_name(buf, 256)

To retrieve the contents of the buffer as a Python byte string, just use its .raw attribute. The complete script now looks like this:

import ctypes
idaname = "ida64" if __EA64__ else "ida"
if sys.platform == "win32":
    dll = ctypes.windll[idaname + ".wll"]
elif sys.platform == "linux2":
    dll = ctypes.cdll["lib" + idaname + ".so"]
elif sys.platform == "darwin":
    dll = ctypes.cdll["lib" + idaname + ".dylib"]
buf = ctypes.create_string_buffer(256)
dll.get_loader_name(buf, 256)
print "loader:", buf.raw

ctypes offers many means to interface with C code, so you can use it to call almost any IDA API.

When we are re-playing a recorded trace, we can step forward and backward, set breakpoints, inspect register values, change the instruction pointer to any recorded IP, etc…

Also, trace management capabilities have been added to IDA in order to allow saving and loading recorded execution traces. Let’s see an example.

A vulnerable sample program

For this blog post, I will show you how this plugin can be used to analyze a bug in a toy executable program. This sample application receives 2 arguments: a message to display and the size of it. The program checks if the size of the given buffer (calling strlen) is longer than the size specified, printing out an error message and exiting. If not, memory of the given size is reserved for a local variable, the contents of the buffer copied to it and a message based on this string printed out to stdout. After this, the memory reserved is freed and the application simply exits.

In this application there is a little integer overflow bug that can be triggered giving to the size argument a negative value. Let’s record a trace of the program crashing and replay it in IDA to understand why the program is crashing:

• Set a breakpoint in the entry point.
• Set the program arguments to “whatever -1” in Debugger → Process options.
• Run the application with the correspondent debugger (in my case, the “Local Linux” debugger).
• When the breakpoint is reached, enable instruction tracing (via the menu item “Debugger → Tracing → Instruction tracing”).
• Let the application continue (press F9).

At some point it will crash with a message similar to this:

When the application crashes, stop the debugger, go to the trace window (Debugger → Tracing → Trace window) and save the trace to a file (right click on the window and, from the pop-up menu, select the option Other options → Save binary trace file to disk). Specify the file name and a description for this trace and click OK:

Next, switch to the “Trace replayer debugger” (from the menu Debugger → Switch to debugger, and then select “Trace replayer”). After this, go to the trace window to see where is it crashing and set a breakpoint in the function call that segfaults. In our example, it’s crashing in a call to strcpy in function “foo” as we may see here:

We will set a breakpoint in the call to strcpy in function “foo” and press F9 to replay the trace. When the breakpoint is reached we have all the register values as they were when the program was really executed (click to enlarge):

There is a check at 0×08048502 for the size of the given buffer and the size given in the command line. As -1 is lower than strlen(“AAAA”) the developer didn’t expected the program to reach the basic block where the malloc and strcpy calls are made. There is another bug here: before the call to strcpy there is a call to malloc to reserve memory with the size we gave in the command line, but the developer didn’t performed any check to see if the memory was correctly reserved or not. Let’s step back to this position by selecting from the menu Debugger → Step back. Click this menu item various times until EIP points to 0x804851D. Alternatively, at the IDC prompt in the bottom part of IDA, we may enter the command “StepBack()” and press enter. No matter how we moved to the instruction after the malloc call, we will see the following in IDA:

The program failed to reserve memory as the returned pointer is NULL. Step back until the “call _malloc” instruction to see the size passed to malloc:

The program is trying to reserve 0xFFFFFFFF bytes (4 GB) and it fails to do so. Let’s analyse how the program reached the point where the memory is reserved, the strcpy call is performed, etc… when it wasn’t supposed to do so. Click on the 1^st instruction of the function “foo”, right click and select from the pop-up menu “Set IP” (this way, we are telling the re-player to change IP to the nearest event with this IP). The instruction pointer and all the other register values changes:

Now, step over the instructions until we reach the JBE one (alternatively, we could simply right click in the JBE instruction and select from the pop-up menu “Set IP”). Once moved to this instruction, we will see the following register values:

The result of the call to strlen passing the given buffer (unsigned) is compared against the size value we gave in the command line (which is calculated as atoi(argv[2]), which returns a signed integer). Then, an unsigned comparison (JBE) is performed and the check “0×4 <= 0xFFFFFFFF” passes. Now you can document this bug and continue searching for more or fix it and recompile your application.

We hope you like this new IDA feature!

The code viewer

The first new thing I’m going to talk about is the code viewer. This control is just a container for a generic custom viewer. In fact, I just took the custom viewer sample plugin and added some lines to add the code viewer:

  // create a custom viewer
  si->cv = create_custom_viewer("", NULL, &s1, &s2, &s1, 0, &si->sv);
  // create a code viewer container for the custom viewer
  si->codeview = create_code_viewer(form, si->cv, CDVF_LINEICONS);
  // set handlers for the code viewer
  set_code_viewer_line_handlers(si->codeview, lines_click, NULL, NULL, lines_icon, lines_linenum);
  // draw maximal 2 icons in the lines widget
  set_code_viewer_lines_icon_margin(si->codeview, 2);

After having created the custom viewer with create_custom_viewer, it is possible to create a code viewer by passing the custom viewer as argument to create_code_viewer.

The code viewer basically features for the moment a widget left to the embedded custom viewer called ‘the line widget’. This control can show the current line number, either automatically or by providing it ourselves. It can also show one or more icons and it informs us about user interaction such as mouse clicks.

As you can see, create_code_viewer was called specifying the CDVF_LINEICONS flag, which instructs the code viewer that we want icons to be drawn inside the line widget.

Available flags are:

#define CDVF_NOLINES        0x0001    // don't show line numbers
#define CDVF_LINEICONS      0x0002    // icons can be drawn over the line control
#define CDVF_STATUSBAR      0x0004    // keep the status bar in the custom viewer

However, specifying the flag to show the icons is not enough. We also need to specify how the maximum number of icons we want to display on a single line:

  // draw maximal 2 icons in the lines widget
  set_code_viewer_lines_icon_margin(si->codeview, 2);

Then there’s the callback we have set by calling set_code_viewer_line_handlers:

static int idaapi lines_icon(
        TCustomControl * /*cv*/,
        const place_t *p,
        int *x,
        void * /*ud*/)
{
  bool b = p->touval(NULL) == 6 && *x == 0;
  // setting the highest bit signals that there's another icon 
  // to draw on the current line
  int icon_id = *x == 0 ? 51 : 177;
  return b ? 0x80000000 | icon_id : icon_id;
}

This callback needs some explanation. The line widget calls this callback for every line. The argument x represents the position of the icon to draw. Thus, for every line the callback is called the first time with *x == 0. The return value of the callback is the id of the icon to display. If you want to display another icon for the current line, then set the highest bit in the return value. If you don’t want to display an icon at all for the current line, simply return -1.

You might wonder why x is a pointer. Imagine you don’t want to display an icon at position 0, but skip that position and display an icon at the next one. This can be done without calling the callback again, since we can set the value of x. Remember that you can load custom icons and use them in the code viewer.

Apart from the usual mouse and context menu callbacks, there’s a handy callback to be notified when the user clicks in the space reserved for icons:

static void idaapi lines_click(
        TCustomControl * /*cv*/,
        const place_t *p,
        int pos,
        int /*shift*/,
        void * /*ud*/)
{
  if ( p != NULL )
    msg("Line click at position: %d\n", pos);
  else
    msg("Click occurred outside of a line\n");
}

This is useful to set/unset an icon for instance.

I’ve also created another sample plugin with the code viewer featuring a hex view.

This can give you some more ideas of what this control might good for. Most of the logic behind this sample stands in the custom view and not in the code view, which, in this case, handles everything automatically without the use of callbacks:

  // create a code viewer container for the custom view
  si->hexview = create_code_viewer(form, si->cv);
  // set the radix and alignment for the offsets
  set_code_viewer_lines_radix(si->hexview, 16);
  set_code_viewer_lines_alignment(si->hexview, si->data.size() > 0xFFFFFFFF ? 16 : 8);

New forms controls

Forms were certainly lacking two very important controls: a generic combo box and a multi-line edit. The combo box comes in two variants: editable and read-only. This is how to declare a combo:

<title:b:is_editable:width::>

When is_editable is omitted or zero, then the combo is read-only, otherwise it’s editable. Every combo takes two arguments. The first argument is a qstrvec_t * to populate the combo with items, while the second one is either an int * or a qstring * to specify the current selection. If the combo is read-only, then the second argument is an int * specifying an index into the qstrvec_t list, otherwise it’s a qstring * with the current text of the combo. Getting and setting the value of a combo follows the same int/qstring rule.

A multi-line edit is declared in this way:

<title:t::width::>

This control is more complex than the combo and requires a structure as argument:

// Multi line text control: used to embed a text control in a form
struct textctrl_info_t
{
   size_t  cb;                 // size of this structure
   qstring text;               // in, out: text control value
   uint16  flags;
#define TXTF_AUTOINDENT 0x0001 // auto-indent on new line
#define TXTF_ACCEPTTABS 0x0002 // Tab key inserts 'tabsize' spaces
#define TXTF_READONLY   0x0004 // text cannot be edited (but can be selected and copied)
#define TXTF_SELECTED   0x0008 // shows the field with its text selected
#define TXTF_MODIFIED   0x0010 // gets/sets the modified status
#define TXTF_FIXEDFONT  0x0020 // the control uses IDA's fixed font
   uint16  tabsize;            // how many spaces a single tab will indent
   textctrl_info_t(): cb(sizeof(textctrl_info_t)), flags(0), tabsize(0) { }
};

Most of the fields are self-evident, but two flags require an explanation. TXTF_ACCEPTTABS tells the control that the user can use tabs and they will be converted to the number of spaces specified by tabsize. TXTF_AUTOINDENT enables auto-indentation, not only when return is pressed, but also by removing tab block spaces on backspace. This feature is very useful when you want code to be entered for instance. In fact, the new script dialog in IDA 6.2 was created by using it.

Don’t forget the TXTF_FIXEDFONT flag to use a fixed font. Here’s a small sample demonstrating both the combo and edit:

//--------------------------------------------------------------------------
static int idaapi modcb(int fid, form_actions_t &fa)
{
  switch ( fid )
  {
    case 10:
      {
        msg("The selection of the combo has changed!\n");
        // set the text of edit to the text of the current combo item
        int sel;
        if ( fa.get_field_value(10, &sel) )
        {
          qstrvec_t *list = (qstrvec_t *)fa.get_ud();
          textctrl_info_t ti;
          ti.flags = TXTF_SELECTED;
          ti.text = list->at(sel);
          fa.set_field_value(11, &ti);
        }
      }
      break;
 
    case 11:
      msg("The multi-line edit text has changed!\n");
      break;
  }
 
  return 1;
}
 
//--------------------------------------------------------------------------
static void idaapi run(int)
{
  static const char form[] =
    "Combo - Multi-line Edit sample\n"
    "%/%*"
    " <Combo:b10::40::>\n"
    " <Text:t11::40::>\n"
    "\n";
  qstrvec_t list;
  static const char *items[] =
  { "first", "second", "third", "fourth" };
  for ( int i = 0; i < qnumber(items); i++ )
    list.push_back(items[i]);
  int sel = 1;
  textctrl_info_t ti;
  ti.flags = TXTF_SELECTED;
  ti.text = "Some text";
  if ( AskUsingForm_c(form, modcb, &list, &list, &sel, &ti) > 0 )
  {
    msg("Combo selection: %s\n", list[sel]);
    msg("Text: %s\n", ti.text.c_str());
  }
}

In this small sample you can see another small addition to the forms syntax: following the usual %/ to specify the callback of the form there’s a %*. This new sequence makes it possible to specify a user data pointer (&list), which can later on be retrieved from the callback like this:

qstrvec_t *list = (qstrvec_t *)fa.get_ud();

Both the combo box and the multi-line edit can be used in the text version of IDA as well.

Timers

Timers are very easy to use and are available in the text version of IDA as well. These are the prototypes of the functions:

qtimer_t register_timer(int interval, int (idaapi *callback)(void *ud), void *ud);
bool unregister_timer(qtimer_t t)

Timer functions are thread-safe and the callback is executed in the context of the main thread. The callback can return -1 to unregister the timer, while any other return greater than or equal to 0 defines the new interval of the timer.

// the timer event message will be displayed 5 times
int idaapi timer_callback(void *)
{
  static int i = 0;
  msg("timer event\n");
  return ++i == 5 ? -1 : 1000;
}
 
// register the timer with a 1-second interval
register_timer(1000, timer_callback, NULL);

That’s all! You can download both code viewer plugin samples from here.

1. Variable mapping

This is probably the simplest new feature and can be used without any extra preparation.

Sometimes the compiler stores the same variable in several places (e.g. a register and a stack slot). While the decompiler often manages to combine such locations, sometimes it’s not able to prove that they always contain the same value (especially in presence of calls that take address of stack variables). In such cases the user can help by performing such a merge or mapping manually.

Consider the following very common case:

int __stdcall SciFreeFilterInstance(_FILTER_INSTANCE *pFilterInstance)
{
  _FILTER_INSTANCE *v1; // esi@1

  v1 = pFilterInstance;
  if ( pFilterInstance->Signature != 'FrtS' )
    RtlAssert(
      "(pFilterInstance)->Signature==SIGN_FILTER_INSTANCE",
      "d:\\xpsprtm\\drivers\\wdm\\dvd\\class\\codinit.c",
      0x17A2u,
      0);
  StreamClassDebugPrint(2, "Freeing filterinstance %p still open streams\n", v1);

The compiler copied an incoming argument (pFilterInstance) into a register (v1==esi). To get rid of the extra name, right-click the left-hand variable and choose “Map to another variable”, or place cursor on it and press ‘=’:

Choose the right-hand variable from the list.

Once decompilation is refreshed, both the left-hand variable (v1) and the assignment are gone. Now we have only one variable – the incoming argument.

int __stdcall SciFreeFilterInstance(_FILTER_INSTANCE *pFilterInstance)
{
  if ( pFilterInstance->Signature != 'FrtS' )
    RtlAssert(
      "(pFilterInstance)->Signature==SIGN_FILTER_INSTANCE",
      "d:\\xpsprtm\\drivers\\wdm\\dvd\\class\\codinit.c",
      0x17A2u,
      0);
  StreamClassDebugPrint(2, "Freeing filterinstance %p still open streams\n",
    pFilterInstance);

You can map several variables to the same name, if necessary.

Made a mistake or mapped too much? It’s simple to fix. Right-click the wrongly mapped name and choose “Unmap variables”. Then choose the variable you want to see again.

2. Union selection.

This feature, naturally, only applies to unions. That means that you need to have union types in your database and assign the types to some variables or fields.

Normally the decompiler tries to choose a union field which matches the expression best, but sometimes there are several equally valid matches, and sometimes other types in the expression are wrong. In such cases, you can override the decompiler’s decision. For example, this code is common in Windows drivers:

NTSTATUS __stdcall DispatchDeviceControl(PDEVICE_OBJECT DeviceObject, PIRP Irp)
{
  PIO_STACK_LOCATION stacklocation; // ebx@1

  stacklocation = Irp->Tail.Overlay.CurrentStackLocation;
  if ( *&stacklocation->Parameters.Create.FileAttributes == 0x224010 )
  {
    v8 = stacklocation->Parameters.Create.Options == 20;
    if ( !v8 )
      goto LABEL_18;
    if ( stacklocation->Parameters.Create.SecurityContext < 1 )
      goto LABEL_87;
    v23 = Irp->AssociatedIrp.MasterIrp;

Since we know we’re in a DeviceControl handler, it’s likely the code is inspecting the Parameters.DeviceIoControl substructure and not Parameters.Create.

Right-click the field and choose “Select union field”, or place cursor on it and press Alt-Y.

Choose the Parameters.DeviceIoControl.IoControlCode field.

Other references to Parameters.Create can be fixed the same way. The updated decompilation makes more sense:

NTSTATUS __stdcall DispatchDeviceControl(PDEVICE_OBJECT DeviceObject, PIRP Irp)
{
  PIO_STACK_LOCATION stacklocation; // ebx@1

  stacklocation = Irp->Tail.Overlay.CurrentStackLocation;
  if ( stacklocation->Parameters.DeviceIoControl.IoControlCode == 0x224010 )
  {
    v8 = stacklocation->Parameters.DeviceIoControl.InputBufferLength == 20;
    if ( !v8 )
      goto LABEL_18;
    if ( stacklocation->Parameters.DeviceIoControl.OutputBufferLength < 1 )
      goto LABEL_87;

3. CONTAINING_RECORD macro

This macro is commonly use in Windows drivers to get a pointer to the parent structure when we have a pointer to one of its fields.

For example, consider these two structures, used in a driver:

struct _HW_STREAM_OBJECT {
  ULONG  SizeOfThisPacket;
  ULONG  StreamNumber;
  PVOID  HwStreamExtension;
  ...
} HW_STREAM_OBJECT, *PHW_STREAM_OBJECT;

struct _STREAM_OBJECT
{
  _COMMON_OBJECT ComObj;
  _FILE_OBJECT *FilterFileObject;
  _FILE_OBJECT *FileObject;
  _FILTER_INSTANCE *FilterInstance;
  _HW_STREAM_OBJECT HwStreamObject;
  ...
};

The following function accepts a pointer to _HW_STREAM_OBJECT:

void __cdecl StreamClassStreamNotification(
  int NotificationType,
  _HW_STREAM_OBJECT *StreamObject,
  _HW_STREAM_REQUEST_BLOCK *pSrb,
  _KSEVENT_ENTRY *EventEntry,
  GUID *EventSet,
  ULONG EventId);

But immediately converts it into the containing _STREAM_OBJECT:

mov     eax, [ebp+StreamObject]
test    eax, eax
push    ebx
push    esi
lea     esi, [eax-_STREAM_OBJECT.HwStreamObject]

Default decompilation doesn’t look great:

  char *v6; // esi@1
  v6 = (char *)&StreamObject[-2] - 36;

There are two ways to make it nicer:

1. Change type of v6 to be _STREAM_OBJECT*. The decompiler will detect that the expression “lines up” and convert it to use the macro.
2. Right-click on the delta being subtracted (-36), select “Structure offset” and choose _STREAM_OBJECT from the list.

In both cases you should get a nice expression:

  v6 = CONTAINING_RECORD(StreamObject, _STREAM_OBJECT, HwStreamObject);

N.B.: currently you need to refresh the decompilation (press F5) to see the changes. We’ll improve it to happen automatically in future.

4. Kernel and user-mode macros involving fs segment access.

On Windows, the fs segment is used to store various thread-specific (for user-mode) or processor-specific (for kernel mode) data. Hex-Rays Decompiler 1.6 detects the most common ways of accessing them and converts them to corresponding macros. However, this functionality requires presence of specific types in the database. For user mode, it is the _TEB structure, for kernel mode it’s the KPCR structure.

For example, consider the following code:

mov     eax, large fs:18h
mov     eax, [eax+30h]
push    24h
push    8
push    dword ptr [eax+18h]
call    ds:__imp__RtlAllocateHeap@12 ; RtlAllocateHeap(x,x,x)
mov     esi, eax

If you don’t have the _TEB structure in types, this will be decompiled to:

  v5 = RtlAllocateHeap(*(_DWORD *)(*(_DWORD *)(__readfsdword(24) + 48) + 24), 8, 36);

However, if you do add the type, it will look much nicer:

  v5 = RtlAllocateHeap(NtCurrentTeb()->ProcessEnvironmentBlock->ProcessHeap, 8, 36);

Currently we support the following macros:

Hint: the easiest way to get _TEB or KPCR types into your database is using the PDB plugin. Invoke it from File|Load file|PDB file…, enter a path to kernel32.dll (for user-mode code) or ntoskrnl.exe (for kernel-mode code), and check the “Types only” checkbox.

PDBs for those two files usually contain the necessary OS structures.

We hope you will like these new additions. Note that the version 1.6 includes even more improvements and fixes, see the full list of the new features and the comparison page.

Proximity view
PE+ support for Bochs (64-bit PE files)
UI shortcut editor
Filters in choosers
Database snapshots

Other new major features:

• GUI installers for Linux and OS X
  
  
  
• Automatic check for new versions:
  
  
• Cross-references to structure members:
  
  
• Floating licenses: our licensing system is now more flexible and allows big enterprises to purchase floating licenses. Contact sales@hex-rays.com for more information.

If you have an active license and would like to test the beta, please send a message to support@hex-rays.com.

Filters

Filters make it possible to either show, hide or highlight one or more categories of items. But enough talk, let’s start with a screenshot.

This list was created by including only items containing “str” and “mem” in their function name and then by highlighting with different colors the remaining items.

Here’s the dialog which allows to add specific filters for any chooser.

Don’t worry about the contrast, the chooser will automatically establish the best foreground color for each item given its background.

However, opening a dialog to set a filter may be cumbersome if the goal is just to quickly visualize a particular category of items. That’s why it will be possible to filter choosers by pressing a shortcut (Ctrl-F), via a small edit field popping up below the list.

This edit field can filter items by additionally specifying three options: case sensitivity, regular expressions and whole words only.

A lot of rewriting and optimizing had to be done to make sure that even with very huge lists (50,000+ items) the filtering would be fast.

Shortcut management

I think many users will appreciate that IDA Pro finally features a complete and advanced shortcut editor.

It’s not only possible to change shortcuts for built-in IDA actions, but also the default shortcuts of plugins, external menu entries and scripts. The shortcut editor will signal modified shortcuts (yellow), conflicting shortcuts (red) or whether both conditions are met (orange). It also provides some additional information about the action such as its origin (built-in, menu, plugin, script) and context. The context tells the user when this action can be executed, e.g.: some actions are globally active, while others may only be executed in the disassembly.

One interesting new addition to the action management is that an action now regains its lost shortcut if the conflict which caused the action to lose it ceases to exist.

And in case you wish to migrate your shortcut settings: all the modifications accomplished through the shortcut editor are saved to the ‘shortcuts.cfg’ file inside the user’s directory.