Hack of the day #0: Somewhat-automating pseudocode HTML generation, with IDAPython.

The problem

As you may already know1, Hex-Rays decompilers can generate HTML files from pseudocode windows.
That feature, however, is limited to generating HTML for a single function, or a portion of a function.

Recently, one of our customers asked us whether there was a way to generate HTML files for multiple functions all at once. I was at the regret of telling him that, no, we don’t have that feature (and it doesn’t really seem to be of interest to many, since AFAICT we have received no request for it)

However, after the Great Actions Refactoring™, decompiler action are now 1st class IDA citizens (just like any other plugin actions, really.) That means we can now invoke them, just like we could already invoke any IDA core action. We just need to take a few precaution, is all.

A first approach

Here’s a piece of IDAPython code I hammered into shape, that will go through all the functions of the program and (if the current function is decompilable) will prompt the user for saving:

from idaapi import *
from PySide import QtGui, QtCore

# Our piece of code relies on the user having already opened a pseudocode view. That's a limitation, but oh well.
tform = find_tform("Pseudocode-A")
if not tform:
    raise Exception("Please open a pseudocode view")
qwidget = PluginForm.FormToPySideWidget(tform)

# Iterate all funcs
for fidx in xrange(get_func_qty()):
    f = getn_func(fidx)
        if decompile(f):
            # Ok, we can jump there w/o being
            # switched to "IDA View-A"

            # Needed. W/o that, focus might be
            # elsewhere, and the 'hx:GenHtml'
            # will refuse to work.

    except DecompilationFailure:
        print "Decompilation failure: %x. Skipping."  % f.startEA


  • I only tried that with IDA 6.8, on a Linux box.
  • this is *NOT* meant to be an elegant, comfortable or rock-solid approach to generating HTML files with pseudocode (hence the ‘Hack of the day’ subject of this post!)
  • this is really just meant to illustrate how IDA APIs can be (ab)used, and perhaps give inspiration to some readers for some of their use-cases
  • this will go through _all_ functions in the program. If you just need to print 20 functions out of 13942, it might become tedious to reject (13942 – 20 =) 13922 “Save file” dialogs.

Improving that code, so only the selected functions will be printed

A possibly nice improvement to this, would be to know what functions the user selected in the “Functions window”, and iterate over those only, prompting for save.

Here’s how I would go about it, knowing that:

  • the ‘Functions window’ is what we call, in IDA-land, a ‘chooser’
  • alas, this is not a user-controlled chooser
  • thus we don’t control the data being printed, nor do we get called back whenever something gets selected/deselected…

    Great Actions Refactoring to the rescue again!

    We will:

    1. create an “Generate HTML files” action that, basically, wraps the above code, then
    2. register a ‘popup being populated’ hook: when the popup is for the ‘Functions window’ we will attach our new action to that popup
    3. then, when that action is invoked (i.e., user clicked it), in our action handler we will retrieve the list of selected indices and iterate over those, prompting for save

    from idaapi import *
    from PySide import QtGui, QtCore
    # Register action
    class GenMultiHTML(action_handler_t):
        def __init__(self):
        def activate(self, ctx):
            tform = find_tform("Pseudocode-A")
            if not tform:
                raise Exception("Please open a pseudocode view")
            qwidget = PluginForm.FormToPySideWidget(tform)
            for fidx in ctx.chooser_selection:
                f = getn_func(fidx - 1)
                    if decompile(f):
                        # ok, we can jump there w/o being
                        # switched to "IDA View-A"
                except DecompilationFailure:
                    print "Decompilation failure: %x. Skipping."  % f.startEA
            return 1
        def update(self, ctx):
            return AST_ENABLE_FOR_FORM if ctx.form_title == "Functions window" else AST_DISABLE_FOR_FORM
            'Generate HTML files',
    # Listen to 'finish populating popup' hook, to add our
    # action if the popup is for the "Functions window"
    class Hooks(idaapi.UI_Hooks):
        def finish_populating_tform_popup(self, form, popup):
            if get_tform_title(form) == "Functions window":
    hooks = Hooks()

    Happy hacking!


    #1: And if you didn’t know: go to a pseudocode window, click on the function’s title (i.e., the first line), and open the context menu. You can also select a portion of text, and generate HTML just for that.

IDAPython + PySide/PyQt: future plans

Intended audience

IDAPython plugin writers who are using the PySide Qt bindings.

PySide: some background

For some time now it has been possible, through IDAPython, to use PySide bindings to the Qt libraries that are shipped with IDA.

Those PySide bindings were first placed on Hex-Rays’s website and, since we noticed a considerable interest for them, we later decided to ship them with IDA (starting at IDA 6.6.)

What about PyQt?

The choice of PySide over PyQt was essentially due to incompatibilities between the licensing model of PyQt, and that of Hex-Rays.

The problem

PySide appears to have pretty much stopped evolving, and now remains stuck with Qt 4 (i.e., the 4.8 branch.)
PySide isn’t available for Qt 5 and, as of today, the last update of the PySide roadmap dates back to “March 26, 2013”.

There is still some activity on the PySide mailing list, but despite that, not much seems to be happening (and the latest release is from 1 year ago.)

This is very problematic for us as Qt 4 is becoming more and more irrelevant by the day, and we would love to move to Qt 5.

We could, in theory, invest in porting PySide to support Qt 5, but it would probably lead us to an undesirable situation where PySide is pretty much used and maintained by Hex-Rays only.

Meanwhile, in PyQt land

In the meantime, the PyQt people at http://www.riverbankcomputing.co.uk/ have changed their licensing options, and after having contacted them recently, we have found that we can now consider the following:

  • Hex-Rays can acquire PyQt licenses
  • with those, we can build PyQt, and ship it with IDA
  • IDA users can make use of PyQt themselves

In other words, we will be able to provide a replacement for PySide, that:

  • is widely used, and has momentum
  • supports Qt 5

Our plan

Our plan is to roll, in a few days, IDA 6.8 that still depends on Qt 4, and still ships with PySide.

The release after that (i.e., 6.9) will then depend on Qt 5, and come with PyQt.

What that means for developers

Thankfully, the interfaces of PySide and PyQt are highly compatible, and thus it should not be necessary to rewrite your plugins from scratch (in fact, very little should have to be modified.)

I would recommend you to stay tuned to the Hex-Rays forums, so that you can participate in the beta program for IDA 6.9 once it is announced. That will give you the possibility of porting your tools to PyQt before IDA 6.9 is released. We will also provide guidance on how to approach that porting effort (even though, once again, it should be fairly small.)

As you may already know, we are very reluctant to any kind of API/compatibility breakage. But unfortunately, it seems we simply don’t have a choice in this particular case.

Augmenting IDA UI with your own actions.

Intended audience

Plugin writers, either using the C SDK or IDAPython, who would like to add actions/commands to IDA UI in order to augment its capabilities.

Rationale: before 6.7

APIs galore

Depending on what type of context you were in, various APIs were available to you:

  • Want to add a main menu item?

    add_menu_item(const char *menupath, const char *name, const char *hotkey, int flags, menu_item_callback_t *callback, void *ud);
    del_menu_item(const char *menupath)
  • Want to add an action to a list view? (or, as we call them: ‘choosers’)

    add_chooser_command(const char *chooser_caption, const char *cmd_caption, chooser_cb_t *chooser_cb, int menu_index=-1, int icon=-1, int flags=0);
    add_chooser_command(const char *chooser_caption, const char *cmd_caption, chooser_cb_t *chooser_cb, const char *hotkey, int menu_index=-1, int icon=-1, int flags=0);

    …or through populating callbacks:

    struct chooser_info_t
        // ...snipped...
        // function is called every time before
        // context menu is shown to fill user_cmds
        prepare_popup_cmds_t *prepare_popup_cmds;
  • Want to add/manage the items that show up in the context menu for a custom code viewer you have created? (that does not work for the ‘core’ viewers like ‘IDA View-A’!)

    set_custom_viewer_popup_menu(TCustomControl *custom_viewer, TPopupMenu *menu);
    add_custom_viewer_popup_item(TCustomControl *custom_viewer, const char *title, const char *hotkey, menu_item_callback_t *cb, void *ud);
  • Want to add/manage the items that show up in the context menu for a custom graph viewer you have created? (again, doesn’t work with ‘core’ viewers)

    viewer_add_menu_item(graph_viewer_t *gv, const char *title, menu_item_callback_t *callback, void *ud, const char *hotkey, int flags);
  • Want to add an item to the Output window’s context menu?

    add_output_popup(char *n, menu_item_callback_t *c, void *u);


Note that not all of those functions take the same set of parameters, or even the same type for seemingly-related ones (e.g., chooser_cb_t *chooser_cb VS menu_item_callback_t *callback)

It is also not possible to specify some attributes of the commands in some cases. E.g.,

  • add_menu_item() doesn’t allow to specify an icon
  • add_output_popup() doesn’t accept a shortcut

And it’s just plain impossible to do some things. E.g.,

  • augment IDA View’s context menu with your own actions
    EDIT: This is incorrect, as the ‘view_popup’ notification could be used for this. However, this is still limited to IDA View-A, and cannot be used for other widgets.
  • remove an item that was previously added to the Output window’s context menu through add_output_popup().
  • in some cases, actions shortcuts would do nothing if the context menu wasn’t currently displayed (i.e., showing the context menu was necessary to enabled the hotkeys)
  • adding custom toolbar buttons was impossible.

These inconsistencies/lack of features were making writing extensions difficult, as the wealth of APIs was somewhat confusing and it was easy to hit the limitations.

In addition, there was no really good way of enabling/disabling your actions, or changing their properties after adding them.

Thus, we decided a refactoring of those APIs was in order.

The new actions API

Borrowing some concepts from Qt itself (upon which IDA’s graphical interface relies) we have decided to redesign the actions API in a hopefully more streamlined manner.

Key aspects

  • An action first needs to be registered. Once registered, it can be triggered using its shortcut (if one was specified), but it doesn’t appear anywhere in the UI just yet.
  • Once registered, it is possible attach/detach that action to/from things:
    • Main menus
    • Toolbars
    • Context menus
  • The same action can be used in multiple places.
  • An action has a “handler”, which is a small structure consisting of two callbacks:
    • an ‘activate’ callback, called when the action was triggered (i.e., user selected a menu item, or entered a shortcut, or pressed a toolbar item, …)
    • an ‘update’ callback, which is responsible for two things:
      • Declaring whether the action is currently enabled, and when it should be queried again for that information.
      • Possibly updating the action’s state (e.g., text, icon, …) (although that’s rarely needed.)

What follows is a series of short examples, using the IDAPython bindings for actions. Note that the IDAPython API is purposely similar to the C SDK API.

Registering an action

    # 1) Create the handler class
    class MyHandler(idaapi.action_handler_t):
        def __init__(self):

        # Say hello when invoked.
        def activate(self, ctx):
            print "Hello!"
            return 1

        # This action is always available.
        def update(self, ctx):
            return idaapi.AST_ENABLE_ALWAYS

    # 2) Describe the action
    action_desc = idaapi.action_desc_t(
        'my:action',   # The action name. This acts like an ID and must be unique
        'Say hello!',  # The action text.
        MyHandler(),   # The action handler.
        'Ctrl+H',      # Optional: the action shortcut
        'Says hello',  # Optional: the action tooltip (available in menus/toolbar)
        199)           # Optional: the action icon (shows when in menus/toolbars)

    # 3) Register the action

From there on, the action is created and, although it has not yet been attached to menus, toolbars or context menus, it can already be invoked by pressing Ctrl+H:

This is slightly more code than before (and in this case, it is explicitely verbose), but regardless of the context/widget in which you want to use that action, it will always be the same API.

Note: In this example, we used icon number 199, which corresponds to a stock icon. It is perfectly possible to define your own icons by using the API function load_custom_icon(). See kernwin.hpp for more information.

Inserting the action into the menu

        'Edit/Other/Manual instruction...', # The relative path of where to add the action
        'my:action',                        # The action ID (see above)
        idaapi.SETMENU_APP)                 # We want to append the action after the 'Manual instruction...'

Inserting the action into a toolbar

        "AnalysisToolBar",  # The toolbar name
        'my:action')        # The action ID

Inserting the action into a widget’s context menu

There are two ways to add actions to a widget’s context menu:

  • Permanently: the action will be there every time the context menu is shown
  • On the fly: you can attach an action as the context menu is being populated, and the action will be displayed in this invocation of the context menu, but not others (unless you attach it again, obviously.)

Attaching an action permanently

To permanently attach an action to a widget’s context menu:

    # Create a widget, or retrieve a pointer to it.
    form = idaapi.get_current_tform()
    idaapi.attach_action_to_popup(form, None, "my:action", None)

Notice that:

  • We used get_current_tform(), to retrieve the widget that currently has the keyboard focus.
  • We passed None as second parameter to attach_action_to_popup(). That means we want to attach the action to that widget permanently.


  • Trivial to use.
  • This is what you typically want to use when you create your own widgets


  • Only feasible when you can get a reference (i.e., pointer) to the widget.

Attaching an action on-the-fly

To attach an action to a context menu when it is being created, you need to listen to a UI notification, and use the same attach_action_to_popup function:

    class Hooks(idaapi.UI_Hooks):
        def populating_tform_popup(self, form, popup):
            # You can attach here.

        def finish_populating_tform_popup(self, form, popup):
            # Or here, after the popup is done being populated by its owner.

            # We will attach our action to the context menu
            # for the 'Functions window' widget.
            # The action will be be inserted in a submenu of
            # the context menu, named 'Others'.
            if idaapi.get_tform_type(form) == idaapi.BWN_FUNCS:
                idaapi.attach_action_to_popup(form, popup, "my:action", "Others/")

    hooks = Hooks()

Notice that:

  • This time, the 2nd parameter is not ‘None’. That instructs the function to attach the action for this invocation only.
  • We know that we are currently in the “Functions window”, by checking the type of the current form against BWN_FUNCS. Many other BWN_* values are available; see kernwin.hpp for a list of those!
  • As an alternative, we could check the form’s title, using get_tform_title(form).


  • Finer-grained control.
  • You typically want to use this when you cannot easily retrieve a reference (i.e., pointer) to the widget.


  • More difficult to use: requires UI hooks.

A word about the update callback

As stated above, the update callback is responsible not only for possibly updating some of the actions properties (See the update_action_* functions in kernwin.hpp), but also for stating whether the action is currently available or not, and when update should be queried again.

Per contract, update should return one of the following values (again, from kernwin.hpp):

    AST_ENABLE_ALWAYS     // enable action and do not call action_handler_t::update() anymore
    AST_ENABLE_FOR_IDB    // enable action for the current idb. Call action_handler_t::update() when a database is opened/closed
    AST_ENABLE_FOR_FORM   // enable action for the current form. Call action_handler_t::update() when a form gets/loses focus
    AST_ENABLE            // enable action - call action_handler_t::update() when anything changes

    AST_DISABLE_ALWAYS    // disable action and do not call action_handler_t::action() anymore
    AST_DISABLE           // analog of ::AST_ENABLE

Thus, not only does update tell IDA whether the action is available or not, but it also tells IDA the “time span” during which the action will be (un)available.

For example, returning AST_ENABLE_FOR_FORM will tell IDA: “this action is available for the current widget; don’t call update again until the user switches to another widget.”

The AST_* values really represent various levels of “granularity”, where

  • AST_ENABLE_ALWAYS is the coarser level: the action will be available all the time, whatever happens.
  • AST_ENABLE is the finest level: whenever the user moves the cursor, switches to another form, triggers another action, …, update will be called again.

“Dynamic” actions

In most cases, you will want your action to be registered in IDA, and be available through menus, shortcuts, etc…

But there can be “one-shot” situations where you really want to add an action to a context menu only once, and you really don’t want to go through the trouble of registering & deregistering the action.

For those “one-shot” situations, although they are rare, we have created a helper:

    class Hooks(idaapi.UI_Hooks):
        def finish_populating_tform_popup(self, form, popup):
            tft = idaapi.get_tform_type(form)
            if tft == idaapi.BWN_EXPORTS:

                # Define a silly handler.
                class MyHandler(idaapi.action_handler_t):
                    def activate(self, ctx):
                        print "Hello from exports"
                    def update(self, ctx):
                        return idaapi.AST_ENABLE_ALWAYS

                # Note the 'None' as action name (1st parameter).
                # That's because the action will be deleted immediately
                # after the context menu is hidden anyway, so there's
                # really no need giving it a valid ID.
                desc = idaapi.action_desc_t(None, 'My dynamic action', MyHandler())
                idaapi.attach_dynamic_action_to_popup(form, popup, desc, None)

    hooks = Hooks()

The main difference When using attach_dynamic_action_to_popup is that the action will be available only during the time the context menu is displayed. Once the context menu is hidden, the action will be unregistered & destroyed.

Removing action from things

  • detach_action_from_menu
  • detach_action_from_toolbar
  • unregister_action

Older APIs: Backward compatibility

All the previous API is still supported. We have fairly good test coverage of those, and therefore are quite confident the backward compatibility is working.

If your plugin suddenly stops working properly in 6.7, that’s a bug. Should that happen, please let us know about it and we’ll do our best to address it.

IDA Dalvik debugger: tips and tricks

One of the new features of IDA 6.6 is the Dalvik debugger, which allows us to debug Dalvik binaries on the bytecode level.

Let us see how it can help when analysing Dalvik files.

Encoded strings

Let us consider the package with the encrypted strings:

STRINGS:0001F143 unk_1F143:.byte 0x30 # 0  # DATA XREF: STR_IDS:off_70
STRINGS:0001F144 aFda8sohchnidgh:
.string "FDA8sOhCHNidghM2hzFxMXUsivl2k7hFOhkJrW7O2ml8qLVM",0
STRINGS:0001F144                           # DATA XREF: q_b@V
STRINGS:0001F144                           # String #277 (0x115)
STRINGS:0001F175 unk_1F175:.byte 0x3C # <  # DATA XREF: STR_IDS:off_70
STRINGS:0001F176 aCgv01n8li2s3ok:
.string "CGv01N8li2s3OKN29j6exe6-rvzgIRaCcWoOt5y30zjP1k43-f7WVOtXjbg="
STRINGS:0001F176                           # DATA XREF: q_b@V+C

There is a data reference, let us see where this string is used (e.g. using Ctrl-X).

CODE:000090C0 const-string        v0, aFda8sohchnidgh # "FDA8sOhCH"...
CODE:000090C4 invoke-static       {v0}, <ref RC4.decryptBase64(ref) 
CODE:000090CA move-result-object  v0

So, apparently the strings are encrypted with RC4+Base64.

Let us set a breakpoint after the RC4.decryptBase64() call and start the debugger.

After hitting the breakpoint, open the “Locals” debugger window. Even if the application was stripped of debug information, IDA makes the best effort to show function’s input arguments and return value.


Note the local variable named retval. It is a synthetic variable created IDA to show the function return value.

This is how we managed to decode the string contents.

How to debug Dalvik and ARM code together

Let us have a look at application that uses a native library. On a button press, the function stringFromJNI() implemented in the native library is called.

package ida.debug.hellojni;
public class MainActivity extends Activity {
    public native String stringFromJNI();
    static {
    public void onCreate(Bundle savedInstanceState) {
        final TextView tv = new TextView(this);
        final Button btn = new Button(this);
        btn.setText("Press me to call the native code");
        btn.setOnClickListener(new Button.OnClickListener() {
                public void onClick(View v) {
        LinearLayout layout = new LinearLayout(this);
        layout.setLayoutParams(new LayoutParams(
            LayoutParams.FILL_PARENT, LayoutParams.FILL_PARENT));

Native library function returns a well-known string.

jstring Java_ida_debug_hellojni_MainActivity_stringFromJNI(
        JNIEnv* env,
        jobject thiz)
  return (*env)->NewStringUTF(env, "Hello, JNI world!");

So, we have application packaged in hellojni.apk file and installed in Android Emulator.

Because IDA cannot analyse or debug both Dalvik and native (ARM) code at the same time, we’ll need to use two IDA instances to perform debugging in turns.

To prepare the first IDA instance:

  1. load hellojni.apk into IDA, select classes.dex to analyze.

  2. go to the native function call and set breakpoint there.

    CODE:0000051C  iget-object         v0, this, MainActivity$1_val$tv
    CODE:00000520  iget-object         v1, this, MainActivity$1_this$0
    CODE:00000524  invoke-virtual      {v1},
    <ref MainActivity.stringFromJNI() imp. @ _def_MainActivity_stringFromJNI@L>
    CODE:0000052A  move-result-object  v1

  3. change the default port in “Debugger/Process options” to any other value.

  4. start the Dalvik debugger and wait until breakpoint is hit.

Then prepare the second IDA instance:

  1. prepare to debug native ARM Android application (copy and start android_server and so on).

  2. load hellojni.apk into IDA, and now select lib/armeabi-v7a/libhello-jni.so to analyze.

  3. the name of the native function was formed by special rules and in our case it is Java_ida_debug_hellojni_MainActivity_stringFromJNI(), so go to to it and set a breakpoint:

    .text:00000BC4 EXPORT Java_ida_debug_hellojni_MainActivity_stringFromJNI
    .text:00000BC4 Java_ida_debug_hellojni_MainActivity_stringFromJNI
    .text:00000BC4 var_C  = -0xC
    .text:00000BC4 var_8  = -8
    .text:00000BC4        STMFD   SP!, {R11,LR}
    .text:00000BC8        ADD     R11, SP, #4
    .text:00000BCC        SUB     SP, SP, #8
    .text:00000BD0        STR     R0, [R11,#var_8]
    .text:00000BD4        STR     R1, [R11,#var_C]
    .text:00000BD8        LDR     R3, [R11,#var_8]
    .text:00000BDC        LDR     R3, [R3]
    .text:00000BE0        LDR     R2, [R3,#0x29C]
    .text:00000BE4        LDR     R0, [R11,#var_8]
    .text:00000BE8        LDR     R3, =(aHelloJniWorld - 0xBF4)
    .text:00000BEC        ADD     R3, PC, R3 ; "Hello, JNI world!"
    .text:00000BF0        MOV     R1, R3
    .text:00000BF4        BLX     R2
    .text:00000BF8        MOV     R3, R0
    .text:00000BFC        MOV     R0, R3
    .text:00000C00        SUB     SP, R11, #4
    .text:00000C04        LDMFD   SP!, {R11,PC}
    .text:00000C04 ; End of function

  4. select “Remote ARM Linux/Android debugger” and attach to the application process.

  5. press F9 to continue.

Now switch to the first IDA session and press, for example, F8 to call native function. If we return back to the second IDA session then we can notice the breakpoint event.


Now we can continue to debug the native code. When we finish, press F9 and return to the first IDA session.

The full source code of the example you can download from our site.

Extending IDAPython in IDA 6.5: Be careful about the GIL

Target audience

You may want to read this if you have been writing an IDA C++ plugin, that itself uses the CPython runtime.

Prior art

In 2010, Elias Bachaalany wrote a blog post about extending IDAPython: http://www.hexblog.com/?p=126

Note that this is not about writing your own plugins in Python. Rather, that blog post instruct on how you may want to write a plugin that, itself, links against libpython27, so that it interacts nicely with IDAPython (which links against the same libpython27 library).

Before 6.5

The following plugin body was enough to have a working C++ plugin, that will execute a simple Python statement:

void idaapi run(int)
PyRun_SimpleString("print \"Hello from plugin\"");

Notes about the previous example

It is worth pointing out that, whenever one wants to execute any CPython code (such as the ‘PyRun_SimpleString’ call above), one must have acquired what is called in CPython-land as the Global Interpreter Lock, or GIL: https://wiki.python.org/moin/GlobalInterpreterLock.
It may seem strange, therefore, that the code above even works, since there’s nothing that even remotely looks like a “GIL-acquiring” sequence of instructions.

The reason the code above used to work is because IDAPython was not releasing the GIL: it was acquiring it at startup-time, and never releasing it. Ever.

IDA 6.5 & later versions

In IDA 6.5, many efforts have been made to improve the use of multithreading in IDAPython: The IDAPython plugin acquires the GIL just in time to execute CPython code and then releases it, as it should. This will permit a reliable use of threading in IDAPython (in some circumstances, when using multithreading, IDA < 6.5 could freeze because of the GIL not being released.)

Starting with 6.5, such a plugin would require to be written in the following manner:

void idaapi run(int)
PyGILState_STATE state = PyGILState_Ensure(); // Acquire the GIL
PyRun_SimpleString("print \"Hello from plugin\"");
PyGILState_Release(state); // Release the GIL

Note the use of the CPython-defined PyGILState_* helpers.

Existing plugins

Unfortunately, we couldn’t do anything in this case to ensure backwards-compatibility: existing plugins will have to be adapted & recompiled. Sorry about this, but we just could not find a reasonable solution to maintain bw-compat in this case; hence this blog post.

So far, we have received a grand total of 1 report for this issue, so this doesn’t appear to impact many of our users. But in case you are running into problems with your previously-compiled CPython-aware plugin, hopefully this will have explained why.

x64 decompiler not far away

Just a short post to show you the current state of the x64 decompiler. In fact, it already mostly works but we still have to solve some minor problems. Let us consider this source code:

struct color_t
  short red;
  short green;
  short blue;
  short alpha;

extern color_t lighten(color_t c);

color_t func(int red, int green, int blue, int alpha)
  color_t c;
  c.red = red;
  c.green = green;
  c.blue = blue;
  c.alpha = alpha;
  return lighten(c);

After compilation we get the following binary code:

.text:0000000000000000 ?func@@YA?AUcolor_t@@HHHH@Z proc near
c = color_t ptr -18h
.text:0000000000000000 var_10 = qword ptr -10h
.text:0000000000000000 arg_0 = dword ptr 8
.text:0000000000000000 arg_8 = dword ptr 10h
.text:0000000000000000 arg_10 = dword ptr 18h
.text:0000000000000000 arg_18 = dword ptr 20h
mov [rsp+arg_18], r9d ; $LN3
.text:0000000000000005 mov [rsp+arg_10], r8d
.text:000000000000000A mov [rsp+arg_8], edx
.text:000000000000000E mov [rsp+arg_0], ecx
.text:0000000000000012 sub rsp, 38h
.text:0000000000000016 movzx eax, word ptr [rsp+38h+arg_0]
.text:000000000000001B mov [rsp+38h+c.red], ax
.text:0000000000000020 movzx eax, word ptr [rsp+38h+arg_8]
.text:0000000000000025 mov [rsp+38h+c.green], ax
.text:000000000000002A movzx eax, word ptr [rsp+38h+arg_10]
.text:000000000000002F mov [rsp+38h+c.blue], ax
.text:0000000000000034 movzx eax, word ptr [rsp+38h+arg_18]
.text:0000000000000039 mov [rsp+38h+c.alpha], ax
.text:000000000000003E mov rcx, qword ptr [rsp+38h+c.red] ; c
.text:0000000000000043 call ?lighten@@YA?AUcolor_t@@U1@@Z ; lighten(color_t)
.text:0000000000000048 mov [rsp+38h+var_10], rax
.text:000000000000004D mov rax, [rsp+38h+var_10]
.text:0000000000000052 add rsp, 38h
.text:0000000000000056 retn
.text:0000000000000056 ?func@@YA?AUcolor_t@@HHHH@Z endp

Please note that the c, which is a structure, is passed by value in 2 registers: rcx and rdx. We had to rework quite many things in the decompiler to support such variables (we call them scattered variables). However, the output was worth it:

color_t __fastcall func(__int16 cx0, __int16 dx0, __int16 r8_0, __int16 r9_0)
  color_t c;

  c.red = cx0;
  c.green = dx0;
  c.blue = r8_0;
  c.alpha = r9_0;
  return lighten(c);

There is still some work to be done, but it seems we solved most problematic issues. Stay tuned, there will be more decompiler news soon!


Interacting with IDA through IPC channels

I’m happy to present you a guest post by David Zimmer <dzzie@yahoo.com>. The approach he describes can be used to develop plugins more conveniently (but not limited to that):

In this article we are going to discuss a mechanism that can be used to interact with IDA through external applications.

The reason this technique was developed was to provide a convenient way for utility applications to query information from IDA databases, and automate its interface.

Over the years, several methods have been tried such as pipes, and sockets. In the end, the easiest Inter-Process Communication (IPC) technique I have found is the Windows specific SendMessage API along with the WM_COPYDATA message. This technique was chosen for its simplicity, reliability, and its inherently synchronous nature.

With this technique, the IDA server plugin creates a window and watches for command messages to come in.

(abbreviated C source to CreateWindow and receive messages)

The plugin contains its own text based API which can be used to automate actions and return queried data back to the calling process. Example commands and data transfer routines are shown below:

(hwnd arguments specify which window handle receives data callback)

There are several advantages to this technique. Traditional plugin development usually requires compiling your code and launching the plugin from the host application.
Debugging often entails wading through runtime debug logs, inferring problems through observed behavior, and chasing crashs in a debugger.

This technique allows you to debug your executable in the development IDE as normal, harnessing all of its powerful capabilities such as edit and continue, call stack viewing, variable watches, breakpoints etc.

Complex projects can be run directly while you iron out interface behaviors and are not dependant upon the host. Datasets can even be loaded from test files so that many routines can be debugged independently of IDA.

One project where this technique was put to good use was an experiment to see if a Javascript IDE could be created for IDA automation tasks. The desire was for a scriptable interface that supported intellisense, function prototype tool tips, and syntax highlighting.


With a project such as this, coding for the IDE behaviors is a major part of the development task. To develop such an application strictly as a plugin would be a daunting endeavor. Developing it as a standalone application was a great advantage where it could be run directly from the Visual Studio IDE without any intermediate steps.

One other advantage to this technique is that it opens up plugin development to include higher level languages that do not have native support for the IDA API*. Languages such as C#, Delphi, and VB6 can easily interact with IDA through this mechanism. These languages have excellent rapid GUI development capabilities and a wealth of complex components already available to them. This technique is even open to Java developers through the JNI.

Once the intermediate API and the client access library is written, creating utilities to integrate with IDA becomes a pretty quick process. Another example project that has come in handy is a Wingraph32 replacement that was coded in about a day.

The interface shown below automatically syncs the IDA disassembly when graph nodes are clicked and can perform several renaming operations.


There are some limitations to this technique as well. The particular implementation detailed in this article is Windows specific. It also requires both applications to be running on the same machine.

Another consideration is that the data is crossing process boundaries which can be a performance hit depending on what is being transferred and how often it is being invoked. For example, extracting or patching large blocks of data would best be optimized by reserving the use of the SendMessage API as the command channel, and utilizing files or shared memory for the data transfer. This will likely be an optimization included in future revisions.

The example IDASRVR project linked to below currently supports 36 API and uses a simple text based command protocol. Sample code is provided in C and C#. Client libraries are also available for C# and VB6 in the associated projects below.

Sample Projects:

IDASrvr – IDA IPC Server example

IDA_Jscript – Javascript IDE PoC for IDA (VB6)

GleeGraph – C# Wingraph32 replacement

* You can create C# and VB6 in process plugins for IDA through COM.
This was my first approach, and was used in my
IDACompare plugin.

Loading your own modules from your IDAPython scripts with idaapi.require()


If you were using import to import your own “currently-in-development” modules from your IDAPython scripts, you may want to use idaapi.require(), starting with IDA 6.5.


When using IDAPython scripts, users were sometimes facing the following issue


  • User loads script
  • Script imports user’s module mymodule
  • Script ends
  • User modifies code of mymodule (Note: the module is modified, not the script)
  • User reloads script
  • Modifications to mymodule aren’t taken into consideration.

While that’s perfectly understandable (the python runtime doesn’t have to reload mymodule if it has been compiled & loaded already), this is somewhat of an annoyance for users that were importing modules that were often modified.

IDA <= 6.4: Ensuring a user-specified module gets reloaded, by destroying it.

Up until IDA 6.4, the IDAPython plugin would do some magic after you have run your user script.
(click “expand all” to reveal the diff)

The sequence becomes:

  • User loads script
  • Script imports user’s module mymodule
  • Script ends
  • [module mymodule is deleted]
  • User modifies code of mymodule
  • User reloads script
  • Modifications to mymodule are taken into consideration, since module was deleted.

Unfortunately we have to stop doing this because:

  • That prevents us from using python-based hooks to be used after the script is finished (see below).
  • That goes against the rest of the python philosophy (i.e., modifications to objects are not reverted), and is therefore unexpected.

Issues with hooks.

Imagine you have the following script, dbghooks.py:

from idaapi import *
import mydbghelpers

class MyHooks(DBG_Hooks):

  def __init__(self):

  def dbg_bpt(self, tid, ea):
    return 0

  def dbg_step_into(self):

hooks = MyHooks()
  • User loads script
  • Scripts imports mydbghelpers
  • Script creates instance of MyHooks, and hooks it into IDA’s debugger APIs
  • Script ends
  • [module mydbghelpers is deleted]
  • User runs debugger, and a breakpoint is hit. Two things can happen:
    • The hook fails executing
    • IDA crashes (that can happen if the form from mydbghelpers import * was used)

IDA > 6.4: Introducing idaapi.require()

Everywhere else in python, when you modify a runtime object, those changes will remain visible.

We decided it would be better to not go against that standard behaviour anymore, and provide a helper to achieve the same results as what was achieved before with the deletion of user modules.

You can now import & re-import of a module with: idaapi.require(name)

Here is its definition:

def require(modulename):
    if modulename in sys.modules.keys():
        import importlib
        import inspect
        m = importlib.import_module(modulename)
        frame_obj, filename, line_number, function_name, lines, index = inspect.stack()[1]
        importer_module = inspect.getmodule(frame_obj)
        if importer_module is None: # No importer module; called from command line
            importer_module = sys.modules['__main__']
        setattr(importer_module, modulename, m)
        sys.modules[modulename] = m


The example debugger hooks script above becomes:

from idaapi import *

class MyHooks(DBG_Hooks):

  def __init__(self):

  def dbg_bpt(self, tid, ea):
    return 0

  def dbg_step_into(self):

hooks = MyHooks()

I.e., only the second line changes.

Installing PIP packages, and using them from IDA on a 64-bit machine

Recently, one of our customers came to us asking how he should proceed to be able to install python packages, using PIP, and use those from IDA.

The issue he was facing is that his system is a 64-bit Ubuntu 12.04 VM.
Therefore using the Ubuntu-bundled PIP will just result in installing the desired package (let’s say ssdeep) for the system Python runtime, which is a 64-bit runtime and therefore not compatible with IDA.

The best (as in: cleanest) solution I have found is to:

  • build a 32-bits python on the system.
  • pip-install packages in that 32-bits python’s sub-directories.
  • export PYTHONPATH to point to the 32-bits python’s sub-directories.

We figured we’d write it down here just in case it might help others.


  • Install autoconf
  • Install ia32-libs

Building & installing a 32-bits python

  • ..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH
  • Download Python2.7.4
    • Note:You should make sure that the MD5 checksum and the size of the file you downloaded match those that are advertised on the page. That would prevent a man-in-the-middle attacker from providing you with a malicious Python bundle.
  • Build it. Note that you’ll probably have to sudo-create a few symlinks. I had to do this, on the Ubuntu 12.04 64-bit VM I tested this on:
    • /lib/i386-linux-gnu/libssl.so/lib/i386-linux-gnu/libssl.so.1.0.0
    • /lib/i386-linux-gnu/libcrypto.so/lib/i386-linux-gnu/libcrypto.so.1.0.0
    • /lib/i386-linux-gnu/libz.so/lib/i386-linux-gnu/libz.so.1
  • For the sake of completeness, here are my build commands (don’t forget the flags, of course):
    • ..$ CFLAGS=-m32 LDFLAGS=-m32 ./configure --prefix=/opt/Python2.7.4-32bits
    • ..$ CFLAGS=-m32 LDFLAGS=-m32 make -j 8

Once the build completes

Here’s what I have as last lines of the build:

INFO: Can't locate Tcl/Tk libs and/or headers

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _curses            _curses_panel
_sqlite3           _tkinter           bsddb185
bz2                dbm                gdbm
readline           sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

If you see, below that, that it failed to build, say 'binascii', then something went wrong.

Make sure you run make -j 1 to check out what went wrong (i.e., what library it claims not being able to find)

Once you have succesfully built your 32-bits Python, it’s time to install it: sudo make install

Trying your freshly-built python

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii

No complaint so far. Good.

Checking that pkg_resources is available.

Try importing pkg_resources. If it fails, you’ll probably have to do the following:

..$ cd /tmp
..$ curl -O http://python-distribute.org/distribute_setup.py
..$ less distribute_setup.py  # (*)
..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 distribute_setup.py

That will print out quite a fair amount of info, and should succeed.

(*) Note: A careful reader has pointed out that it would be fairly easy to intercept (man-in-the-middle) such an HTTP request, and serve malicious content that would then be piped (as root) to Python.
That’s why I think it’s important to mention, as a third step (i.e., less ...), that the code that was downloaded should ideally be checked. Hopefully, http://python-distribute.org will soon provide HTTPS support, which will limit such MITM attack risks.

Trying your freshly-built python, again

We want to make sure pkg_resources can be imported.

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources

Still no complaint. Good.

If yours complains, you’ll have to first make sure you fix whatever is causing it to fail, because the next will not work without that.

Installing PIP for your new Python build

Since using your system’s PIP will probably not work (as it would build & install things in a 64-bits python sub-directory), you’ll have to install a PIP package specifically for your freshly-built Python.

Here’s how I proceeded:

..$ cd /tmp;
..$ curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py;
..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 get-pip.py

PIP is now installed.

PIP-installing a package (i.e., ssdeep)

To download/build/install the ssdeep package I ran, as root (either that, or you’ll have to give your user the rights to write in /opt/Python2.7.4-32bits):

..$ su
root ..$ export CFLAGS=-m32
root ..$ export LDFLAGS=-m32
root ..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH
root ..$ /opt/Python2.7.4-32bits/bin/python2.7 /opt/Python2.7.4-32bits/bin/pip install ssdeep

Notice how I use my freshly-built python, with my fresly-installed PIP (and not the system one.)

Note: Don’t forget the export lines, or PIP will partially build stuff for x64, and partially for x86. That, as you can guess, won’t quite work.

If you forgot the export lines and started building anyway (and the build failed because of the mixed architecture issue I just wrote about), make sure you delete whatever is in /tmp/pip-build-*, so that there won’t be stale object files of inappropriate architecture in there.

Check out the PIP-installed package works

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssdeep
>>> ssdeep
<module 'ssdeep' from '/opt/Python2.7.4-32bits/lib/python2.7/site-packages/ssdeep.so'>
>>> dir(ssdeep)
['Error', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__test__', '__version__', 'compare', 'hash', 'hash_from_file', 'sys']

So far so good.

Testing the PIP-installed package in IDA

Since that’s still the goal (though you might have forgotten by now, given the amount of directions above.. 😉 ),
we’ll now try and make use of that PIP-installed package in IDA.

  • ..$ export PYTHONPATH=/opt/Python2.7.4-32bits/lib/python2.7/site-packages:/opt/Python2.7.4-32bits/:$PYTHONPATH
  • ..$ idaq

If all went well, typing import ssdeep in the Python input line should properly, silently, nicely import the package.