Extending IDAPython in IDA 6.5: Be careful about the GIL

Target audience

You may want to read this if you have been writing an IDA C++ plugin, that itself uses the CPython runtime.

Prior art

In 2010, Elias Bachaalany wrote a blog post about extending IDAPython: http://www.hexblog.com/?p=126

Note that this is not about writing your own plugins in Python. Rather, that blog post instruct on how you may want to write a plugin that, itself, links against libpython27, so that it interacts nicely with IDAPython (which links against the same libpython27 library).

Before 6.5

The following plugin body was enough to have a working C++ plugin, that will execute a simple Python statement:

void idaapi run(int)
PyRun_SimpleString("print \"Hello from plugin\"");

Notes about the previous example

It is worth pointing out that, whenever one wants to execute any CPython code (such as the ‘PyRun_SimpleString’ call above), one must have acquired what is called in CPython-land as the Global Interpreter Lock, or GIL: https://wiki.python.org/moin/GlobalInterpreterLock.
It may seem strange, therefore, that the code above even works, since there’s nothing that even remotely looks like a “GIL-acquiring” sequence of instructions.

The reason the code above used to work is because IDAPython was not releasing the GIL: it was acquiring it at startup-time, and never releasing it. Ever.

IDA 6.5 & later versions

In IDA 6.5, many efforts have been made to improve the use of multithreading in IDAPython: The IDAPython plugin acquires the GIL just in time to execute CPython code and then releases it, as it should. This will permit a reliable use of threading in IDAPython (in some circumstances, when using multithreading, IDA < 6.5 could freeze because of the GIL not being released.)

Starting with 6.5, such a plugin would require to be written in the following manner:

void idaapi run(int)
PyGILState_STATE state = PyGILState_Ensure(); // Acquire the GIL
PyRun_SimpleString("print \"Hello from plugin\"");
PyGILState_Release(state); // Release the GIL

Note the use of the CPython-defined PyGILState_* helpers.

Existing plugins

Unfortunately, we couldn’t do anything in this case to ensure backwards-compatibility: existing plugins will have to be adapted & recompiled. Sorry about this, but we just could not find a reasonable solution to maintain bw-compat in this case; hence this blog post.

So far, we have received a grand total of 1 report for this issue, so this doesn’t appear to impact many of our users. But in case you are running into problems with your previously-compiled CPython-aware plugin, hopefully this will have explained why.

x64 decompiler not far away

Just a short post to show you the current state of the x64 decompiler. In fact, it already mostly works but we still have to solve some minor problems. Let us consider this source code:

struct color_t
  short red;
  short green;
  short blue;
  short alpha;

extern color_t lighten(color_t c);

color_t func(int red, int green, int blue, int alpha)
  color_t c;
  c.red = red;
  c.green = green;
  c.blue = blue;
  c.alpha = alpha;
  return lighten(c);

After compilation we get the following binary code:

.text:0000000000000000 ?func@@YA?AUcolor_t@@HHHH@Z proc near
c = color_t ptr -18h
.text:0000000000000000 var_10 = qword ptr -10h
.text:0000000000000000 arg_0 = dword ptr 8
.text:0000000000000000 arg_8 = dword ptr 10h
.text:0000000000000000 arg_10 = dword ptr 18h
.text:0000000000000000 arg_18 = dword ptr 20h
mov [rsp+arg_18], r9d ; $LN3
.text:0000000000000005 mov [rsp+arg_10], r8d
.text:000000000000000A mov [rsp+arg_8], edx
.text:000000000000000E mov [rsp+arg_0], ecx
.text:0000000000000012 sub rsp, 38h
.text:0000000000000016 movzx eax, word ptr [rsp+38h+arg_0]
.text:000000000000001B mov [rsp+38h+c.red], ax
.text:0000000000000020 movzx eax, word ptr [rsp+38h+arg_8]
.text:0000000000000025 mov [rsp+38h+c.green], ax
.text:000000000000002A movzx eax, word ptr [rsp+38h+arg_10]
.text:000000000000002F mov [rsp+38h+c.blue], ax
.text:0000000000000034 movzx eax, word ptr [rsp+38h+arg_18]
.text:0000000000000039 mov [rsp+38h+c.alpha], ax
.text:000000000000003E mov rcx, qword ptr [rsp+38h+c.red] ; c
.text:0000000000000043 call ?lighten@@YA?AUcolor_t@@U1@@Z ; lighten(color_t)
.text:0000000000000048 mov [rsp+38h+var_10], rax
.text:000000000000004D mov rax, [rsp+38h+var_10]
.text:0000000000000052 add rsp, 38h
.text:0000000000000056 retn
.text:0000000000000056 ?func@@YA?AUcolor_t@@HHHH@Z endp

Please note that the c, which is a structure, is passed by value in 2 registers: rcx and rdx. We had to rework quite many things in the decompiler to support such variables (we call them scattered variables). However, the output was worth it:

color_t __fastcall func(__int16 cx0, __int16 dx0, __int16 r8_0, __int16 r9_0)
  color_t c;

  c.red = cx0;
  c.green = dx0;
  c.blue = r8_0;
  c.alpha = r9_0;
  return lighten(c);

There is still some work to be done, but it seems we solved most problematic issues. Stay tuned, there will be more decompiler news soon!


Interacting with IDA through IPC channels

I’m happy to present you a guest post by David Zimmer <dzzie@yahoo.com>. The approach he describes can be used to develop plugins more conveniently (but not limited to that):

In this article we are going to discuss a mechanism that can be used to interact with IDA through external applications.

The reason this technique was developed was to provide a convenient way for utility applications to query information from IDA databases, and automate its interface.

Over the years, several methods have been tried such as pipes, and sockets. In the end, the easiest Inter-Process Communication (IPC) technique I have found is the Windows specific SendMessage API along with the WM_COPYDATA message. This technique was chosen for its simplicity, reliability, and its inherently synchronous nature.

With this technique, the IDA server plugin creates a window and watches for command messages to come in.

(abbreviated C source to CreateWindow and receive messages)

The plugin contains its own text based API which can be used to automate actions and return queried data back to the calling process. Example commands and data transfer routines are shown below:

(hwnd arguments specify which window handle receives data callback)

There are several advantages to this technique. Traditional plugin development usually requires compiling your code and launching the plugin from the host application.
Debugging often entails wading through runtime debug logs, inferring problems through observed behavior, and chasing crashs in a debugger.

This technique allows you to debug your executable in the development IDE as normal, harnessing all of its powerful capabilities such as edit and continue, call stack viewing, variable watches, breakpoints etc.

Complex projects can be run directly while you iron out interface behaviors and are not dependant upon the host. Datasets can even be loaded from test files so that many routines can be debugged independently of IDA.

One project where this technique was put to good use was an experiment to see if a Javascript IDE could be created for IDA automation tasks. The desire was for a scriptable interface that supported intellisense, function prototype tool tips, and syntax highlighting.


With a project such as this, coding for the IDE behaviors is a major part of the development task. To develop such an application strictly as a plugin would be a daunting endeavor. Developing it as a standalone application was a great advantage where it could be run directly from the Visual Studio IDE without any intermediate steps.

One other advantage to this technique is that it opens up plugin development to include higher level languages that do not have native support for the IDA API*. Languages such as C#, Delphi, and VB6 can easily interact with IDA through this mechanism. These languages have excellent rapid GUI development capabilities and a wealth of complex components already available to them. This technique is even open to Java developers through the JNI.

Once the intermediate API and the client access library is written, creating utilities to integrate with IDA becomes a pretty quick process. Another example project that has come in handy is a Wingraph32 replacement that was coded in about a day.

The interface shown below automatically syncs the IDA disassembly when graph nodes are clicked and can perform several renaming operations.


There are some limitations to this technique as well. The particular implementation detailed in this article is Windows specific. It also requires both applications to be running on the same machine.

Another consideration is that the data is crossing process boundaries which can be a performance hit depending on what is being transferred and how often it is being invoked. For example, extracting or patching large blocks of data would best be optimized by reserving the use of the SendMessage API as the command channel, and utilizing files or shared memory for the data transfer. This will likely be an optimization included in future revisions.

The example IDASRVR project linked to below currently supports 36 API and uses a simple text based command protocol. Sample code is provided in C and C#. Client libraries are also available for C# and VB6 in the associated projects below.

Sample Projects:

IDASrvr – IDA IPC Server example

IDA_Jscript – Javascript IDE PoC for IDA (VB6)

GleeGraph – C# Wingraph32 replacement

* You can create C# and VB6 in process plugins for IDA through COM.
This was my first approach, and was used in my
IDACompare plugin.

Loading your own modules from your IDAPython scripts with idaapi.require()


If you were using import to import your own “currently-in-development” modules from your IDAPython scripts, you may want to use idaapi.require(), starting with IDA 6.5.


When using IDAPython scripts, users were sometimes facing the following issue


  • User loads script
  • Script imports user’s module mymodule
  • Script ends
  • User modifies code of mymodule (Note: the module is modified, not the script)
  • User reloads script
  • Modifications to mymodule aren’t taken into consideration.

While that’s perfectly understandable (the python runtime doesn’t have to reload mymodule if it has been compiled & loaded already), this is somewhat of an annoyance for users that were importing modules that were often modified.

IDA <= 6.4: Ensuring a user-specified module gets reloaded, by destroying it.

Up until IDA 6.4, the IDAPython plugin would do some magic after you have run your user script.
(click “expand all” to reveal the diff)

The sequence becomes:

  • User loads script
  • Script imports user’s module mymodule
  • Script ends
  • [module mymodule is deleted]
  • User modifies code of mymodule
  • User reloads script
  • Modifications to mymodule are taken into consideration, since module was deleted.

Unfortunately we have to stop doing this because:

  • That prevents us from using python-based hooks to be used after the script is finished (see below).
  • That goes against the rest of the python philosophy (i.e., modifications to objects are not reverted), and is therefore unexpected.

Issues with hooks.

Imagine you have the following script, dbghooks.py:

from idaapi import *
import mydbghelpers

class MyHooks(DBG_Hooks):

  def __init__(self):

  def dbg_bpt(self, tid, ea):
    return 0

  def dbg_step_into(self):

hooks = MyHooks()
  • User loads script
  • Scripts imports mydbghelpers
  • Script creates instance of MyHooks, and hooks it into IDA’s debugger APIs
  • Script ends
  • [module mydbghelpers is deleted]
  • User runs debugger, and a breakpoint is hit. Two things can happen:
    • The hook fails executing
    • IDA crashes (that can happen if the form from mydbghelpers import * was used)

IDA > 6.4: Introducing idaapi.require()

Everywhere else in python, when you modify a runtime object, those changes will remain visible.

We decided it would be better to not go against that standard behaviour anymore, and provide a helper to achieve the same results as what was achieved before with the deletion of user modules.

You can now import & re-import of a module with: idaapi.require(name)

Here is its definition:

def require(modulename):
    if modulename in sys.modules.keys():
        import importlib
        import inspect
        m = importlib.import_module(modulename)
        frame_obj, filename, line_number, function_name, lines, index = inspect.stack()[1]
        importer_module = inspect.getmodule(frame_obj)
        if importer_module is None: # No importer module; called from command line
            importer_module = sys.modules['__main__']
        setattr(importer_module, modulename, m)
        sys.modules[modulename] = m


The example debugger hooks script above becomes:

from idaapi import *

class MyHooks(DBG_Hooks):

  def __init__(self):

  def dbg_bpt(self, tid, ea):
    return 0

  def dbg_step_into(self):

hooks = MyHooks()

I.e., only the second line changes.

Installing PIP packages, and using them from IDA on a 64-bit machine

Recently, one of our customers came to us asking how he should proceed to be able to install python packages, using PIP, and use those from IDA.

The issue he was facing is that his system is a 64-bit Ubuntu 12.04 VM.
Therefore using the Ubuntu-bundled PIP will just result in installing the desired package (let’s say ssdeep) for the system Python runtime, which is a 64-bit runtime and therefore not compatible with IDA.

The best (as in: cleanest) solution I have found is to:

  • build a 32-bits python on the system.
  • pip-install packages in that 32-bits python’s sub-directories.
  • export PYTHONPATH to point to the 32-bits python’s sub-directories.

We figured we’d write it down here just in case it might help others.


  • Install autoconf
  • Install ia32-libs

Building & installing a 32-bits python

  • ..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH
  • Download Python2.7.4
    • Note:You should make sure that the MD5 checksum and the size of the file you downloaded match those that are advertised on the page. That would prevent a man-in-the-middle attacker from providing you with a malicious Python bundle.
  • Build it. Note that you’ll probably have to sudo-create a few symlinks. I had to do this, on the Ubuntu 12.04 64-bit VM I tested this on:
    • /lib/i386-linux-gnu/libssl.so/lib/i386-linux-gnu/libssl.so.1.0.0
    • /lib/i386-linux-gnu/libcrypto.so/lib/i386-linux-gnu/libcrypto.so.1.0.0
    • /lib/i386-linux-gnu/libz.so/lib/i386-linux-gnu/libz.so.1
  • For the sake of completeness, here are my build commands (don’t forget the flags, of course):
    • ..$ CFLAGS=-m32 LDFLAGS=-m32 ./configure --prefix=/opt/Python2.7.4-32bits
    • ..$ CFLAGS=-m32 LDFLAGS=-m32 make -j 8

Once the build completes

Here’s what I have as last lines of the build:

INFO: Can't locate Tcl/Tk libs and/or headers

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _curses            _curses_panel
_sqlite3           _tkinter           bsddb185
bz2                dbm                gdbm
readline           sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

If you see, below that, that it failed to build, say 'binascii', then something went wrong.

Make sure you run make -j 1 to check out what went wrong (i.e., what library it claims not being able to find)

Once you have succesfully built your 32-bits Python, it’s time to install it: sudo make install

Trying your freshly-built python

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii

No complaint so far. Good.

Checking that pkg_resources is available.

Try importing pkg_resources. If it fails, you’ll probably have to do the following:

..$ cd /tmp
..$ curl -O http://python-distribute.org/distribute_setup.py
..$ less distribute_setup.py  # (*)
..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 distribute_setup.py

That will print out quite a fair amount of info, and should succeed.

(*) Note: A careful reader has pointed out that it would be fairly easy to intercept (man-in-the-middle) such an HTTP request, and serve malicious content that would then be piped (as root) to Python.
That’s why I think it’s important to mention, as a third step (i.e., less ...), that the code that was downloaded should ideally be checked. Hopefully, http://python-distribute.org will soon provide HTTPS support, which will limit such MITM attack risks.

Trying your freshly-built python, again

We want to make sure pkg_resources can be imported.

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources

Still no complaint. Good.

If yours complains, you’ll have to first make sure you fix whatever is causing it to fail, because the next will not work without that.

Installing PIP for your new Python build

Since using your system’s PIP will probably not work (as it would build & install things in a 64-bits python sub-directory), you’ll have to install a PIP package specifically for your freshly-built Python.

Here’s how I proceeded:

..$ cd /tmp;
..$ curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py;
..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 get-pip.py

PIP is now installed.

PIP-installing a package (i.e., ssdeep)

To download/build/install the ssdeep package I ran, as root (either that, or you’ll have to give your user the rights to write in /opt/Python2.7.4-32bits):

..$ su
root ..$ export CFLAGS=-m32
root ..$ export LDFLAGS=-m32
root ..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH
root ..$ /opt/Python2.7.4-32bits/bin/python2.7 /opt/Python2.7.4-32bits/bin/pip install ssdeep

Notice how I use my freshly-built python, with my fresly-installed PIP (and not the system one.)

Note: Don’t forget the export lines, or PIP will partially build stuff for x64, and partially for x86. That, as you can guess, won’t quite work.

If you forgot the export lines and started building anyway (and the build failed because of the mixed architecture issue I just wrote about), make sure you delete whatever is in /tmp/pip-build-*, so that there won’t be stale object files of inappropriate architecture in there.

Check out the PIP-installed package works

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssdeep
>>> ssdeep
<module 'ssdeep' from '/opt/Python2.7.4-32bits/lib/python2.7/site-packages/ssdeep.so'>
>>> dir(ssdeep)
['Error', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__test__', '__version__', 'compare', 'hash', 'hash_from_file', 'sys']

So far so good.

Testing the PIP-installed package in IDA

Since that’s still the goal (though you might have forgotten by now, given the amount of directions above.. 😉 ),
we’ll now try and make use of that PIP-installed package in IDA.

  • ..$ export PYTHONPATH=/opt/Python2.7.4-32bits/lib/python2.7/site-packages:/opt/Python2.7.4-32bits/:$PYTHONPATH
  • ..$ idaq

If all went well, typing import ssdeep in the Python input line should properly, silently, nicely import the package.

Recon 2012: Compiler Internals

This year I again was lucky to present at Recon in Montreal. There were many great talks as usual. I combined the topic of my last year’s talk on C++ reversing and my OpenRCE article on Visual C++ internals. New material was implementation of exceptions and RTTI in MSVC x64 and GCC (including Apple’s iOS).

The videos are not up yet but here are the slides of my presentation and a few demo scripts I made for it to parse GCC’s RTTI structures and exception tables. I also added my old scripts from OpenRCE which I amended slightly for the current IDA versions (mostly changed hotkeys).


Calling IDA APIs from IDAPython with ctypes

IDAPython provides wrappers for a big chunk of IDA SDK. Still, there are some APIs that are not wrapped because of SWIG limitations or just because we didn’t get to them yet. Recently, I needed to test the get_loader_name() API which is not available in IDAPython but I didn’t want to write a full plugin just for one call. For such cases it’s often possible to use the ctypes module to call the function manually.

The IDA APIs are provided by the kernel dynamic library. In Windows, it’s called ida.wll (or ida64.wll), in Linux libida[64].so and on OS X libida[64].dylib. ctypes provides a nice feature that dynamically creates a callable wrapper for a DLL export by treating it as an attribute of a special class instance. Here’s how to get that instance under the three platforms supported by IDA:

import ctypes
idaname = "ida64" if __EA64__ else "ida"
if sys.platform == "win32":
    dll = ctypes.windll[idaname + ".wll"]
elif sys.platform == "linux2":
    dll = ctypes.cdll["lib" + idaname + ".so"]
elif sys.platform == "darwin":
    dll = ctypes.cdll["lib" + idaname + ".dylib"]

We use “windll” because IDA APIs use stdcall calling convention on Windows (check the definition of idaapi in pro.h).

Now we just need to call our function just as if it was an attribute of the “dll” object. But first we need to prepare the arguments. Here’s the declaration from loader.hpp:

idaman ssize_t ida_export get_loader_name(char *buf, size_t bufsize);

ctypes provides a convenience functions for creating character buffers:

buf = ctypes.create_string_buffer(256)

And now we can call the function:

dll.get_loader_name(buf, 256)

To retrieve the contents of the buffer as a Python byte string, just use its .raw attribute. The complete script now looks like this:

import ctypes
idaname = "ida64" if __EA64__ else "ida"
if sys.platform == "win32":
    dll = ctypes.windll[idaname + ".wll"]
elif sys.platform == "linux2":
    dll = ctypes.cdll["lib" + idaname + ".so"]
elif sys.platform == "darwin":
    dll = ctypes.cdll["lib" + idaname + ".dylib"]
buf = ctypes.create_string_buffer(256)
dll.get_loader_name(buf, 256)
print "loader:", buf.raw

ctypes offers many means to interface with C code, so you can use it to call almost any IDA API.

The trace replayer

One of the new features that will be available in the next version of IDA is a trace re-player. This pseudo-debugger allows to re-play execution traces of programs debugged in IDA. The replayer debugger allows replaying traces recorded with any of the currently supported debuggers, ranging from local Linux or win32 debuggers to remote GDB targets. Currently supported targets include x86, x86_64, ARM, MIPS and PPC.

When we are re-playing a recorded trace, we can step forward and backward, set breakpoints, inspect register values, change the instruction pointer to any recorded IP, etc…

Also, trace management capabilities have been added to IDA in order to allow saving and loading recorded execution traces. Let’s see an example.

Continue reading The trace replayer