Extending IDAPython in IDA 6.5: Be careful about the GIL

Target audience

You may want to read this if you have been writing an IDA C++ plugin, that itself uses the CPython runtime.

Prior art

In 2010, Elias Bachaalany wrote a blog post about extending IDAPython: http://www.hexblog.com/?p=126

Note that this is not about writing your own plugins in Python. Rather, that blog post instruct on how you may want to write a plugin that, itself, links against libpython27, so that it interacts nicely with IDAPython (which links against the same libpython27 library).

Before 6.5

The following plugin body was enough to have a working C++ plugin, that will execute a simple Python statement:

void idaapi run(int)
PyRun_SimpleString("print \"Hello from plugin\"");

Notes about the previous example

It is worth pointing out that, whenever one wants to execute any CPython code (such as the ‘PyRun_SimpleString’ call above), one must have acquired what is called in CPython-land as the Global Interpreter Lock, or GIL: https://wiki.python.org/moin/GlobalInterpreterLock.
It may seem strange, therefore, that the code above even works, since there’s nothing that even remotely looks like a “GIL-acquiring” sequence of instructions.

The reason the code above used to work is because IDAPython was not releasing the GIL: it was acquiring it at startup-time, and never releasing it. Ever.

IDA 6.5 & later versions

In IDA 6.5, many efforts have been made to improve the use of multithreading in IDAPython: The IDAPython plugin acquires the GIL just in time to execute CPython code and then releases it, as it should. This will permit a reliable use of threading in IDAPython (in some circumstances, when using multithreading, IDA < 6.5 could freeze because of the GIL not being released.)

Starting with 6.5, such a plugin would require to be written in the following manner:

void idaapi run(int)
PyGILState_STATE state = PyGILState_Ensure(); // Acquire the GIL
PyRun_SimpleString("print \"Hello from plugin\"");
PyGILState_Release(state); // Release the GIL

Note the use of the CPython-defined PyGILState_* helpers.

Existing plugins

Unfortunately, we couldn’t do anything in this case to ensure backwards-compatibility: existing plugins will have to be adapted & recompiled. Sorry about this, but we just could not find a reasonable solution to maintain bw-compat in this case; hence this blog post.

So far, we have received a grand total of 1 report for this issue, so this doesn’t appear to impact many of our users. But in case you are running into problems with your previously-compiled CPython-aware plugin, hopefully this will have explained why.

x64 decompiler not far away

Just a short post to show you the current state of the x64 decompiler. In fact, it already mostly works but we still have to solve some minor problems. Let us consider this source code:

struct color_t
  short red;
  short green;
  short blue;
  short alpha;

extern color_t lighten(color_t c);

color_t func(int red, int green, int blue, int alpha)
  color_t c;
  c.red = red;
  c.green = green;
  c.blue = blue;
  c.alpha = alpha;
  return lighten(c);

After compilation we get the following binary code:

.text:0000000000000000 ?func@@YA?AUcolor_t@@HHHH@Z proc near
c = color_t ptr -18h
.text:0000000000000000 var_10 = qword ptr -10h
.text:0000000000000000 arg_0 = dword ptr 8
.text:0000000000000000 arg_8 = dword ptr 10h
.text:0000000000000000 arg_10 = dword ptr 18h
.text:0000000000000000 arg_18 = dword ptr 20h
mov [rsp+arg_18], r9d ; $LN3
.text:0000000000000005 mov [rsp+arg_10], r8d
.text:000000000000000A mov [rsp+arg_8], edx
.text:000000000000000E mov [rsp+arg_0], ecx
.text:0000000000000012 sub rsp, 38h
.text:0000000000000016 movzx eax, word ptr [rsp+38h+arg_0]
.text:000000000000001B mov [rsp+38h+c.red], ax
.text:0000000000000020 movzx eax, word ptr [rsp+38h+arg_8]
.text:0000000000000025 mov [rsp+38h+c.green], ax
.text:000000000000002A movzx eax, word ptr [rsp+38h+arg_10]
.text:000000000000002F mov [rsp+38h+c.blue], ax
.text:0000000000000034 movzx eax, word ptr [rsp+38h+arg_18]
.text:0000000000000039 mov [rsp+38h+c.alpha], ax
.text:000000000000003E mov rcx, qword ptr [rsp+38h+c.red] ; c
.text:0000000000000043 call ?lighten@@YA?AUcolor_t@@U1@@Z ; lighten(color_t)
.text:0000000000000048 mov [rsp+38h+var_10], rax
.text:000000000000004D mov rax, [rsp+38h+var_10]
.text:0000000000000052 add rsp, 38h
.text:0000000000000056 retn
.text:0000000000000056 ?func@@YA?AUcolor_t@@HHHH@Z endp

Please note that the c, which is a structure, is passed by value in 2 registers: rcx and rdx. We had to rework quite many things in the decompiler to support such variables (we call them scattered variables). However, the output was worth it:

color_t __fastcall func(__int16 cx0, __int16 dx0, __int16 r8_0, __int16 r9_0)
  color_t c;

  c.red = cx0;
  c.green = dx0;
  c.blue = r8_0;
  c.alpha = r9_0;
  return lighten(c);

There is still some work to be done, but it seems we solved most problematic issues. Stay tuned, there will be more decompiler news soon!