Loading your own modules from your IDAPython scripts with idaapi.require()

TL;DR

If you were using import to import your own “currently-in-development” modules from your IDAPython scripts, you may want to use idaapi.require(), starting with IDA 6.5.

Rationale

When using IDAPython scripts, users were sometimes facing the following issue

Specifically:

  • User loads script
  • Script imports user’s module mymodule
  • Script ends
  • User modifies code of mymodule (Note: the module is modified, not the script)
  • User reloads script
  • Modifications to mymodule aren’t taken into consideration.

While that’s perfectly understandable (the python runtime doesn’t have to reload mymodule if it has been compiled & loaded already), this is somewhat of an annoyance for users that were importing modules that were often modified.

IDA <= 6.4: Ensuring a user-specified module gets reloaded, by destroying it.

Up until IDA 6.4, the IDAPython plugin would do some magic after you have run your user script.
(click “expand all” to reveal the diff)

The sequence becomes:

  • User loads script
  • Script imports user’s module mymodule
  • Script ends
  • [module mymodule is deleted]
  • User modifies code of mymodule
  • User reloads script
  • Modifications to mymodule are taken into consideration, since module was deleted.

Unfortunately we have to stop doing this because:

  • That prevents us from using python-based hooks to be used after the script is finished (see below).
  • That goes against the rest of the python philosophy (i.e., modifications to objects are not reverted), and is therefore unexpected.

Issues with hooks.

Imagine you have the following script, dbghooks.py:

from idaapi import *
import mydbghelpers

class MyHooks(DBG_Hooks):

  def __init__(self):
    ...

  def dbg_bpt(self, tid, ea):
    mydbghelpers.do_something()
    return 0

  def dbg_step_into(self):
    ...

hooks = MyHooks()
hooks.hook()
  • User loads script
  • Scripts imports mydbghelpers
  • Script creates instance of MyHooks, and hooks it into IDA’s debugger APIs
  • Script ends
  • [module mydbghelpers is deleted]
  • User runs debugger, and a breakpoint is hit. Two things can happen:
    • The hook fails executing
    • IDA crashes (that can happen if the form from mydbghelpers import * was used)

IDA > 6.4: Introducing idaapi.require()

Everywhere else in python, when you modify a runtime object, those changes will remain visible.

We decided it would be better to not go against that standard behaviour anymore, and provide a helper to achieve the same results as what was achieved before with the deletion of user modules.

You can now import & re-import of a module with: idaapi.require(name)

Here is its definition:

def require(modulename):
    if modulename in sys.modules.keys():
        reload(sys.modules[modulename])
    else:
        import importlib
        import inspect
        m = importlib.import_module(modulename)
        frame_obj, filename, line_number, function_name, lines, index = inspect.stack()[1]
        importer_module = inspect.getmodule(frame_obj)
        if importer_module is None: # No importer module; called from command line
            importer_module = sys.modules['__main__']
        setattr(importer_module, modulename, m)
        sys.modules[modulename] = m

EDIT (September 16th, 2016): After one of our users has reported an issue with this, we have updated the definition to:

def require(modulename, package=None):
    import inspect
    frame_obj, filename, line_number, function_name, lines, index = inspect.stack()[1]
    importer_module = inspect.getmodule(frame_obj)
    if importer_module is None: # No importer module; called from command line
        importer_module = sys.modules['__main__']
    if modulename in sys.modules.keys():
        reload(sys.modules[modulename])
        m = sys.modules[modulename]
    else:
        import importlib
        m = importlib.import_module(modulename, package)
        sys.modules[modulename] = m
    setattr(importer_module, modulename, m)

The problem with the previous definition, is that if module A loads module C, and then module A loads module B which, itself, also loads module C, then module A will have the module C properly bound to its globals(), but module B won’t (since the module was already present in sys.modules, we were only reloading it, not setting the attribute in module B)

Example

The example debugger hooks script above becomes:

from idaapi import *
idaapi.require("mydbghelpers")

class MyHooks(DBG_Hooks):

  def __init__(self):
    ...

  def dbg_bpt(self, tid, ea):
    mydbghelpers.do_something()
    return 0

  def dbg_step_into(self):
    ...

hooks = MyHooks()
hooks.hook()

I.e., only the second line changes.

Installing PIP packages, and using them from IDA on a 64-bit machine

Recently, one of our customers came to us asking how he should proceed to be able to install python packages, using PIP, and use those from IDA.

The issue he was facing is that his system is a 64-bit Ubuntu 12.04 VM.
Therefore using the Ubuntu-bundled PIP will just result in installing the desired package (let’s say ssdeep) for the system Python runtime, which is a 64-bit runtime and therefore not compatible with IDA.

The best (as in: cleanest) solution I have found is to:

  • build a 32-bits python on the system.
  • pip-install packages in that 32-bits python’s sub-directories.
  • export PYTHONPATH to point to the 32-bits python’s sub-directories.

We figured we’d write it down here just in case it might help others.

Prerequisites

  • Install autoconf
  • Install ia32-libs

Building & installing a 32-bits python

  • ..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH
  • Download Python2.7.4
    • Note:You should make sure that the MD5 checksum and the size of the file you downloaded match those that are advertised on the page. That would prevent a man-in-the-middle attacker from providing you with a malicious Python bundle.
  • Build it. Note that you’ll probably have to sudo-create a few symlinks. I had to do this, on the Ubuntu 12.04 64-bit VM I tested this on:
    • /lib/i386-linux-gnu/libssl.so/lib/i386-linux-gnu/libssl.so.1.0.0
    • /lib/i386-linux-gnu/libcrypto.so/lib/i386-linux-gnu/libcrypto.so.1.0.0
    • /lib/i386-linux-gnu/libz.so/lib/i386-linux-gnu/libz.so.1
  • For the sake of completeness, here are my build commands (don’t forget the flags, of course):
    • ..$ CFLAGS=-m32 LDFLAGS=-m32 ./configure --prefix=/opt/Python2.7.4-32bits
    • ..$ CFLAGS=-m32 LDFLAGS=-m32 make -j 8

Once the build completes

Here’s what I have as last lines of the build:

INFO: Can't locate Tcl/Tk libs and/or headers

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _curses            _curses_panel
_sqlite3           _tkinter           bsddb185
bz2                dbm                gdbm
readline           sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

If you see, below that, that it failed to build, say 'binascii', then something went wrong.

Make sure you run make -j 1 to check out what went wrong (i.e., what library it claims not being able to find)

Once you have succesfully built your 32-bits Python, it’s time to install it: sudo make install

Trying your freshly-built python

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii
>>>

No complaint so far. Good.

Checking that pkg_resources is available.

Try importing pkg_resources. If it fails, you’ll probably have to do the following:

..$ cd /tmp
..$ curl -O http://python-distribute.org/distribute_setup.py
..$ less distribute_setup.py  # (*)
..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 distribute_setup.py

That will print out quite a fair amount of info, and should succeed.

(*) Note: A careful reader has pointed out that it would be fairly easy to intercept (man-in-the-middle) such an HTTP request, and serve malicious content that would then be piped (as root) to Python.
That’s why I think it’s important to mention, as a third step (i.e., less ...), that the code that was downloaded should ideally be checked. Hopefully, http://python-distribute.org will soon provide HTTPS support, which will limit such MITM attack risks.

Trying your freshly-built python, again

We want to make sure pkg_resources can be imported.

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>>

Still no complaint. Good.

If yours complains, you’ll have to first make sure you fix whatever is causing it to fail, because the next will not work without that.

Installing PIP for your new Python build

Since using your system’s PIP will probably not work (as it would build & install things in a 64-bits python sub-directory), you’ll have to install a PIP package specifically for your freshly-built Python.

Here’s how I proceeded:

..$ cd /tmp;
..$ curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py;
..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 get-pip.py

PIP is now installed.

PIP-installing a package (i.e., ssdeep)

To download/build/install the ssdeep package I ran, as root (either that, or you’ll have to give your user the rights to write in /opt/Python2.7.4-32bits):

..$ su
Password:
root ..$ export CFLAGS=-m32
root ..$ export LDFLAGS=-m32
root ..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH
root ..$ /opt/Python2.7.4-32bits/bin/python2.7 /opt/Python2.7.4-32bits/bin/pip install ssdeep

Notice how I use my freshly-built python, with my fresly-installed PIP (and not the system one.)

Note: Don’t forget the export lines, or PIP will partially build stuff for x64, and partially for x86. That, as you can guess, won’t quite work.

If you forgot the export lines and started building anyway (and the build failed because of the mixed architecture issue I just wrote about), make sure you delete whatever is in /tmp/pip-build-*, so that there won’t be stale object files of inappropriate architecture in there.

Check out the PIP-installed package works

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssdeep
>>> ssdeep
<module 'ssdeep' from '/opt/Python2.7.4-32bits/lib/python2.7/site-packages/ssdeep.so'>
>>> dir(ssdeep)
['Error', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__test__', '__version__', 'compare', 'hash', 'hash_from_file', 'sys']
>>>

So far so good.

Testing the PIP-installed package in IDA

Since that’s still the goal (though you might have forgotten by now, given the amount of directions above.. 😉 ),
we’ll now try and make use of that PIP-installed package in IDA.

  • ..$ export PYTHONPATH=/opt/Python2.7.4-32bits/lib/python2.7/site-packages:/opt/Python2.7.4-32bits/:$PYTHONPATH
  • ..$ idaq

If all went well, typing import ssdeep in the Python input line should properly, silently, nicely import the package.