IDAPython + PySide/PyQt: future plans

Intended audience

IDAPython plugin writers who are using the PySide Qt bindings.

PySide: some background

For some time now it has been possible, through IDAPython, to use PySide bindings to the Qt libraries that are shipped with IDA.

Those PySide bindings were first placed on Hex-Rays’s website and, since we noticed a considerable interest for them, we later decided to ship them with IDA (starting at IDA 6.6.)

What about PyQt?

The choice of PySide over PyQt was essentially due to incompatibilities between the licensing model of PyQt, and that of Hex-Rays.

The problem

PySide appears to have pretty much stopped evolving, and now remains stuck with Qt 4 (i.e., the 4.8 branch.)
PySide isn’t available for Qt 5 and, as of today, the last update of the PySide roadmap dates back to “March 26, 2013″.

There is still some activity on the PySide mailing list, but despite that, not much seems to be happening (and the latest release is from 1 year ago.)

This is very problematic for us as Qt 4 is becoming more and more irrelevant by the day, and we would love to move to Qt 5.

We could, in theory, invest in porting PySide to support Qt 5, but it would probably lead us to an undesirable situation where PySide is pretty much used and maintained by Hex-Rays only.

Meanwhile, in PyQt land

In the meantime, the PyQt people at http://www.riverbankcomputing.co.uk/ have changed their licensing options, and after having contacted them recently, we have found that we can now consider the following:

  • Hex-Rays can acquire PyQt licenses
  • with those, we can build PyQt, and ship it with IDA
  • IDA users can make use of PyQt themselves

In other words, we will be able to provide a replacement for PySide, that:

  • is widely used, and has momentum
  • supports Qt 5

Our plan

Our plan is to roll, in a few days, IDA 6.8 that still depends on Qt 4, and still ships with PySide.

The release after that (i.e., 6.9) will then depend on Qt 5, and come with PyQt.

What that means for developers

Thankfully, the interfaces of PySide and PyQt are highly compatible, and thus it should not be necessary to rewrite your plugins from scratch (in fact, very little should have to be modified.)

I would recommend you to stay tuned to the Hex-Rays forums, so that you can participate in the beta program for IDA 6.9 once it is announced. That will give you the possibility of porting your tools to PyQt before IDA 6.9 is released. We will also provide guidance on how to approach that porting effort (even though, once again, it should be fairly small.)

As you may already know, we are very reluctant to any kind of API/compatibility breakage. But unfortunately, it seems we simply don’t have a choice in this particular case.

Installing PIP packages, and using them from IDA on a 64-bit machine

Recently, one of our customers came to us asking how he should proceed to be able to install python packages, using PIP, and use those from IDA.

The issue he was facing is that his system is a 64-bit Ubuntu 12.04 VM.
Therefore using the Ubuntu-bundled PIP will just result in installing the desired package (let’s say ssdeep) for the system Python runtime, which is a 64-bit runtime and therefore not compatible with IDA.

The best (as in: cleanest) solution I have found is to:

  • build a 32-bits python on the system.
  • pip-install packages in that 32-bits python’s sub-directories.
  • export PYTHONPATH to point to the 32-bits python’s sub-directories.

We figured we’d write it down here just in case it might help others.

Prerequisites

  • Install autoconf
  • Install ia32-libs

Building & installing a 32-bits python

  • ..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH
  • Download Python2.7.4
    • Note:You should make sure that the MD5 checksum and the size of the file you downloaded match those that are advertised on the page. That would prevent a man-in-the-middle attacker from providing you with a malicious Python bundle.
  • Build it. Note that you’ll probably have to sudo-create a few symlinks. I had to do this, on the Ubuntu 12.04 64-bit VM I tested this on:
    • /lib/i386-linux-gnu/libssl.so/lib/i386-linux-gnu/libssl.so.1.0.0
    • /lib/i386-linux-gnu/libcrypto.so/lib/i386-linux-gnu/libcrypto.so.1.0.0
    • /lib/i386-linux-gnu/libz.so/lib/i386-linux-gnu/libz.so.1
  • For the sake of completeness, here are my build commands (don’t forget the flags, of course):
    • ..$ CFLAGS=-m32 LDFLAGS=-m32 ./configure --prefix=/opt/Python2.7.4-32bits
    • ..$ CFLAGS=-m32 LDFLAGS=-m32 make -j 8

Once the build completes

Here’s what I have as last lines of the build:

INFO: Can't locate Tcl/Tk libs and/or headers

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _curses            _curses_panel
_sqlite3           _tkinter           bsddb185
bz2                dbm                gdbm
readline           sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

If you see, below that, that it failed to build, say 'binascii', then something went wrong.

Make sure you run make -j 1 to check out what went wrong (i.e., what library it claims not being able to find)

Once you have succesfully built your 32-bits Python, it’s time to install it: sudo make install

Trying your freshly-built python

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii
>>>

No complaint so far. Good.

Checking that pkg_resources is available.

Try importing pkg_resources. If it fails, you’ll probably have to do the following:

..$ cd /tmp
..$ curl -O http://python-distribute.org/distribute_setup.py
..$ less distribute_setup.py  # (*)
..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 distribute_setup.py

That will print out quite a fair amount of info, and should succeed.

(*) Note: A careful reader has pointed out that it would be fairly easy to intercept (man-in-the-middle) such an HTTP request, and serve malicious content that would then be piped (as root) to Python.
That’s why I think it’s important to mention, as a third step (i.e., less ...), that the code that was downloaded should ideally be checked. Hopefully, http://python-distribute.org will soon provide HTTPS support, which will limit such MITM attack risks.

Trying your freshly-built python, again

We want to make sure pkg_resources can be imported.

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>>

Still no complaint. Good.

If yours complains, you’ll have to first make sure you fix whatever is causing it to fail, because the next will not work without that.

Installing PIP for your new Python build

Since using your system’s PIP will probably not work (as it would build & install things in a 64-bits python sub-directory), you’ll have to install a PIP package specifically for your freshly-built Python.

Here’s how I proceeded:

..$ cd /tmp;
..$ curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py;
..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 get-pip.py

PIP is now installed.

PIP-installing a package (i.e., ssdeep)

To download/build/install the ssdeep package I ran, as root (either that, or you’ll have to give your user the rights to write in /opt/Python2.7.4-32bits):

..$ su
Password:
root ..$ export CFLAGS=-m32
root ..$ export LDFLAGS=-m32
root ..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH
root ..$ /opt/Python2.7.4-32bits/bin/python2.7 /opt/Python2.7.4-32bits/bin/pip install ssdeep

Notice how I use my freshly-built python, with my fresly-installed PIP (and not the system one.)

Note: Don’t forget the export lines, or PIP will partially build stuff for x64, and partially for x86. That, as you can guess, won’t quite work.

If you forgot the export lines and started building anyway (and the build failed because of the mixed architecture issue I just wrote about), make sure you delete whatever is in /tmp/pip-build-*, so that there won’t be stale object files of inappropriate architecture in there.

Check out the PIP-installed package works

..$ /opt/Python2.7.4-32bits/bin/python2.7
Python 2.7.4 (default, Apr 26 2013, 16:03:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssdeep
>>> ssdeep
<module 'ssdeep' from '/opt/Python2.7.4-32bits/lib/python2.7/site-packages/ssdeep.so'>
>>> dir(ssdeep)
['Error', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__test__', '__version__', 'compare', 'hash', 'hash_from_file', 'sys']
>>>

So far so good.

Testing the PIP-installed package in IDA

Since that’s still the goal (though you might have forgotten by now, given the amount of directions above.. 😉 ),
we’ll now try and make use of that PIP-installed package in IDA.

  • ..$ export PYTHONPATH=/opt/Python2.7.4-32bits/lib/python2.7/site-packages:/opt/Python2.7.4-32bits/:$PYTHONPATH
  • ..$ idaq

If all went well, typing import ssdeep in the Python input line should properly, silently, nicely import the package.

Recon 2012: Compiler Internals

This year I again was lucky to present at Recon in Montreal. There were many great talks as usual. I combined the topic of my last year’s talk on C++ reversing and my OpenRCE article on Visual C++ internals. New material was implementation of exceptions and RTTI in MSVC x64 and GCC (including Apple’s iOS).

The videos are not up yet but here are the slides of my presentation and a few demo scripts I made for it to parse GCC’s RTTI structures and exception tables. I also added my old scripts from OpenRCE which I amended slightly for the current IDA versions (mostly changed hotkeys).

Slides
Scripts

The trace replayer

One of the new features that will be available in the next version of IDA is a trace re-player. This pseudo-debugger allows to re-play execution traces of programs debugged in IDA. The replayer debugger allows replaying traces recorded with any of the currently supported debuggers, ranging from local Linux or win32 debuggers to remote GDB targets. Currently supported targets include x86, x86_64, ARM, MIPS and PPC.

When we are re-playing a recorded trace, we can step forward and backward, set breakpoints, inspect register values, change the instruction pointer to any recorded IP, etc…

Also, trace management capabilities have been added to IDA in order to allow saving and loading recorded execution traces. Let’s see an example.

Continue reading The trace replayer

Recon 2011: Practical C++ Decompilation

Last month I visited the Recon conference and had a great time again. I gave a talk on C++ decompilation and how to handle it in IDA and Hex-Rays decompiler. You can get the slides here, and download the recorded talk here.

Edit: for some reason the streaming version does not show anything after the intro, please download the Quicktime version until it’s fixed.

 

Challenging job for software developers

We should permanently and prominently publish this ad on our site :)

We are looking for strong software engineers to join our team and participate in the development of unique software security tools. The candidates must know low-level details of modern software as well as high-level data structures and algorithms.

Requirements:

  • strong knowledge of C/C++
  • knowledge of the x86 assembler and unwillingness to use it in development
  • cross platform development (Windows/Linux/Mac) is a plus
  • knowing the graph theory and how compilers work is a plus
  • ability and willingness to write secure yet fast code
  • good problem solving and communication skills

If you want a challenging job in a friendly environment, please apply by sending your resume to info@hex-rays.com
Thanks!

IDA Pro 5.5 and Hex-Rays 1.1 have been released!

IDA Pro 5.5

We are happy to announce a new version of IDA Pro! The major news is the
new docking user interface. There are many other improvements: processor modules,
file formats, analysis tweaks, well, the usual stuff. There is a new MS Windows
Crash Dump Loader and improved Bochs debugger. The complete list of new
features and bug fixes is available here

http://www.hex-rays.com/idapro/55/index.htm

Hex-Rays 1.1

We also release a new version of our decompiler: now with the floating point
support. It was a technically challenging task and required lots of testing, but
we are very happy with the end result. It can really handle floating point
computations and generates reliable output. All subtle nuances, like conversion
rules, fpu stack state, predefined compiler helper functions, are all taken care of.

The decompiler uses debug information if it is available: in this case, even local
variable names and types will be restored. If there is no debug information, the
decompiler will still generate correct and precise output. In fact, it is designed
to work without debug information, which means that virtually any
compiler-generated executable can be analyzed and turned into C output.

New pricing and support plans

With this release, we update the pricing of IDA Pro and Hex-Rays Decompiler.
While the initial purchase prices are increased, upgrade prices go down.
In order to streamline the upgrade process, we will use the same rules for
all our products: now a support plan is renewable any time while it is active
and also three months after its expiration. The new support period is counted from
the expiration date of the previous support period.

If you upgraded your IDA/Hex-Rays copy the last month with older prices,
do not worry. For you, we will add a month of support for the IDA license,
and three months of support for Hex-Rays Decompiler.

We will continue to accept old-style upgrade orders until 12 October 2009.

How to request the new versions

As usual, the new versions are free for users whose licenses are within active
support plan. Submit your ida.key to

https://www.hex-rays.com/updida.shtml

and expect a message from us within 5-10 minutes. Sometimes we do not have your
email in the database, so please specify it (otherwise we will have no means of
communicating with you).

To request the new version of the decompiler, please use Edit, Plugins, Hex-Rays,
Check for updates in IDA.

Is your key too old?

If your key is too old for a free update, you might still be
eligible for a discounted upgrade. Until 12 October 2009 we offer the upgrade
prices for all purchases made two years ago or less. The order forms can be
found here:

http://www.hex-rays.com/idapro/idaorder.htm

We will arrange an electronic delivery to existing customers.

That’s all folks! Enjoy the release.