News about the x64 edition

Sorry for the long silence since IDA v6.95, we all were incredibly busy with the transition to the 64-bit version. We are happy to say now that we are close to the finish line and will announce the beta test soon.

Transition to x64 itself was not that hard. We have been compiling IDA in x64 mode since many years, so making it actually work was a piece of cake.

It was much more time consuming to clean up the API: make it more logical, easier to use, and even remove some obsolete stuff that we have been carrying around since ages. Switching to the x64 is the unique opportunity for this cleanup because there are no existing x64 plugins yet and we won’t be breaking any working plugins. We hope that you will like the new API much more.

Just to give you an idea: we made more than 8000 commits to our source code repository and just the commit descriptions are more than 1MB. We reached 1.6M lines of source code without taking into account any third party or auto-generated files. I haven’t counted how many lines had changed since v6.95, but it can easily be that every second line got modified during the transition. In short, it was a huge undertaking.

While it was huge, we did not manage to make everything ideal (is it possible at all?…) We will continue to work on the API in the future.

Naturally, we were also busy fixing our past bugs (309 bugs since the public release of v6.95, to be exact; fortunately almost all of them reveal themselves in very specific circumstances).

We were also working on new features. Just to give you an idea of the new stuff, see very short descriptions below.

First of all, IDA fully switched to UTF-8. It will be possible to use Unicode strings everywhere, and specify any specific encoding for a string in the input file. The databases will be kept in UTF-8 as well, which will allow us to get rid of inconsistencies between Windows and Unix versions of IDA. In fact it is deeper that a simple support of UTF-8, we have a new system for international character support but it deserves a separate blog entry. We will talk about it in more detail after the release.

We will release the PowerPC 64-bit decompiler along with IDA v7. This assembler code (which did not fit into the screenshot):

Gets converted into one line:

We added support for exception handling. IDA recognizes try/except blocks and neatly comments them in the listing:

The iOS debugger improves support for debugging dylibs from dyld_shared_cache and adds support for source code level debugging.

We have tons of other improvements: better GDBServer support, updated FLAIR signatures, improved decompiler heuristics, updated built-in functions for IDAPython and IDC, new switch table patterns, etc.

Stay tuned, we will announce the beta test soon!

IDA 6.95: Qt 5.6.0 configure options & patch

A handful of our users have already requested information regarding the Qt 5.6.0 build, that is shipped with IDA 6.95.

Configure options

Here are the options that were used to build the libraries on:

  • Windows: ...\5.6.0\configure.bat "-nomake" "tests" "-qtnamespace" "QT" "-confirm-license" "-accessibility" "-opensource" "-force-debug-info" "-platform" "win32-msvc2015" "-opengl" "desktop" "-prefix" "C:/Qt/5.6.0"
    • Note that you will have to build with Visual Studio 2015, to obtain compatible libs
  • Linux: .../5.6.0/configure "-nomake" "tests" "-qtnamespace" "QT" "-confirm-license" "-accessibility" "-opensource" "-force-debug-info" "-platform" "linux-g++-32" "-developer-build" "-fontconfig" "-qt-freetype" "-qt-libpng" "-glib" "-qt-xcb" "-dbus" "-qt-sql-sqlite" "-gtkstyle" "-prefix" "/usr/local/Qt/5.6.0"
  • Mac OSX: .../5.6.0/build/../configure "-nomake" "tests" "-qtnamespace" "QT" "-confirm-license" "-accessibility" "-opensource" "-force-debug-info" "-platform" "macx-g++-32" "-debug-and-release" "-fontconfig" "-qt-freetype" "-qt-libpng" "-qt-sql-sqlite" "-prefix" "/Users/Shared/Qt/5.6.0"

patch

In addition to the specific configure options, the Qt build that ships with IDA includes the following patch. You should therefore apply it to your own Qt 5.6.0 sources before compiling, in order to obtain similar binaries.

Note that this patch should work without any modification, against the 5.6.0 release as found there. You may have to fiddle with it, if your Qt 5.6.0 sources come from somewhere else.

Installing IDA 6.95 on Linux

IDA is still, as of this writing (August 9th, 2015), a 32-bit application and both IDA & its installer(*) require certain 32-bit libraries to be present on your Linux system before they can run.

Here is the list of commands you will have to run in order to install those dependencies, for the following systems:

  • Debian & derivative systems such as Ubuntu, Xubuntu, …
  • Red Hat Enterprise Linux 7.2 (and likely other versions as well)
Note: we cannot possibly install & try IDA on all flavors/versions of all Linux distributions, but we will do our best to update this post with relevant information, whenever we learn of a distribution requiring special attention.

(*) that is: if you want the installer to run a graphical interface, instead of a command-line one.

Debian & Ubuntu

Common dependencies

The following should allow IDA to run on most Linux systems deriving from Debian distributions:

sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install libc6-i686:i386 libexpat1:i386 libffi6:i386 libfontconfig1:i386 libfreetype6:i386 libgcc1:i386 libglib2.0-0:i386 libice6:i386 libpcre3:i386 libpng12-0:i386 libsm6:i386 libstdc++6:i386 libuuid1:i386 libx11-6:i386 libxau6:i386 libxcb1:i386 libxdmcp6:i386 libxext6:i386 libxrender1:i386 zlib1g:i386 libx11-xcb1:i386 libdbus-1-3:i386 libxi6:i386 libsm6:i386 libcurl3:i386 
Note: one of our users kindly warned us that the package “libpng12-0:i386” has been replaced by “libpng16-16:i386” in Debian Stretch repositories. Consequently, you may have to change the last command above to:

sudo apt-get install libc6-i686:i386 libexpat1:i386 libffi6:i386 libfontconfig1:i386 libfreetype6:i386 libgcc1:i386 libglib2.0-0:i386 libice6:i386 libpcre3:i386 libpng16-16:i386 libsm6:i386 libstdc++6:i386 libuuid1:i386 libx11-6:i386 libxau6:i386 libxcb1:i386 libxdmcp6:i386 libxext6:i386 libxrender1:i386 zlib1g:i386 libx11-xcb1:i386 libdbus-1-3:i386 libxi6:i386 libsm6:i386 libcurl3:i386 

It is necessary to also run the following command, for IDA to present a usable & well-integrated GUI on many Debian & Ubuntu desktops. If the set of dependencies above are not enough to obtain a slick UI, please try the following:

sudo apt-get install libgtk2.0-0:i386 gtk2-engines-murrine:i386 gtk2-engines-pixbuf:i386 libpango1.0-0:i386

Red Hat Enterprise Linux 7.2

IDA will require the following packages to be installed, in order to run properly on RHEL 7.2 (and probably any other RPM-based distribution) :

redhat-lsb-core.i686
glib2.i686
libXext.i686
libXi.i686
libSM.i686
libICE.i686
freetype.i686
fontconfig.i686
dbus-libs.i686 

Arch Linux

While the author of this post is not quite familiar with Arch Linux, one of our users reported that adding the required dependencies can be performed through the AUR package that can be found at: https://aur.archlinux.org/packages/ida-pro-6.4/.

(And then, installing a 32-bit system-wide interpreter (i.e., if one favors that option over the default that consists of using the Python runtime shipped with IDA), should be performed by installing the package ‘lib32-python2’.)

IDA 6.9, Mac OS X, ‘random’ crashes

Intended audience

IDA 6.9 users on Mac OS X, who have suffered seemingly-apparent crashes while using IDA.

The problem

The Qt 5.4.1 libraries shipped with IDA 6.9 suffer from the following bug: https://bugreports.qt.io/browse/QTBUG-44708, which was apparently fixed in Qt 5.5.0.

If, when IDA crashes, you ever spotted a backtrace that looks like the following:

frame #0: 0x00000000
frame #1: 0x00d8a50d QtGui'QT::QTextEngine::shapeText(int) const + 1187
frame #2: 0x00d8b517 QtGui'QT::QTextEngine::shape(int) const + 1199
frame #3: 0x00d8c977 QtGui'QT::QTextEngine::width(int, int) const + 155
frame #4: 0x00d73571 QtGui'QT::QFontMetricsF::width(QT::QString const&) const + 163
frame #5: 0x00041184 idaq'___lldb_unnamed_function853$$idaq + 420
...

then you’ve been a victim of this rather tiresome issue.

(note: frame #0 doesn’t quite matter; the 2nd line, QT::QTextEngine::shapeText(int), is the important one)

The solution

We have applied the patch mentionned in the Qt bugreport & re-built the libqcocoa.dylib Qt platform support.

You will have to:

  1. download & unzip the following archive: qcocoa.zip.
  2. verify the shasum of the libqcocoa.dylib file:
    $ shasum libqcocoa.dylib 
    afcf3603f593776c6f39f41f81e98843897cf0ed  libqcocoa.dylib
    
  3. place the libqcocoa.dylib binary instead of the one in /path/to/IDA_6.9/idaq.app/Contents/Plugins/

Once that is done, those crashes shouldn’t happen anymore.

A big, big thank you to Willem Jan Hengeveld & Vladimir Putin, who have reported this!

Beware: IDA C++ plugins, Qt 5.x, QStringLiteral: crash at exit-time

Intended audience

IDA C++ plugin authors, who wish to link such plugins against Qt 5.x libraries.

The problem

One of our customers, Aliaksandr Trafimchuk, recently reported that whenever IDA was run with a plugin of his that links against the Qt libraries that we ship, IDA would crash at exit-time (at least on Windows.)

Aliaksandr already did most of the work of figuring out exactly what was causing the crash, and even had a work-around (more like a kludge, as he pointed out, really) for it, but he still wanted to let us know about it so we are aware of the problem & perhaps can communicate about it.

The crash is an access violation, in an area of memory that doesn’t seem to be mapped by any stack, heap, DLL code or data.

The stack reveals that the crash happens at QCoreApplication::~QCoreApplication()-time (i.e., at application exit), when the QFontCache is freeing/releasing its entries:

  Qt5Core.dll!QT::QSettingsGroup::~QSettingsGroup()
  Qt5Gui.dll!QT::QMapNode::destroySubTree()
  Qt5Gui.dll!QT::QFontCache::clear()
  Qt5Gui.dll!QT::QFontCache::~QFontCache()
  [External Code] 
  Qt5Gui.dll!QT::QThreadStorage::deleteData(void * x)
  Qt5Core.dll!QT::QThreadStorageData::set(void * p)
  Qt5Gui.dll!QT::QFont::cleanup()
  Qt5Gui.dll!QT::QGuiApplicationPrivate::~QGuiApplicationPrivate()
  [External Code] 
  Qt5Core.dll!QT::QObject::~QObject()
  Qt5Core.dll!QT::QCoreApplication::~QCoreApplication()
  idaq.exe!013426c5() Unknown
  (...)

Why does that happen?

Our customer’s plugin uses a UI description file, that needs to be processed by Qt’s uic (UI-compiler). The generated code contains lines such as these:

label = new QLabel(TestDialog);
label->setObjectName(QStringLiteral("label"));
QFont font;
font.setFamily(QStringLiteral("Comic Sans MS"));

Note the use of QStringLiteral.

The QStringLiteral type

This is an optimization that came in Qt 5.x, and that causes actual QString instances to be laid out in the .rodata section of the program (together with a special refcount value that is -1, meaning “don’t touch this refcount”.)

Although at exit-time, this “static const” in-.rodata-QString instance wouldn’t be modified (because of the -1 refcount), simply reading it will cause a crash, since the section holding it has been removed from memory.

This is a known limitation/problem, too: https://bugreports.qt.io/browse/QTBUG-46880

The plugin lifecycle

This is where the problem lies: at exit-time, IDA will:

  1. unload plugins
  2. proceed until its QCoreApplication goes out of scope, which will perform (among other things) the QFontCache cleanup.
  3. alas, at that time, the QFontCache still refers to literal QString data, in a section that is now gone (it was discarded at #1)

In fact, Qt expects that any binary that uses Qt libraries should remain in memory, so that some optimizations (such as the QStringLiteral) will continue to work. That’s why, when Qt unloads some of its own plugins, it doesn’t really unload those from memory.

Although the Qt library maintainers consider that having such limitations on binaries that link against Qt is acceptable, I personally hope they try to keep those restrictions as minimal as possible.

The solution

In any case, concerning this QStringLiteral issue, we have a way out: at compilation-time, pass the compiler the following flag: -DQT_NO_UNICODE_LITERAL

This will turn the QStringLiteral() expression into a QString::fromUtf8(), which will allocate the memory on the heap and the plugin should work just fine.


Another possible solution

Another possibility reported by an IDA user (but untested by us), is to add the following after the Qt headers #include directives:

#undef QStringLiteral
#define QStringLiteral(_str) _str

With this method, the literal C-style string will be implicitly converted to a QString, using the default conversion rules.


Footnotes

The kludge (which is Windows-specific) consists of calling LoadLibrary(szMyPluginFilePath), thereby somewhat artificially incrementing the refcount of his plugin, which will cause it to remain in memory & thus the ~QFontCache cleanup will succeed.

IDAPython: migrating PySide code to PyQt5

Background

Contrary to previous versions that shipped with Qt 4.8.4, IDA 6.9 ships with Qt 5.4.1 and as we announced some time ago, this will force some changes for plugin writers who were using IDAPython + PySide in order to build custom interfaces.

What’s more, in addition to the Qt4 -> Qt5 switch, we have also chosen to drop PySide in favor of PyQt5, as it seems to be more actively developed.

Porting

While we were porting some internal plugins & test scripts, we have come across a number of ‘issues’ that are due both to the change in major Qt version, and also to the switch to PyQt5.

Here is a (non-exhaustive) list of such problems that needed to be dealt with.

Use ‘from PyQt5’ instead of ‘from PySide’

   from PySide import QtCore, QtGui

Becomes:

   from PyQt5 import QtCore, QtGui, QtWidgets

(notice the added ‘QtWidgets’. Keep reading.)

Most widgets are now in QtWidgets, not QtGui anymore (Qt5 architecture change)

   from PySide import QtGui
   edit = QtGui.QTextEdit()

Becomes:

   from PyQt5 import QtWidgets
   edit = QtWidgets.QTextEdit()

Converting TForm instances must be done to PyQt

   from PySide import QtGui, QtCore
   form = idaapi.find_tform("Output window")
   w = idaapi.PluginForm.FormToPySideWidget(form)

Becomes:

   from PyQt5 import QtGui, QtCore, QtWidgets
   form = idaapi.find_tform("Output window")
   w = idaapi.PluginForm.FormToPyQtWidget(form)

Public QEvent types are in QtCore.QEvent namespace

   QtCore.QEvent.Type.XXX

Becomes:

   QtCore.QEvent.XXX

Using QObject.connect() to connect signal with slot doesn’t work anymore. Use alternative approach.

   obj.connect(query_action, QtCore.SIGNAL("triggered()"), self.ui.queryGraph)

Becomes:

   query_action.triggered.connect(self.ui.queryGraph)

Misc. required changes (reported by our users)

  • QHeaderView.setResizeMode => QHeaderView.setSectionResizeMode
  • QtCore.Qt.ItemFlag enum was moved to be accessed directly on QtCore.Qt
  • QColorDialog.setCustomColor now takes a QColor instead of an RGB int as the second argument.

Final words

As I said above, this is a non-exhaustive list of issues we encountered. Should you encounter other issues, please let us know about them, so we can update & improve this page to help the community!

IDA 6.9: Qt 5.4.1 configure options & patch

A handful of our users have already requested information regarding the Qt 5.4.1 build, that is shipped with IDA 6.9.

Configure options

Here are the options that were used to build the libraries on:

  • Windows: ...\5.4.1\configure.bat "-debug-and-release" "-nomake" "tests" "-qtnamespace" "QT" "-confirm-license" "-accessibility" "-platform" "win32-msvc2015" "-opengl" "desktop" "-force-debug-info" "-prefix" "C:/Qt/5.4.1"
    • Note that you will have to build with Visual Studio 2015, to obtain compatible libs
  • Linux: .../5.4.1/configure "-debug-and-release" "-nomake" "tests" "-qtnamespace" "QT" "-confirm-license" "-accessibility" "-platform" "linux-g++-32" "-developer-build" "-fontconfig" "-qt-freetype" "-qt-libpng" "-glib" "-qt-xcb" "-dbus" "-qt-sql-sqlite" "-gtkstyle" "-prefix" "/usr/local/Qt/5.4.1"
  • Mac OSX: .../5.4.1/configure "-debug-and-release" "-nomake" "tests" "-qtnamespace" "QT" "-confirm-license" "-accessibility" "-platform" "macx-g++-32" "-fontconfig" "-qt-freetype" "-qt-libpng" "-qt-sql-sqlite" "-prefix" "/Users/Shared/Qt/5.4.1"

patch

In addition to the specific configure options, the Qt build that ships with IDA includes the following patch. You should therefore apply it to your own Qt 5.4.1 sources before compiling, in order to obtain similar binaries.

Installing IDA 6.9 on Linux

IDA is still, as of this writing (December 23rd, 2015), a 32-bit application and both IDA & its installer(*) require certain 32-bit libraries to be present on your Linux system before they can run.

Here is the list of commands you will have to run in order to install those dependencies, for the following systems:

  • Debian & derivative systems such as Ubuntu, Xubuntu, …
  • Red Hat Enterprise Linux 7.2 (and likely other versions as well)
Note: we cannot possibly install & try IDA on all flavors/versions of all Linux distributions, but we will do our best to update this post with relevant information, whenever we learn of a distribution requiring special attention.

(*) that is: if you want the installer to run a graphical interface, instead of a command-line one.

Debian & Ubuntu

Common dependencies

The following should allow IDA to run on most Linux systems deriving from Debian distributions:

sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install libc6-i686:i386 libexpat1:i386 libffi6:i386 libfontconfig1:i386 libfreetype6:i386 libgcc1:i386 libglib2.0-0:i386 libice6:i386 libpcre3:i386 libpng12-0:i386 libsm6:i386 libstdc++6:i386 libuuid1:i386 libx11-6:i386 libxau6:i386 libxcb1:i386 libxdmcp6:i386 libxext6:i386 libxrender1:i386 zlib1g:i386 libx11-xcb1:i386 libdbus-1-3:i386 libxi6:i386 libsm6:i386 libcurl3:i386 

Xubuntu 15.10

It is necessary to also run those commands, for IDA to present a usable GUI on Xubuntu 15.10

sudo apt-get install libgtk2.0-0:i386 gtk2-engines-murrine:i386 gtk2-engines-pixbuf:i386 

Red Hat Enterprise Linux 7.2

IDA will require the following packages to be installed, in order to run properly on RHEL 7.2 (and probably any other RPM-based distribution) :

redhat-lsb-core.i686
glib2.i686
libXext.i686
libXi.i686
libSM.i686
libICE.i686
freetype.i686
fontconfig.i686
dbus-libs.i686 

IDA + Windows + system python 2.7.11

In case you:

  • are running IDA on Windows
  • are using the system’s Python (as opposed to the bundled Python distribution, that one can opt for at installation-time)
  • have installed the recently released Python 2.7.11

…you will have noticed that IDAPython fails to load the ‘site’ module and, consequently, IDAPython is not available.


As far as I understand, this is a Python installer issue, which I reported ( https://bugs.python.org/issue25824 )

While waiting for feedback from the Python authors, we can already offer the two following solutions:

  1. remove Python 2.7.11 and pick an earlier version (2.7.10 is known to work)
  2. open regedit.exe, and rename the key:
    HKLM\Software\Wow6432Node\Python\PythonCore\2.7\PythonPath into
    HKLM\Software\Wow6432Node\Python\PythonCore\2.7-32\PythonPath

This post will be updated whenever more information becomes available on the Python side.

HTH!

(Thanks to Tamir Bahar, who reported this issue very early on!)

Hack of the day #0: Somewhat-automating pseudocode HTML generation, with IDAPython.

The problem

As you may already know1, Hex-Rays decompilers can generate HTML files from pseudocode windows.
That feature, however, is limited to generating HTML for a single function, or a portion of a function.

Recently, one of our customers asked us whether there was a way to generate HTML files for multiple functions all at once. I was at the regret of telling him that, no, we don’t have that feature (and it doesn’t really seem to be of interest to many, since AFAICT we have received no request for it)

However, after the Great Actions Refactoring™, decompiler action are now 1st class IDA citizens (just like any other plugin actions, really.) That means we can now invoke them, just like we could already invoke any IDA core action. We just need to take a few precaution, is all.

A first approach

Here’s a piece of IDAPython code I hammered into shape, that will go through all the functions of the program and (if the current function is decompilable) will prompt the user for saving:

from idaapi import *
from PySide import QtGui, QtCore

# Our piece of code relies on the user having already opened a pseudocode view. That's a limitation, but oh well.
tform = find_tform("Pseudocode-A")
if not tform:
    raise Exception("Please open a pseudocode view")
qwidget = PluginForm.FormToPySideWidget(tform)

# Iterate all funcs
for fidx in xrange(get_func_qty()):
    f = getn_func(fidx)
    try:
        if decompile(f):
            # Ok, we can jump there w/o being
            # switched to "IDA View-A"
            jumpto(f.startEA)

            # Needed. W/o that, focus might be
            # elsewhere, and the 'hx:GenHtml'
            # will refuse to work.
            qwidget.setFocus()

            process_ui_action("hx:GenHtml")
    except DecompilationFailure:
        print "Decompilation failure: %x. Skipping."  % f.startEA
        pass

Notes:

  • I only tried that with IDA 6.8, on a Linux box.
  • this is *NOT* meant to be an elegant, comfortable or rock-solid approach to generating HTML files with pseudocode (hence the ‘Hack of the day’ subject of this post!)
  • this is really just meant to illustrate how IDA APIs can be (ab)used, and perhaps give inspiration to some readers for some of their use-cases
  • this will go through _all_ functions in the program. If you just need to print 20 functions out of 13942, it might become tedious to reject (13942 – 20 =) 13922 “Save file” dialogs.

Improving that code, so only the selected functions will be printed

A possibly nice improvement to this, would be to know what functions the user selected in the “Functions window”, and iterate over those only, prompting for save.

Here’s how I would go about it, knowing that:

  • the ‘Functions window’ is what we call, in IDA-land, a ‘chooser’
  • alas, this is not a user-controlled chooser
  • thus we don’t control the data being printed, nor do we get called back whenever something gets selected/deselected…

    Great Actions Refactoring to the rescue again!

    We will:

    1. create an “Generate HTML files” action that, basically, wraps the above code, then
    2. register a ‘popup being populated’ hook: when the popup is for the ‘Functions window’ we will attach our new action to that popup
    3. then, when that action is invoked (i.e., user clicked it), in our action handler we will retrieve the list of selected indices and iterate over those, prompting for save

    from idaapi import *
    from PySide import QtGui, QtCore
    
    # Register action
    class GenMultiHTML(action_handler_t):
        def __init__(self):
            action_handler_t.__init__(self)
    
        def activate(self, ctx):
            tform = find_tform("Pseudocode-A")
            if not tform:
                raise Exception("Please open a pseudocode view")
            qwidget = PluginForm.FormToPySideWidget(tform)
            for fidx in ctx.chooser_selection:
                f = getn_func(fidx - 1)
                try:
                    if decompile(f):
                        # ok, we can jump there w/o being
                        # switched to "IDA View-A"
                        jumpto(f.startEA)
                        qwidget.setFocus()
                        process_ui_action("hx:GenHtml")
                except DecompilationFailure:
                    print "Decompilation failure: %x. Skipping."  % f.startEA
                    pass
    
            return 1
    
        def update(self, ctx):
            return AST_ENABLE_FOR_FORM if ctx.form_title == "Functions window" else AST_DISABLE_FOR_FORM
    
    register_action(
        action_desc_t(
            'my:gen_multi_html',
            'Generate HTML files',
            GenMultiHTML()))
    
    # Listen to 'finish populating popup' hook, to add our
    # action if the popup is for the "Functions window"
    class Hooks(idaapi.UI_Hooks):
        def finish_populating_tform_popup(self, form, popup):
            if get_tform_title(form) == "Functions window":
                attach_action_to_popup(
                    form,
                    popup,
                    "my:gen_multi_html",
                    None)
    
    hooks = Hooks()
    hooks.hook()
    

    Happy hacking!


    Footnotes

    #1: And if you didn’t know: go to a pseudocode window, click on the function’s title (i.e., the first line), and open the context menu. You can also select a portion of text, and generate HTML just for that.