IDA Pro: what should not be SDK
This article will be about how you should not do it when you are developing an SDK for your product. And an example, one might even say, the most striking, will be IDA Pro. Those who at least once developed something for it and tried to support, while reading these lines, probably now startled and covered with cold sweat. Here I gathered experience supporting projects, starting with IDA v6.5, and ending with the latest version at the time of writing of the article - v7.5. In general, they drove.
SDK for IDA Pro allows you to develop the following types of applications:
- Downloaders of various formats
- Processor Modules
- Plugins that extend the functionality (processor modules, interface, etc.)
- IDC scripts (your own language) and Python scripts (using third-party IDAPython development, which has become an integral part of IDA)
According to information from the Hex-Rays website, the cost of the SDK support plan is 10,000 USD . In practice, if you have a license, you are given an access code for the Support zone in which you download and work with it. The cost is indicated in case you have questions and want to ask the developers: without a support plan they will tell you, they say, write it yourself; with support, as I understand it, they cannot refuse you. Unfortunately, I do not know a single person (company) who bought this plan.
A little more detail
From the moment you have a desire to write something under the IDA, and you download the SDK, you step on a rather slippery track, which, moreover, is very easy to go crazy. And here's why:
1) In modern development, you are most likely already used to the fact that any more or less serious project has either doxygen -generated help for the API, or manually written good help that suits you says that where it is transferred, what needs to be filled in order to work, etc., with a bunch of examples.
In the IDA, in most of the APIs available to the user, only parameter names and their types are indicated, and what to pass there can be guessed only by argument names. Recently, of course, with help from the IDA it has become better, but not that much.
2) In many populated structures, callback functions are required, some of which are indicated as optional, they say you won’t specify (pass CDMY0CDMY) - okay. In reality, application crashes when trying to run your plugin. And, because there are a lot of callbacks (an example is a debugging plug-in), you start to ask in turn all that you can not set. As a result, it is very tiring, you open x64dbg/ollyDbg, in it - idaq.exe/ida.exe, load the plugin, put breakpoints, and try to catch the moment when control is transferred to 0x00000000.
Eh, I remember those times when so many project folders were clogged with 200MB dmp files that were created during the IDA crash... But they didn’t help me.
3) The most painful topic for IDA Pro is backward compatibility. In principle, this is a rather difficult task for any developer - to think through interfaces, structures, modularity, etc. in advance. Therefore, there are two ways:
- Keep backward compatible with all old versions
- Don’t do backward compatibility
In the first case, a legacy code accumulates very quickly, which then becomes impossible to get rid of. It is rarely possible for anyone to clean it. Because, in fact, you need to throw all your strength not on the development of new functionality, but on sawing/transferring the old.
In the second case, we do not have a backward compatibility problem, but there is another one: we have a publicly available SDK, we need to regularly monitor what changes are happening and document them so that all those who started development for the previous version can easily switch to a new one.
What about Hex-Rays? You will be surprised, but they went two ways at the same time! It is known that the project has been developing from very, very bearded times, when the main target platform was only MS-DOS (and, therefore, reverse engineering of applications written for it).It was necessary to support segment registers, selectors, paragraphs and other similar attributes. As time went on, other platforms, processor modules, and loaders began to appear in the IDA, where the memory model was already flat ( flat ), but a relic in the form of the MS-DOS features I listed was still preserved! The entire IDA interface is riddled with this. When developing processor modules, in which only flat, you still have to specify segment registers (though they are already virtual).
But with the SDK, there is no question of any backward compatibility at all. In each new version (even inside minor builds 6.x and 7.x) something breaks: callbacks have new arguments, fields are renamed and lost for old structures, API functions that previously did one task now do another. And there are many such examples.
And it’s okay if all this was somehow accompanied by the developer, they say, this and that has changed in this version, now you need this and that. So no! There is a guide, yes: IDA 7.0 SDK: Porting from IDA 4.9- 6.x API to IDA 7.0 API , but that’s it. Moreover, according to it, you will not be able to transfer your project to a new version, because it does not include very many, but minor, changes, which, of course, no one will tell you. In addition, this is the last guide for the C/C++ developer, and since then it has been released somewhere else 5-6 versions of the SDK.
As a result, it turns out that there is backward compatibility support inside the company, but not for external users who paid money.
Once I took the liberty of trying to develop my first Motorola 68000 debugger plugin for IDA. There was an example of a debugger in the supplied SDK (which, in fact, is used in IDA Pro now as a local and remote one), but it was performed so poorly that it was impossible to try to make one on it. Then I crawled on the Internet and found the only debugging plug-in for PS3, which, funnily, was based on the same code from the SDK.
So that you understand why I didn’t dare to use the code from the SDK, here’s a screenshot of a section of the code where I already understood that it would be hard:
See the cpp files that are included through CDMY1CDMY? And so on throughout the source. Nevertheless, having carefully studied the source code of the PS3 debugger, I managed to isolate something working from it and make my own - for Sega Mega Drive.
There was one catch: I did not have the source code for the processor module, so necessary for writing a debugger. And it was necessary to turn to registers, to know which indexes and flags for which opcodes were installed, to fill out structures displaying calculated links.
To do this, I had to debug the IDA, the m68k processor module, and make corrections for the latter (I wrote about this in " Modernization of IDA Pro. We fix jambs of processor modules ").
Despite all the difficulties, I managed to write a good debugger! Unfortunately, a new version of the SDK was released... The structure of CDMY2CDMY responsible for the debugger and its callbacks changed, and everything I tried to do led to crashes of the IDA itself. After a while, I dealt with this.
But a new version of the SDK... x64 has been released, without x86 compatibility! And the Gens emulator, on the basis of which I made the debugger, did not know how to x64, and the project stalled for many years. When I found an emulator that was able to work in x64, there were so many versions of the SDK that again trying to understand why my plugin was not working, I did not dare.
SDK problems for IDA Pro in the absence of normal documentation of changes in each version; that changes come out in droves - there are a lot of them, they are cardinal, and it’s very hard to keep up with them.
If your company/team is posting some kind of public SDK, and it has users who pay money - please, think about them too! Their development can help your product become better and more popular, as happened with IDAPython.It is clear that keeping backward compatibility is very difficult, but if you decide not to support the old versions, try to document all the changes you make.
I saw on Github a large number of useful projects that have remained unported on IDA v7.x. You might think that their functionality has become unnecessary in new versions? Maybe so, but, as for me, it’s fatigue and unwillingness to deal with an ever-changing API, coupled with the hobbiness of the project.
IDA Pro Book
I would also like to recall one invaluable book that once helped me a lot, but which is now absolutely useless for the IDA plugin developer - IDA Pro Book by Chris Eagle . Everything described in it refers to version 6.x (tentatively v6.5-v6.8). Almost everything has changed since then.