Some hours debugging later, and it was apparent that the crash was happening consistently by just letting the application idle after putting something on the clipboard. Apparently there was something wrong in the idle mechanism that updates the paste-command button... but what?
So I went surfing teh intarweb! And found this MSDN page from Microsoft explaining how to use the clipboard from a .Net application. Then I noticed this:
To access data from the Clipboard by using versions earlier than .NET Framework 2.0, use the GetDataObject method and call the methods of the returned IDataObject. To determine whether a particular format is available in the returned object, for example, call theGetDataPresent method.That looked familiar, and indeed the code was using the old way of accessing the clipboard, by first using GetDataObject() and then using GetData() on the result.
Now the page didn't say anything about this way of doing things being deprecated, but I still got that nagging sensation that this was an important clue.
So I set down and create two simple test applications that did the same thing, only each using a different variation of the clipboard API. The applications were simple: put something on the clipboard, and repeatedly try and read it back in a loop.
The result was that using the new (and preferred?) way of clipboard access had no problems. I ran it through a loop of million clipboard accesses without a hiccup. On the other hand, the application using the old variation of the API had a very familiar problem. After some iterations it started throwing the familiar first-chance OutOfMemoryException. What more, the clipboard data stopped being available for the test application. Very similar indeed to the problem in the main application!
A few repeat runs, trying stuff like forced garbage collection (which did not help at all, except the performance went down the drain) a clear pattern emerged: My test application would stop working after 65525 invocations of GetData(). The 65526th time and onward it would just silently fail with the OutOfMemoryException in the debugger output.
65526 is pretty darn close to the magical 0xFFFF 16 bit limit, so allowing for some initialization and general overhead, it seems clear that the old variation of the clipboard API (pre v2.0) is leaking a handler, global resource or otherwise doesn't free up some counter internally.
If you are lucky enough to be able to use the newer version of the .Net API I suggest you do so, and remove any code using the old API, since its quite simply broken!
