How to properly support Windows Fibers in Delphi without breaking exception handling

If you are giving a shot to Windows Fibers under Delphi, you probably faced the issue of how to properly handle Delphi's structured exception handling AND switching between fibers transparently.

Assuming you already know what Fibers are, how to use them and when to use them, I will cut to the chase. If you don't know, or don't know enough, I recommend you the following reads to get started:


https://msdn.microsoft.com/en-us/library/windows/desktop/ms682661%28v=vs.85%29.aspx
http://blogs.technet.com/b/markrussinovich/archive/2009/07/08/3261309.aspx


When exceptions happen in a Delphi program, the compiler generates code to store nested exceptions on an exception stack. This stack is stored in a TLS allocated by the program upon startup.

When using Delphi compiled in 32 bits, the issue of persisting and restoring this stack is trivial. Up to the latest versions of Delphi the RTL exposes a couple of functions in the system unit that allow you to do the trick in two lines of code (literally).
The challenge with Delphi programs compiled in 64 bits is that Delphi doesn't expose anymore an API to persist and restore the Exception stack. It's not hard overcome this, but it requires a bit of reverse engineering and watching what the compiler does by enabling the CPU debugger.
I'll spare you from the pain, and below you will find a unit that does the trick:

uWin64ExceptionStack.pas


Here's the full code that I use for switching between fibers:



Before I forget!!!

Here's a sample test that performs a SwitchToFiber() inside an exception handler block. In fact, it's a triple nested exception handler block to test the mechanism of persisting and restored more than just one element of the exception stack.

I hope this helps you on your adventure trying to implement fibers using Delphi.

A note on testing the approach:

Code was verified to work properly on:
Delphi 2007
DelphiXE4 running in WIN32 and WIN64

Special note about Delphi 5. SetRaiseList was totally broken. When used, it will send the application into a tailspin of access violations. See the fixed code, mainly borrowed from Delphi 2007 implementation.

Never, ever dismiss the most innocent failing test on your continuousintegration system!

Situation

As part of a project in C that requires some threading primitives, I wrote a piece of code to abstract a Windows timer into a small C class (yes, C class... please don't bash me here for mixing "classes" with plain C code, that will be material for another discussion).

You can find the header file of the basic classes provided by this library here:

https://gist.github.com/jsbattig/9379060

This timer primitive uses Windows threadpool timers. They are kind of nice because they have nothing to do with traditional windows handle linked timers which require a message loop marshaling messages sent to the handle neither they are "multimedia timers". Besides, this library already required thread pools so adding support for threadpool timers was a natural extension to the library itself.

By looking at Windows API documentation I find that to get one of these timers going, besides the fact you need a callback environment variable pre-initialized (or NULL) you need to use the following two APIs:

CreateThreadpoolTimer
SetThreadpoolTimer

SetThreadpoolTimer will actually "start" the timer.

The key parameters this function receives are the timer handle, a "due time" and a recurrence parameter or interval. There's a fourth parameter named msWindowLength which we don't particularly care at this point.

The due time, has the oddity to be the number of units of 100 nanoseconds since January 1, 1601 (UTC) expressed in FILETIME format (I personally found this kind of odd, for a timer... but anyway, MS folks did it like this for some reason I imagine). This parameter, if negative, can also mean a relative time since current time.
The msPeriod parameter is expressed in milliseconds. Why one parameter denoting timer timing in FILETIME and the other in milliseconds? Don't know...

Anyway, at first sight it seemed pretty straightforward, so I crafted the code bellow to get my timer created and going:


So, as you can quickly see, I missed the boat from the get going since I set the FILETIME parameter to be the actual interval in milliseconds converted to 100 nanosecond units since 0 (zero) time.

Here's where Mr Jenkins came to the rescue!
There's a particular test I run to exercise the functioning of this timer:

This test failed once today just before I wrote this blog entry.
This is the GoogleTest jenkins entry for the failure:

c_driver_SSvcBusThreadPoolTest_Win32_Release_singleMongo.SvcBusConsumer_testThreadPoolTimer

 Error Details

Value of: SvcBusThreadPoolTimer_getThreadId( timer)
  Actual: 24652
Expected: 0

 Stack Trace

threadpool_unittest.cc:132
Value of: SvcBusThreadPoolTimer_getThreadId( timer)
  Actual: 24652
Expected: 0

I could have easily dismiss it as a "fluke" or some kind of oddity, maybe the CPU was too busy and the program took to long to go from the creation of the timer to the actual check that's why the threadId recorded was != 0 (even tough the timer was created with 100ms due time).

I don't believe in "oddities" anymore at this point in my career and haven't done so for a while...

So, I dug a little bit to find the obvious which I pretty much imagined since I saw the Jenkins log.

The timer was kicking right away because I pass a positive number not adjusted to January 1st 1601. Now, I did write code to do the adjustment and showed the new working code to a colleague only for him to quickly point me down after checking the documentation why I didn't use the "relative to current" feature. Always good to have another pair of eyes checking at your code to uncover your own naiveness!

So, finally I did that, which by the way it was a really easy solution. All it took was casting the millis parameter to __int64 and negate it. That's it!

Solution

Here's the fixed code:

That's it. That fixes our "oddity", our "cosmic ray flipping that bit to zero", our bad bad Jenkins making our perfectly good code and perfectly written test to fail.

No, there's no oddities, cosmic rays, bad Jenkins or slow CPUs (maybe sometimes there's slow CPUs). A failing test is a failing test, it's signaling something. Either the test is flawed, or there's a bug hiding somewhere behind it.

Happy coding!

DLLs deadlocking when getting unloaded if attempting to exit threads

The Problem

At Convey, we use many different languages to construct our solutions, one of them and probably the most commonly used today for a lot of our back-end services is Delphi. 

As any developer with even minor knowledge of Delphi knows, applications based on it are broken up on Units and Units have a "initialization" block of code, and a "finalization" block of code.

Typically these to blocks are utilized to initialize and finalize globals used by each module. These two blocks are guaranteed to be called upon startup/first use of a module and when the module it's going out of scope either by a containing library being unloaded or a program finishing.

Traditionally, developers using other languages such as C rely on explicit calls to initialize or finalize resources on modules... but not in Delphi.

So, this was the root of our problem.

We had code that when compiled and run as part of a standalone EXE or a BPL (Borland Package Library) worked as a charm. Programs started, used the code with no issue and unloaded themselves with no problems. BUT... when the same modules where linked as a part of a DLL, it simply "locked" the program when trying to unload and it was necessary to kill the process for the outside.

Because of this, we ended up relying on all kind of dirty tricks, from leaving memory leaks by prevent freeing resources that "seemed" to cause the freeze to incorporating "auto-kill" code on DLLs that when detected that an app was trying to shutdown it will simply kill the process from the inside.

A while ago I read an article by Chris Wenham ( Signs you are a bad programmer ) and decided that it was time to clean the house to be less of a "bad programmer" according to his definitions. I took on an old thread based timer I wrote many years ago when there no such facility on Windows, but over the years Microsoft added decent timer support. The refactoring resulted on the dreaded DLL deadlocking upon unload of a library which contained the newly refactored timer code.

After some time researching and scratching my head, I came across this article:

Exit thread upon deleting static object during unload DLL causes deadlock?

I decided to go to MSDN to read more about DLL entry point mechanics ( DllMain entry point ) and found this piece of text:

Because DLL notifications are serialized, entry-point functions should not attempt to communicate with other threads or processes. Deadlocks may occur as a result.

Now, that was the first glaring warning sign that I was trying to do something I was not supposed to. The fact I was trying to terminate a job thread on the finalization section of the unit "smelled" to me as "communicating" with other thread on some way.

First thing I tried after that was leaving a resource leak (the actual job threads used by the timer) and that proved to prevent the deadlock from happening.

A second finding as I was experimenting with the finalization section of this unit was that a call to CoUninitialize()causes the same deadlock behavior. And if you read the specs of the function you find this:

Because there is no way to control the order in which in-process servers are loaded or unloaded, do not call CoInitialize, CoInitializeEx, or CoUninitializefrom the DllMain function.

Shamefully, without relying on an explicit call from the host application, the only "solution" for the CoUninitialize() limitation is simply to detect the scenario of FreeLibrary() being called and avoid calling the function on the finalization section.

The Solution

So, what do to here? One recommendation will be to make every module or DLL have a couple of exported procedures to Init and Finalize global resources on the DLL. Then you can free your global threads on those procedures. The problem with this approach is that in many cases it might not be possible to modify the host application to adhere to this new protocol, and the second issue I see is that for probably all Delphi developers it's good practice to write code on the finalization section to free global resources without special consideration about all the limitations that DllMain imposes. As said before, under standalone EXE or when using BPLs, no limitations on the finalization code are present that I'm aware of. 

So, I decided to go for something I considered an "elegant hack" (free to interpretation here if there's such a thing as "elegant hacks"...) that will make applications that try to finish a thread on the finalization section of a unit, compatible with DLLs without further modifications other than linking a particular unit that contains the hack on it.

The Code

The first thing I did was to create my own installable custom DllMain handler. This is easy to do with Delphi using the global variable DllProc.

interface
...

var
  ShuttingDownDll : Boolean; // Use this flag to know when a DLL is in DETACH mode

implementation
...

{$IFNDEF DELPHI2007}
type
  THookedDllProc = procedure (Reason: DWORD);
{$ENDIF}

var
  {$IFDEF DELPHI2007}
  OldDllProc : TDLLProc;
  {$ELSE}
  OldDllProc : Pointer;
  {$ENDIF}

// Hooked DllProc used to flag when DLL is being detached
procedure HookedDllProc(Reason: DWORD);
begin
  if not ShuttingDownDll then
    ShuttingDownDll := Reason = DLL_PROCESS_DETACH;
  if assigned(OldDllProc) then
    {$IFNDEF DELPHI2007}THookedDllProc({$ENDIF}OldDllProc{$IFNDEF DELPHI2007}){$ENDIF}(Reason);
end;

initialization
  OldDllProc := DllProc;
  DllProc := @HookedDllProc;
...

finalization
  ...
  DllProc := OldDllProc;
end.

With this now we have the flag ShuttingDownDll set to True when the DLL receives the DLL_PROCESS_DETACH signal.

The second part is the real hack. For this I decided to change the semantics of Delphi EndThread() system procedure. For this I did a simple hack well known on the Delphi community, which involves overwriting the first bytes of the actual code with a relative JMP to the new code.

For this to work you need this:

implementation
...
type
  PJump = ^TJump;
  TJump = packed record
    OpCode:byte;
    Distance:integer;
  end;

var
  OldCode : TJump;
  NewCode : TJump;

procedure HookedEndThread(ExitCode: Integer);
begin
  {$IFDEF DELPHI2007}
  if Assigned(SystemThreadEndProc) then
    SystemThreadEndProc(ExitCode);
  {$ENDIF}
  if (not IsLibrary) or (not ShuttingDownDll) then
    ExitThread(ExitCode)
  else TerminateThread(GetCurrentThread, ExitCode); // Forceful termination of thread if library mode and DLL_PROCESS_DETACH mode
end;

procedure PatchEndThread;
begin
  NewCode.Distance := Integer(@HookedEndThread) - (Integer(@EndThread) + 5);
  PatchMemory (@EndThread, 5, @NewCode, @OldCode);
  FlushInstructionCache (GetCurrentProcess, @EndThread, 5);
end;

procedure UnPatchEndThread;
begin
  PatchMemory (@EndThread, 5, @OldCode);
  FlushInstructionCache (GetCurrentProcess, @EndThread, 5);
end;

initialization
  ...
  NewCode.OpCode := $E9;
  NewCode.Distance := 0;
  PatchEndThread;
  ...

finalization
  ...
  UnPatchEndThread;
  ...
end.

Something to note here is that TerminateThread() doesn't cause the deadlock that ExitThread() causes. It can be assumed that TerminateThread() doesn't attempt to "communicate" with the target thread to be terminated. Of course TerminateThread() is not the same as a clean ExitThread() call, but at least we can get as far as possible following the normal path of execution of the program.

Finally, this is a typical implementation of PatchMemory():

procedure PatchMemory(p : Pointer; DataSize : Integer; Data : Pointer; OldData : pointer); 
{$IFNDEF DELPHIXE2}
type
  SIZE_T = DWORD;
{$ENDIF}  
var
  OldProtect : DWORD;
  BytesWritten : SIZE_T;
begin
  VirtualProtect (p, DataSize, PAGE_EXECUTE_READWRITE, OldProtect);
  Move (p^, OldData^, DataSize);
  WriteProcessMemory(GetCurrentProcess, p, Data, DataSize, BytesWritten);
  VirtualProtect (p, DataSize, OldProtect, OldProtect);
end;

Notice the call to WriteProcessMemory() instead of Delphi's standard Move() procedure. This is key to avoid being caught by Windows DEP protection. Even tough we called VirtualProtect() to make memory writable, DEP doesn't like a process writing anything to the code segment unless it's done using WriteProcessMemory(). If you use that function, you can pretty much overwrite any piece of the code segment as long as you unprotect the memory first.

A caveat with this example is that it's not 64 bits compatible. Obvious things that need to be adjusted are pointer arithmetics and potentially the relative jump used to overwrite ExitThread() ( JMP - Jump ).

Happy coding!