Asynchronous I/O is a scary thing. It requires a carefully choreographed dance between potentially many different threads, all the way from the top of the user-mode stack down to the kernel level. I have a lot of respect for the crew that implements this stuff. It's complicated enough just to use it correctly at the app level- I can't imagine how much worse it is down in the kernel guts.
That said, I think I found a nasty bug in .NET's asynchronous file handling support this weekend. I have a service that accepts streamed file uploads and writes them to disk. In order to scale better, I'm using WCF's AsyncPattern ServiceContracts- this allows me to yield the server's request thread when it's blocked on an external I/O operation, leaving it free to start servicing another request. This, of course, necessitates using asynchronous I/O for both reading the stream from the client as well as writing the received bits to disk. It appears that there's a bug in the Asynchronous FileStream's implementation of BeginWrite- when multiple request threads are doing a BeginWrite on a "cold" application (eg, one that's just started up, most code is un-JITted, etc), one or more of the BeginWrite calls will hang and never come back. Not a friendly thing for an asynchronous operation to do. If I make a single request, then pile a whole bunch of concurrent requests on right after, everything is fine. My current workaround is to make a single BeginWrite call to a dummy tempfile opened for async (FileOptions.Asynchronous) before I start the ServiceHost. Stupid, but it works like a charm.
Here's a link to the bug I filed, for those interested in that kind of thing (includes sample repro code). Go vote for it, if you like reliable async I/O. Hopefully there's a "silly user" answer for this, but I'm not holding my breath.