[NTTCP-8] schedule/runSchedule may execute actions out of order #440

[Imported from JIRA. Reported by Facundo Dominguez @facundominguez) as NTTCP-8 on 2015-04-06 14:49:04]
The [git history|https://github.com/haskell-distributed/network-transport-tcp/commit/3855e91dcad7381f6626fc47adb8f4c9c60d5279] seems to claim that a scheduled action is supposed to be executed only after all earlier scheduled actions have been executed. The current implementation does not ensure this however. Note that multiple threads calling {{runScheduledAction}} can pull actions almost simultaneously from the channel and interleave their execution in any order.

Possible fixes are:
(1) Have a dedicated thread per remote endpoint (heavy-weight connection) executing scheduled actions.
(2) Have a {{runScheduledAction}} read an action from the channel and execute it atomically.

What is the need to execute actions in the order they are queued? The git comment is not clear about that. It says:

{code}
thingToSend <- modifyMVar (....)
send thingToSend

is not good enough, because now if we have to threads doing the above the sends
may be done in the wrong order.
{code}

But the n-t interface does not ask to ensure ordering if messages are submitted concurrently.
The only problem I see is that sending messages could be attempted on a closed socket if they are submitted concurrently with a connection close call.