How to Kill a Keep Alive with a Weak Reference (C#)

Taskling.NET uses a keep alive or heartbeat to signal that it is still running. This is useful because when running batch jobs in unstable hosts like IIS the process can be killed off with a ThreadAbortException and the job isn't always able to log it's demise. With a keep alive we know that the job really died if a few minutes pass without a keep alive and the status of the job is "In Progress".

But one problem is how do you reliably kill a keep alive? In Taskling.NET the keep alive runs on another thread and the parent context implements IDisposable, so when disposed is called it stops the keep alive. However Taskling doesn't prevent developers from not using a using block or a try finally block and so there is no guarantee that Dispose will be called. Which means that a keep alive thread could keep going and going until the parent process shuts down.

Taskling.NET avoids this by holding a weak reference to the parent ITaskExecutionContext. When a Taskling job completes and passes out of scope the parent context is garbage collected. The keep alive thread checks the status of the parent context in its keep alive loop and as soon as it sees that it is no longer living it breaks out of the loop and dies itself.

We instantiate the KeepAliveDaemon from the ITaskExecutionContext, passing it a new WeakReference(this).

private void StartKeepAlive()
{
	var keepAliveRequest = new SendKeepAliveRequest()
	{
		TaskId = new TaskId(_taskExecutionInstance.ApplicationName, _taskExecutionInstance.TaskName),
		TaskExecutionId = _taskExecutionInstance.TaskExecutionId,
		ExecutionTokenId = _taskExecutionInstance.ExecutionTokenId
	};

	_keepAliveDaemon = new KeepAliveDaemon(_taskExecutionRepository, new WeakReference(this));
	_keepAliveDaemon.Run(keepAliveRequest, _taskExecutionOptions.KeepAliveInterval.Value);
}

The KeepAliveDaemon stores the WeakReference and checks its IsAlive property on each iteration.

internal class KeepAliveDaemon
{
    private delegate void KeepAliveDelegate(SendKeepAliveRequest sendKeepAliveRequest, TimeSpan keepAliveInterval);
    private WeakReference _owner;
    private readonly ITaskExecutionRepository _taskExecutionRepository;
    private bool _completeCalled;

    public KeepAliveDaemon(ITaskExecutionService taskExecutionRepository, WeakReference owner)
    {
        _owner = owner;
        _taskExecutionRepository = taskExecutionRepository;
    }

	public void Stop()
	{
		_completeCalled = true;
	}

    public void Run(SendKeepAliveRequest sendKeepAliveRequest, TimeSpan keepAliveInterval)
    {
        var sendDelegate = new KeepAliveDelegate(StartKeepAlive);
        sendDelegate.BeginInvoke(sendKeepAliveRequest, keepAliveInterval, new AsyncCallback(KeepAliveCallback), sendDelegate);
    }

	private void StartKeepAlive(SendKeepAliveRequest sendKeepAliveRequest, TimeSpan keepAliveInterval)
	{
		DateTime lastKeepAlive = DateTime.UtcNow;
		_taskExecutionRepository.SendKeepAlive(sendKeepAliveRequest);
	
		while (!_completeCalled && _owner.IsAlive)
		{
			var timespanSinceLastKeepAlive = DateTime.UtcNow - lastKeepAlive;
			if (timespanSinceLastKeepAlive > keepAliveInterval)
			{
				lastKeepAlive = DateTime.UtcNow;
				_taskExecutionRepository.SendKeepAlive(sendKeepAliveRequest);
			}
			Thread.Sleep(1000);
		}
	}



    private void KeepAliveCallback(IAsyncResult ar)
    {
        try
        {
            var caller = (KeepAliveDelegate)ar.AsyncState;
            caller.EndInvoke(ar);
        }
        catch (Exception)
        { }
    }
}

In order to be able to use Taskling on legacy applications I built Taskling with .NET 3.5 entirely. That is why there is no use of Task o async await. I am planning on releasing a .NET 4.5 version that will offer async methods, and mean that the KeepAliveDameon can run in a Task and use await Task.Delay(1000) instead of Thread.Sleep(1000).

What is a Weak Reference?

A weak reference is a reference to another object but the key difference is that it does not count towards a reference that would prevent garbage collection. With a normal reference, as long as that reference to the target object is around, the target object cannot be garbage collected - it is in use. Weak references don't get treated like that, if the only reference to an object is a weak reference then the garbage collector will go ahead and reclaim the memory of that object.

The keep alive thread leverages that to detect when its parent context is dead so it knows when to stop sending keep alives. In normal situations this should not happen. The developer should be disposing the ITaskExecutionContext correctly and so the keep alive should be explicitly told to stop by the context. But never rely on good developer behaviour, we've all written that code that doesn't properly dispose a connection or a file stream.