C#’s async, await, and .Result

In Market Invoice, there are many places where async and await are used. Recently, I introduced a bug that an operation gets deadlocked by replacing await with .Result. I was bitten hard 🙂

Non-blocking execution

When using async and await, C# run time generates a state machine in the background

public async Task CallingMethodAsync()
{
    Task<int> longRunningTask = LongRunningOperationAsync(); // 1)
    // independent work which doesn't need the result of LongRunningOperationAsync can be done here

    //and now we call await on the task
    int result = await longRunningTask; // 2)

    //use the result
    Console.WriteLine(result);
}

public async Task<int> LongRunningOperationAsync()
{
    await Task.Delay(1000); //1 seconds delay
    return 1;
}

1) LongRunningOperationAsync is running. But it doesn’t block the execution of CallingMethodAsync, until the execution point reaches 2)

Now the execution point reached 2). If LongRunningOperationAsync() is fully done, the result will be ready, and it will be assigned to result straight away. However, if LongRunningOperationAsync() is still running, the execution of CallingMethodAsync will stop there, waiting until LongRunningOperationAsync() finishes. Once it finishes, CallingMethodAsync will resume the execution.

Call-back without its hell

Let’s look at Eric’s Serve Breakfast example.

void ServeBreakfast(Customer diner)
{
    var order = ObtainOrder(diner);
    var ingredients = ObtainIngredients(order);
    var recipe = ObtainRecipe(order);
    var meal = recipe.Prepare(ingredients);
    diner.Give(meal);
}

In this example, every customer must wait until the previous customer’s breakfast is fully prepared and served. You can see people would get angry very soon.

In order to receive orders while preparing for breakfast, you have to take orders in an asynchronous manner. It will bring it javascript’ call-back hell.

void ServeBreakfast(Diner diner)
{
  ObtainOrderAsync(diner, order =>
  {
    ObtainIngredientsAsync(order, ingredients =>
    {
      ObtainRecipeAsync(order, recipe =>
      {
        recipe.PrepareAsync(ingredients, meal =>
        {
          diner.Give(meal);
        })})})});
}

The code is not very readable. Computers may like it, but humans are not good at following up the callbacks.

This can be rewritten in the new style, reads much more nicely.

async void ServeBreakfast(Diner diner)
{
  var order = await ObtainOrderAsync(diner);
  var ingredients = await ObtainIngredientsAsync(order);
  var recipe = await ObtainRecipeAsync(order);
  var meal = await recipe.PrepareAsync(ingredients);
  diner.Give(meal);
}

Now, the methods, ObtainOrderAsync() doesn’t return order. It returns Task<Order>. It’s a callback pointer. When the execution finishes, it return the result, and order is passed into ObtainIngredientsAsync()

await or .Result

Stephen Cleary recommends using await over Result.

First, await doesn’t wrap the exception in an AggregateException, which represents one or more errors that occur during application execution. So, you will see the real exception, not the bland AggregateException. .Result wrap an exceptions that happens in the async method into AggregateException.

try {
    details = await _service.GetDetails(personId);
    ...
} catch (ApplicationException) { // this catch will work, as await pass the exception as it is.
    ...
}

For .Result, you have to catch AggregateException.

Second, Result / Wait can cause deadlocks. The async method will continue to run, and the task will be returned to it. When the task comes back, and if it’s not completed yet, it will hang in the current context.

public class CompanyDetailsController : ApiController
{
    public string Get()
    {
       var task = GetCompanyDetails(...);
       return task.Result.ToString(); // if task hasn't been completed, this will block the thread.
    }
}

public static async Task<CompanyDetails> GetCompanyDetails(Uri uri)
{
    using (var client = new HttpClient())
    {
        var jsonString = await client.GetStringAsync(uri);
        return CompanyDetails.Parse(jsonString);
    }
}

Preventing the deadlock

ConfigureAwait

await Task.Delay(1000).ConfigureAwait(
    continueOnCapturedContext: false);
  // Code here runs without the original context. (if the original context is UI thread, then UI thread context)

By using ConfigureAwait, you enable parallelism that the asynchrounous code can run in parallel with the thread the original context is in. As a result, you can avoid the deadlock

avoid Result / Wait

As Result causes deadlocks, don’t use it. Instead, favour await and use async on the method all they down or up.

 

Resources

 

C#’s async, await, and .Result

Opening an office document from Code

It’s a simple job at a glance, and it should be. But we had a bug with that feature. You open a document on Huddle and it should be opened in an application that is associated with the file, for example, MS Word, if it is word document. You can do that if you have installed Huddle for Windows, which is a desktop application for Huddle.

It worked. Yet, customers often complained that the document opened in the background. When we open any document, it opens in the foreground on dev machine. so you are very tempted to say “it works on my machine!”

This is the code that opens the document. No magic, just simple process.Start()

var process = new Process
{
    StartInfo =
    {
        FileName = path,
        CreateNoWindow = true,
        UseShellExecute = true,
        LoadUserProfile = false,
        ErrorDialog = false,
        Verb = &amp;quot;open&amp;quot;
    },
    EnableRaisingEvents = true
};

process.Start();

The application that’s launched is supposed to be active in the foreground, and I’ve got that behaviour on my dev machine. Yet on non-dev machines, especially our product manager’s laptop, the document opened consistently in the background, especially behind Chrome browser when it was maximised.

So, we made a win api call to set it in the foreground.


[DllImport(&amp;quot;User32.dll&amp;quot;)]
private static extern Int32 SetForegroundWindow(IntPtr hWnd);

...
var handle = process.MainWindowHandle
SetForegroundWindow(handle);

Now it was working on one of my VMs (Virtual Machines) and I thought it should be working. However it was still failing on a laptop.
Why? A little more investigation reveals that process.MainWindowHandle doesn’t return the handle immediately. When MS Word splash screen pops up, the process to MS Word exists, but the main window is not fully loaded yet, so the handle was IntPtr.Zero. You have to wait until MainWindowHandle is populated, and it takes 3 to 6 seconds and up to 10 seconds, if you open Word first time on the day.

So, the last missing bit was waiting for the handle.

int threshold = 0;
while (process.MainWindowHandle == IntPtr.Zero)
{
    if (threshold &amp;gt; 100) break; 

    _log.DebugFormat(&amp;quot;Waiting for the document is fully loaded... - {0}&amp;quot;, threshold);
    Thread.Sleep(TimeSpan.FromMilliseconds(100));
    threshold++;
}

_log.DebugFormat(&amp;quot;Bringing the application (handle: {0}) to the front&amp;quot;, process.MainWindowHandle);
SetForegroundWindow(handle);

And happy developers and customers, finally

Update as of Fri.19/12/2014

The journey didn’t end there unfortunately. The issue was deeper than I previously assumed.

Opening documents in the background had a mixture of causes.

  1. If there’s any Excel document, Excel opens all subsequent documents are in the same instance, just flashing the icon since Windows 7
  2. Event SetForegroundWindow doesn’t work, if the process that’s running the command is not in the foreground. So you have to set your process foreground first, if you want to set any other process you spawn foreground.

To sort them out, I took a different approach. First, I gave up the simple and elegant Process.Start() with UseShellExecute option. Instead, I query windows registry and find the associated application with the file extension.

[Flags]
public enum AssocF : uint
{
    None = 0,
    Init_NoRemapCLSID = 0x1,
    Init_ByExeName = 0x2,
    Open_ByExeName = 0x2,
    Init_DefaultToStar = 0x4,
    Init_DefaultToFolder = 0x8,
    NoUserSettings = 0x10,
    NoTruncate = 0x20,
    Verify = 0x40,
    RemapRunDll = 0x80,
    NoFixUps = 0x100,
    IgnoreBaseClass = 0x200,
    Init_IgnoreUnknown = 0x400,
    Init_FixedProgId = 0x800,
    IsProtocol = 0x1000,
    InitForFile = 0x2000,
}

public enum AssocStr
{
    Command = 1,
    Executable,
    FriendlyDocName,
    FriendlyAppName,
    NoOpen,
    ShellNewValue,
    DDECommand,
    DDEIfExec,
    DDEApplication,
    DDETopic,
    InfoTip,
    QuickTip,
    TileInfo,
    ContentType,
    DefaultIcon,
    ShellExtension,
    DropTarget,
    DelegateExecute,
    SupportedUriProtocols,
    Max,
}

// from http://www.pinvoke.net/default.aspx/shlwapi/AssocQueryString.html
[DllImport(&quot;Shlwapi.dll&quot;, CharSet = CharSet.Unicode)]
static extern uint AssocQueryString(AssocF flags, AssocStr str, string pszAssoc, string pszExtra, [Out] StringBuilder pszOut, ref uint pcchOut);

public string GetApplicationPath(string extension)
{
    const int S_OK = 0;
    const int S_FALSE = 1;

    uint length = 0;
    uint ret = AssocQueryString(AssocF.None, AssocStr.Executable, extension, null, null, 
         ref length);
    if (ret != S_FALSE)
    {
        throw new InvalidOperationException(&quot;Could not determine associated string&quot;);
    }

    var sb = new StringBuilder((int)length); 
    ret = AssocQueryString(AssocF.None, AssocStr.Executable, extension, null, 
          sb, ref length);
    if (ret != S_OK)
    {
        throw new InvalidOperationException(&quot;Could not determine associated string&quot;);
    }

    return sb.ToString();
}

You get the application and run it with Process.Start. You pass the file name as argument. In case of office documents, you can add “/x” switch, so that it creates a new instance all the time, not reusing the existing one.

var extension = Path.GetExtension(path);
try
{
    string application = _windowManager.GetApplicationPath(extension);
    var process = new Process
    {
        StartInfo =
        {
            FileName = application,
            Arguments = GetArguments(application, path),
            WindowStyle = ProcessWindowStyle.Normal
        }
    };

    _log.Debug(string.Format(&quot;Starting {0}&quot;, path));

    var huddleWinHandle = Process.GetCurrentProcess().MainWindowHandle;
    _windowManager.MinimiseWindow(huddleWinHandle);
    _windowManager.BringToFront(huddleWinHandle);
    process.Start();
}
...

private string GetArguments(string application, string path)
{
    if (application.ToLower().Contains(&quot;microsoft&quot;))
    {
        return &quot;/x \&quot;&quot; + path + &quot;\&quot;&quot;;
    }

    return &quot;\&quot;&quot; + path + &quot;\&quot;&quot;;
}

Why do you minimise the window of the current process before setting it foreground? For some reason, SetForegroundWindow didn’t work consistently, if you process main window is in normal mode, hidden behind other windows. When it was minimised programatically and set to foreground, it was always brought to the front.

So, to summarise how to open documents in the foreground,

  1. Set your application to the foreground first by minimising the window and then bringing it to the front
  2. Find the associated application with the file and start it as new instance.
  3. Before you start another process, make sure your process in the foreground first. Only foreground process can make an attached process run in the foreground. There are more rules in setting a process in the foreground.
Opening an office document from Code

Handling Sandwich Code in C#

What is “Sandwich Code”? I found it in Ruby Koans.

Often, you have to do something at the beginning and at the end, even though your main lock is in the middle. Let’s say, you want to retrieve data from database. You have to create a connection, open it, use it to load data, and close it. You are interested in loading data, but you have to do some chores like opening a connection and closing it.

_connection = new SqlConnection(ConfigurationManager.ConnectionStrings["Db"].ConnectionString);
_connection.open()var sql =
    @"SELECT     DATEPART(YEAR, PDate) AS [Year],
                 DATEPART(MONTH, PDate) AS [Month],
                 DATEPART(DAY, PDate) AS [Day],
                 COUNT(*) AS [Count]
        FROM     JobPostingPeriod
       WHERE     PublishToDate >= GETDATE() AND PDate <= GETDATE()
       GROUP BY  DATEPART(YEAR, PDate), DATEPART(MONTH, PDate), DATEPART(DAY, PDate)
       ORDER BY  [Year], [Month], [Day]";

var result = _connection.Query(sql));return result;_connection.close()



Those “bread-in-the-sandwich” code is repeated whenever you write any code that load data, and it can be quite annoying as you go on. In ruby, you can write a method that handles the opening and closing with block. In C#, you can use lamda delegate and Func.

I wrote a simple method, “ManageConnection”, with the help of Dan, my Totaljobs colleague, as he happened to be with me.
It accepts an anonymous method that has SqlConnection as input, and return T.
And now I can re-write the data loading method like this.
private T ManageConnection(Func func)
{
    _connection.Open();
    var result = func(_connection);
    _connection.Close();
    return result;
}
The excution flow is
1. GetJobCountByDate is called
2. It calls ManageConnection and pass conn.Query(sql) as anonymous method
3. In ManageConnection, _connection is opened.
4. In ManageConnection, func(_connection) is called. func is conn.Querysql at this point.
5. ManageConnection returns the result.
6. GetJobCountByDate receives the result from ManageConnection, and now returns the result.
I think this is nice and clean, once you understand how it works.
There are loads of sandwich code. If you write file, access database, write to http stream, …

	ection.open()
var sql =
    @”SELECT    DATEPART(YEAR, PostedDate) AS [Year],
                DATEPART(MONTH, PostedDate) AS [Month],
                DATEPART(DAY, PostedDate) AS [Day],
                COUNT(*) AS [Count]
       FROM     JobPostingPeriod
      WHERE     PublishToDate >= GETDATE() AND PostedDate <= GETDATE()
      GROUP BY  DATEPART(YEAR, PostedDate), DATEPART(MONTH, PostedDate), DATEPART(DAY, PostedDate)
      ORDER BY  [Year], [Month], [Day]”;
var result = _connection.Query(sql));
return result;
_connection.close()_connection = new SqlConnection(ConfigurationManager.ConnectionStrings[“JobSeekerReadOnly”].ConnectionString);
_connection.open()
var sql =
    @”SELECT    DATEPART(YEAR, PostedDate) AS [Year],
                DATEPART(MONTH, PostedDate) AS [Month],
                DATEPART(DAY, PostedDate) AS [Day],
                COUNT(*) AS [Count]
       FROM     JobPostingPeriod
      WHERE     PublishToDate >= GETDATE() AND PostedDate <= GETDATE()
      GROUP BY  DATEPART(YEAR, PostedDate), DATEPART(MONTH, PostedDate), DATEPART(DAY, PostedDate)
      ORDER BY  [Year], [Month], [Day]”;
var result = _connection.Query(sql));
return result;
_connection.close()
_connection = new SqlConnection(ConfigurationManager.ConnectionStrings[“JobSeekerReadOnly”].ConnectionString);
_connection.open()
var sql =
    @”SELECT    DATEPART(YEAR, PostedDate) AS [Year],
                DATEPART(MONTH, PostedDate) AS [Month],
                DATEPART(DAY, PostedDate) AS [Day],
                COUNT(*) AS [Count]
       FROM     JobPostingPeriod
      WHERE     PublishToDate >= GETDATE() AND PostedDate <= GETDATE()
      GROUP BY  DATEPART(YEAR, PostedDate), DATEPART(MONTH, PostedDate), DATEPART(DAY, PostedDate)
      ORDER BY  [Year], [Month], [Day]”;
var result = _connection.Query(sql));
return result;
_connection.close()
Handling Sandwich Code in C#

Testing private method in unit testing

Ideally, it is not recommended to test private methods, as complex internals should be kept hidden and only interface be tested. Well, in real world, sometimes, you need to test your private methods, as it performs very important operation.

In my case, the method was public and later I changed it to private, since it is better to be hidden. Yet, I still wanted to test it, as it does lots of things – calling several other private methods and sum up the result.

You can use .Net reflection and easily do it. One nice article I found by googling was http://www.codeproject.com/KB/cs/testnonpublicmembers.aspx. Simple and straight-forward.

I created a static helper method. I changed variable names in the above example, as it follows old C++ style. These days, C# guys prefer longer variable name without type prefix.

public class TestHelper
{
    public static object RunStaticMethod(Type type, string method, object[] parameters)
    {
        return RunMethod(type, method, null, parameters, BindingFlags.Static | BindingFlags.NonPublic);
    }

    public static object RunInstanceMethod(Type type, string method, object instance, object[] parameters)
    {
        return RunMethod(type, method, instance, parameters, BindingFlags.Instance | BindingFlags.NonPublic);
    }

    private static object RunMethod(Type type, string method, object instance, object[] parameters, BindingFlags flags)
    {
        try
        {
            MethodInfo info = type.GetMethod(method, flags);
            return info.Invoke(instance, parameters);
        }
        catch (Exception ex)
        {
            throw ex;
        }
    }
}
Testing private method in unit testing

File is rejected by a third-party system, when it is utf-8 encoded with BOM

Recently, I had a chance to refactor codes for a system that sends a file to DreamMail. Because we did not touch any FTP functionality, we were complacent and tested it up to where files are exported and sent via FTP. Development is completed and the application was deployed to a testing environment. Very thankfully and luckily, Jon (a team mate) and I happened to see that all new files generated by the refactored code are rejected in DreamMail ftp server. First we thought they fail because we use a testing account but it was worring that all files fail. We started investigating the issue. Picked one file that was successfully processed and uploaded it again. It worked without any problem. Then we compared the two files, successful one and failed one. Yet there was no difference. I assumed that something was different but was not visible, so downloaded Ultra-Edit because it can display file in hex code and checked the two files. The difference was the first 3 characters which is called BOM, Byte-Order Mark.

BOM is zero-width, no-break space and therefore, not visible in most of text editor. “It is conventionally used as a marker to indicate that text is encoded in UTF-8, UTF-16 or UTF-32.” (wikipedia) In hex value it is EF BB BF.

If the file stars with BOM, DreamMail client rejects it for some reason, thinking it is malformed.  My code that prepended BOM to the file was this.

using (StreamWriter sw = new StreamWriter(fs, Encoding.UTF8))

Fix for this is simple. Do not specify encoding in StreamWriter constructor.

using (StreamWriter sw = new StreamWriter(fs))

There is a post in Expert Exchange that tells you to use Encoding.ASCII. It fixes the problem, but all text will be written in ASCII. You will lose lots of characters.

File is rejected by a third-party system, when it is utf-8 encoded with BOM