Remove Server Headers in Azure App Service for ASP.NET Core

I'm late to the game, mostly because I haven't been working on web apps for quite some time, but I recently wanted to do some basic security functions on a .NET Core 2.2 application being deployed to an Azure App Service, namely removing headers that identify the platform being used. I have a pretty standard way that I'd done this previously for ASP.NET MVC, but those methods don't work in .NET Core. After doing some digging around, I found that I needed to add a skeleton web.config to configure the web server in the Azure app.

In the configuration included below, you'll see that I'm using the HttpProtocol element to remove the X-Powered-By header. To remove the Server header, I'm using the request filtering feature of IIS 10.0 that has been added to the Azure App Service web server.

 <?xml version="1.0" encoding="utf-8"?>  
 <configuration>  
  <system.webServer>  
   <handlers>  
    <remove name="aspNetCore" />  
    <add name="aspNetCore"   
       path="*"   
       verb="*"   
       modules="AspNetCoreModuleV2"   
       resourceType="Unspecified" />  
   </handlers>  
   <aspNetCore processPath="%LAUNCHER_PATH%"   
         arguments="%LAUNCHER_ARGS%"   
         stdoutLogEnabled="false"   
         stdoutLogFile=".\logs\stdout"   
         hostingModel="InProcess">  
   </aspNetCore>  
   <httpProtocol>  
    <customHeaders>  
     <remove name="X-Powered-By" />  
    </customHeaders>  
   </httpProtocol>  
   <security>  
    <requestFiltering removeServerHeader="true" />  
   </security>  
  </system.webServer>  
 </configuration>  

NOTE: I'm not using Kestrel for this service; it's being hosted by IIS. If you are using Kestrel, you can use middleware to remove the Server header.

  public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>  
    WebHost.CreateDefaultBuilder(args)  
           .UseKestrel(o => o.AddServerHeader = false)  
           .UseStartup<Startup>();  

Controller Testing

Prompted by a tweet from @amandaiverso (retweeted by Scott Hanselman), I've decided to write down a few thoughts on what I consider to be "industry standard" (or at least my standard) for testing controllers. First, I should say that most often these days I'm writing APIs not web sites. That will likely skew my thoughts, but I'll try to be more general or at least differentiate where necessary between the two because there are subtle differences. My thoughts are going to cover both how and what to test. The what probably goes at least somewhat to what I think controllers ought to do, not just what you ought to test. I haven't spent a lot of time thinking this through - it's mostly a quick response to the tweet. I'd be happy to get additional thoughts about this from others.

Kinds of Tests

For controllers I typically do 2 or 3 types of testing. The two types of tests that I would always create are integration tests and unit tests. In some, but not all scenarios, I might add a quasi-integration test. What do I mean by that? In my practice it would be an automated test using a unit-test framework but invoking an in-memory server implementation. This type of test allows me to test the plumbing set up around the controller to ensure things like any custom model-binding or authentication is properly configured. In this latter case, because I typically mock out everything but the controller, I would consider it mostly a controller integration test, but I'd typically have them executed with the unit tests because they can be done fast enough to be run as unit tests and in isolation from other parts of the environment so that they can easily be run by the build server.  For API tests, I would probably also write unit tests using reflection to validate key parts of my attribute-based Swashbuckle documentation - ensuring for example that the return values of any actions match that documented via the attribute and that all public actions have certain types of document attributes.

Approach to Testing

Generally I follow an approach that keeps controller actions thin. Normally this means that my controllers handle some (but not all) validation, authorization, and output formatting.  They delegate business logic and data access to services. The things that I would test then are both success and failure conditions for the validation that is handled at the controller level, valid and invalid access, and proper translation of service models into view-specific models either for use in an MVC view or for presentation on the wire in an API. In the latter case, I'm mostly checking that the model is populated correctly. I don't typically mock out my mapping services, usually AutoMapper, so this functions as a check that they are configured properly when used.

Validation

Much of the time I used model-based validation. Typically I'll ensure that my controller checks Model.IsValid. If I'm using Fluent validation, then I'll make sure that it invokes the validator and properly responds depending on the validator's response. For Model.IsValid checking, this is typically configured in the test set up by constructing a controller context and setting the ModelState.IsValid property directly. In my controller tests for this I'm not particularly interested in whether the model binder works correctly ("don't test the framework"), just that my code responds correctly when it is either valid or invalid.  I'll test this by checking the output of the action being tested.

For manually invoked validation, I'll use mocks with particular set ups, then verify that the validator is called, then test the output of the action to ensure that the controller properly responded to the validation result the mock returns.

Access

Since access is frequently enforced by attributes, this is a place where I might use quasi-integration tests. If an API is public or all access is controlled by restricting IP ranges, then I might skip these. If I am testing these, then the tests use the set up to establish responses from authentication and authorization services, create a request using the credentials that trigger the configured responses, then checking that the proper response code is return in failure cases or that a valid OK response is returned in success cases. These tests typically don't inspect the response values, just the error codes. Some configuration for other services is required to ensure that you don't get errors from invalid data, but these can be configured loosely to avoid tight coupling to their implementations. Using something like AutoFixture works well to ensure that I get valid data without too much configuration trouble.

Model Translation

I typically treat model translation as whitebox testing. If I'm using something like AutoMapper, then I'm probably not rigorous in testing every individual property is being set. Those things that are a straightforward translation from the entity model to the view model don't really need to be tested. You can verify that they are handled by validating the AutoMapper configuration - that will let you know where you might have missing properties.  Generally, I'll restrict myself to making sure an important property has been copied correctly (the Id property is a good choice if it's a primary key) and then any custom mappings/flattenings that I expect to occur for that particular model.  Again, I don't try to test the framework, in this case AutoMapper, but only those things that I've customized. When testing one property, say of a flattened model, will suffice to ensure that the entire model has been flattened, I'll do that. Where it's possible to make mistakes, I'll be more rigorous.

Sustainability: Refactoring, Design, and Testing

I recently saw a Twitter post by Mark Seemann preferring the term "sustainable code" over "maintainable code." As it happens I had also been having some conversations about TDD (Test Driven Design) at work. Most people, when they think of unit testing, think the purpose is to reduce code errors. To be sure, that is an important aspect of unit testing and writing tests first, I think, has led to me making fewer errors and introducing fewer bugs than I would otherwise. I find this out every time I prototype something without tests then convert it into production code by introducing tests after the fact. (Yes, I'm not a TDD purist) Invariably as I start writing tests I find cases that I hadn't thought about and even - gasp - errors that I didn't even recognize but that were obvious after having written a test.

The concept of "sustainable code" is really why I've settled on TDD as a core practice. I should say TDD and refactoring because I believe the two go hand in hand - at least for me, and if my experience with other developers I work with over time is an indication of general trends, likely you as well.

In my experience, if you don't write tests first, you don't write tests - or at least many tests, the number of tests that it would take to allow you to refactor safely and confidently. In my internal conversations I talked about "refactoring with abandon" but it's not as undirected as that implies, but certainly you will have a lot less fear of breaking things and will engage in refactorings that would otherwise seem foolish indeed. I have both written test-first and test-last. Writing tests last feels like an onerous chore that you have to force yourself to do. It's easy to write just the happy path tests - and since you've already written the code, you know how to write a test that makes that code pass.  When I write tests first, it's part of design. I'm thinking, not of how to write a passing test, but how to make the code do all the things I think it should do. It's a creative activity not a bookkeeping activity. It makes a HUGE difference in the joy I get from writing that code.

Refactoring, continually improving code as you're creating new features, is the key to sustainable code. I'll note here that "refactoring" is not "changing what the code does" but rather "how the code does it."  In other words, if you've written tests, then those tests shouldn't need to change as a result of a refactoring. If they do, then you've over-specified in your tests. That sometimes happens, but usually you can easily tell whether a refactoring has broken behavior or broken a specification that was too prescriptive.  In that way refactoring can improve both your tests AND your code.

The key to being able to refactor with confidence is having a set of tests that will tell you when what you have changed breaks something in an unintended way.  If I make a change and a test starts failing, and that failure isn't merely an over-specification, then that change isn't an improvement, it's a bug.  I need to go back and continue making the improvement to account for the behavior I've broken.  This is especially true once the code has gotten complicated enough that you can no longer keep an accurate model of how things work in your head. At that point you begin to rely more heavily on your tests as a safety net; they're the embodiment of the working knowledge you had about that part of the code at the time that you most understood it.

Once you're able to refactor with confidence, you have a platform upon which you can build a system sustainably.  When you see inevitably see a piece of code that you now know is crappy, given your increased understanding of programming or changes in technology that allow a better solution, you can change it with confidence, knowing that the code still does what it was intended to do.  When you add a new feature to something that is a dependency for something else, you can be sure that you're not breaking anything upstream that relies on it because you have tests to ensure that you're not breaking existing behavior.

For me the big win with TDD is sustainability. You can continue to improve and extend the useful life of your software without the expense of a ground-up rewrite. There are huge cost savings over the lifetime of your software that result from using TDD and refactoring. Fewer bugs is a really, really nice side-effect, but it's not the primary reason I do TDD any more. It's about keeping the codebase clean and easy to work in, and easier to innovate on and add new features that drive revenue or customer satisfaction.

Converting a Console Application to an Azure WebJob

I recently gave a talk at Iowa Code Camp on Working with WebJobs.  I got a follow up request for resources on converting an existing console application to a WebJob. The process is relatively simple and involves three steps.

Create an Azure WebApp to host your WebJob.

Let’s say you have a solution that contains a console application such as the one shown below. We have a Program class that contains the Main method. This method creates the object that performs the task and its dependencies, then invokes one or more methods on the class to perform the work. You’d probably have some logging and some error handling, but we’ll omit that for the sake of clarity.

Now, add a WebApp project to the solution.  We’ll call it WebJob.Host.  I’m creating an MVC project because I want to have a single action that can be used to display the version of the code that’s deployed as a sanity check. If you have an existing site you’re deploying as an Azure WebApp, you could use that instead.  It will need to be in the same solution as the project that we will be converting to a WebJob.  Remember to update all the packages once you’ve created the project and to clean up any boilerplate that you don’t want.  I’m going to get rid of all but the HomeController and Index action.  Don’t be surprised if VisualStudio needs to restart to complete updating your packages.

The best way to autogenerate your version information is using the capabilities of your build server. Both TeamCity and AppVeyor support this. If you’re interested in a way to update your Assembly from your repo with version information without using a CI server, I’ve written up a way to do it using MSBuild Community Tasks.  I’m going to take the easy way out and manually keep my AssemblyInformationalVersion up-to-date for this project.

Make sure your hosting web application works before adding a WebJob to it.

Convert Your App to A WebJob

First, we need to add the appropriate WebJob Packages. At this point I’ll assume you’re not using any Azure resource or, if you are, you can figure out the additional WebJob packages you’ll need to work with those resources. If you’re using a ServiceBus queue or topic listener, it might be easier to start from scratch and create a WebJob with an appropriate listener, then merge your existing dependencies into that rather than try to convert the older service bus client code over.

For a standard WebJob that is run on a schedule, you’ll need the following package:

Microsoft.Azure.WebJobs

(note: This has several dependencies, which in turn have more dependencies. Don’t be alarmed by this.)

Again, it’s a good practice to make sure you update your dependencies after you add them to your project to make sure you have the latest versions.

This is what my packages.config file looks like after installing only the WebJobs package and it’s dependencies.

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Microsoft.Azure.KeyVault.Core" version="1.0.0" targetFramework="net452" />
  <package id="Microsoft.Azure.WebJobs" version="1.1.2" targetFramework="net452" />
  <package id="Microsoft.Azure.WebJobs.Core" version="1.1.2" targetFramework="net452" />
  <package id="Microsoft.Data.Edm" version="5.7.0" targetFramework="net452" />
  <package id="Microsoft.Data.OData" version="5.7.0" targetFramework="net452" />
  <package id="Microsoft.Data.Services.Client" version="5.7.0" targetFramework="net452" />
  <package id="Microsoft.WindowsAzure.ConfigurationManager" version="3.2.1" targetFramework="net452" />
  <package id="Newtonsoft.Json" version="9.0.1" targetFramework="net452" />
  <package id="System.Spatial" version="5.7.0" targetFramework="net452" />
  <package id="WindowsAzure.Storage" version="7.1.2" targetFramework="net452" />
</packages>

Now that we have the appropriate packages, we can convert our Program to get the job set up and invoked as a WebJob. I suggest the following approach.  Create a class named Functions (this is the convention) and add a method – it can be static or an instance method, whichever fits your needs best.  Migrate the code from your Main method to the new method in the Functions class.  Decorate the new method with [NoAutomaticTrigger].

For reference here is the Program that I started with.

internal class Program
{
    public static void Main(string[] args)
    {
        var task = new OfflineTask(new NotificationService(), new WorkDataSource());

        task.DoTask();
    }
}

Here is the Functions class that I created to replicate it's functionality. You'll notice that I'm using manual dependency injection to allow this to be testable. Don't let that throw you. I think it's a good idea, but I could have just copy the contents of Main and pasted them directly into the Execute method. Note also that Execute now has a TextWriter parameter named log. This is the hook into the WebJobs logging facility. For now, we just add log messages that we are starting and finishing. You can continue to use whatever logging you already have in place. If you choose to integrate with the WebJobs logging you can by passing the log parameter to your classes/methods as needed. There are also ways to connect this log facility to an existing one but that's beyond the scope of this article.

internal class Functions
{
    private readonly NotificationService _notificationService;
    private readonly WorkDataSource _workDataSource;

    public Functions()
        : this (new NotificationService(), new WorkDataSource())
    {
        
    }

    public Functions(NotificationService notificationService, WorkDataSource workDataSource)
    {
        _notificationService = notificationService;
        _workDataSource = workDataSource;
    }

    [NoAutomaticTrigger]
    public void Execute(TextWriter log)
    {
        log.WriteLine("starting job");

        var task = new OfflineTask(_notificationService, _workDataSource);

        task.DoTask();

        log.WriteLine("job completed");
    }
}

Now that we have our functions created, we’ll modify the Main method to set up a JobHost and use it to invoke our new Execute method. Before we do that, though, we need to add some configuration values. We need connection strings for the WebJobs dashboard and for WebJobs storage, by convention these are named AzureWebJobsDashboard and AzureWebJobsStorage. I recommend that you store these in a file that is not checked into source control and reference it from your App.Config file.

Create or use an existing storage account. Copy the connection strings from the Azure portal and add them to your connection strings.

<connectionStrings>
  <add name="AzureWebJobsDashboard" connectionString="your-connnection-string-copied-from-Azure"/>
  <add name="AzureWebJobsStorage" connectionString="your-connnection-string-copied-from-azure"/>
</connectionStrings>

Now, in your Main method, remove the code you copied over to Functions.Execute and insert the code to create a JobHostConfiguration, then a JobHost using that configuration. Then use reflection to get the method or methods that you want to execute when the program is run from your Functions class.  For each of these methods, use the Call method on the JobHost to invoke the method.  The JobHost knows to add the TextWriter to the parameters when calling the method.  You can also supply a custom activator to the JobHostConfiguration so that you can hook into your favorite dependency injection framework if you want.  There’s an example of this in my WebJobs talk demonstration code.

Here’s the updated Program code after changing it to invoke the Function as a WebJob.

internal class Program
{
    public static void Main(string[] args)
    {
        var config = new JobHostConfiguration();

        var host = new JobHost(config);

        var tasks = typeof(Functions).GetMethods()
                                     .Where(m => m.GetCustomAttributes(typeof(NoAutomaticTriggerAttribute), false).Any());
        foreach (var method in tasks)
        {
            host.Call(method);
        }
    }
}

Lastly, to get the job to run on a schedule, add a settings.json file as a Content item (I use Copy Always) to the project specifying when the job should run. This is used by the scheduler in Kudu, the underlying framework that manages WebJobs. It has a relatively simple format. The only property we are going to set is the "schedule" - that's always been enough for me. The "schedule" property is a cron-like entry ({second} {minute} {hour} {day} {month} {day of the week}) specifying (in UTC) when to run the program. The following example runs the WebJob at 5AM UTC every day.

{ "schedule" : "0 0 5 * * *" }

Add the WebJob to the WebApp

Now that we have both the Web App and the WebJob set up, we just need to add the WebJob to the WebApp so that when we publish the WebApp, the WebJob is published as well and will run on its schedule.

Right-click on the host project, I named it WebJob.Host, then choose Add, then choose Add Existing Project as Azure WebJob.

This brings up a configuration Wizard. Choose the project you want to add, give it the name you want to see in the Azure console – there are some name restrictions, for example you can’t use dots in the WebJob name. Set the run mode to OnDemand – our schedule is included in the settings.json file so we don’t need to set a schedule here. Click OK to add the WebJob to the WebApp.

This will install the WebJobs publishing package. It adds a webjobs-list.json file in the Properties folder of your Web App.  This lists the jobs that get published with the Web App and their relative location in the project. It also adds a webjob-publish-settings.json file to your WebJob project, again in the Properties folder. You might see some JSON validation errors in this file. I haven’t found these to cause problems when deployed, but I generally clean up the unused properties to remove the errors.

Now, we’re essentially done. Verify your configuration values and publish your host WebApp. Check, using the Azure portal – see the WebJob pane in settings – to make sure your job was deployed and will be run on the schedule that you’ve chosen. Use the Logs to view the log messages written to the TextWriter. You can also access Kudu directly via the Tools pane on the Web App to dig deeper to test and debug your Web Job.

Code used for this post can be found on GitHub. Note that both the original console app and the converted app (now WebJob) are included in the code so you can compare the before and after conversion states. You would probably only have the single project, converted in place, in your solution.

Why are Estimates Often Wrong: A Response

This article, Why are software estimates regularly off by a factor of 2-3 times? showed up in my Twitter feed today It was billed as one of the best explanations of the "futility of software estimation." Go read it now or the rest of this won't make much sense. I'll wait.... ok, now that you're back... While I'm generally in agreement with the #NoEstimates folks (movement?), I don't agree with that description of the problem as being the root cause as a general rule. Yes, it might, maybe even probably, applies in the context in which it was written - start ups. That doesn't include most of us. When I push back on giving an estimate, using that as my defense is going to get me laughed out of the room.

The article lays out a fictitious trip between SF and LA with the narrator looking at a map and using a basic rule of thumb on velocity and the "as the crow flies" distance between SF and LA to come up with an estimate of 10 days to make the trip. The basic premise of the article is that we can't estimate what we don't know. That is absolutely true. But there are some other things that are true as well, especially if you can't avoid making the estimate (and I agree, working in a way that makes estimates unnecessary is the ideal, it's just not always possible).

Here's one truth. You should find someone who has made the trip, or one like it. Have them tell you how long they think it will take. Better yet, have someone who has made the trip several times, with different people, under different conditions. Even better find several such people - with different backgrounds (skills). Put them all in a room to talk about their experiences as it relates to this trip, then have them all come to consensus on what they think it will take.

Here's another. If you truly can't find anyone with experience with a particular journey: don't give an estimate on how long the whole journey will take until you've actually done part of it. In particular, if you can see some spots on the map that look like they might be difficult, do some (not necessarily all) of those parts. Then, revisit your estimate. Ok, you still won't be accurate - but you should be more accurate. At the very least, you'll be more likely to over-estimate than under-estimate if you base it on experience with what you think are the difficult parts.

Another truth. There were a lot of unstated assumptions in that story. The road is flat. The road is smooth. There are no obstacles. No one will need a break during the trip. Anyone who has done any hiking before - in CA or not, along the coast or not - will be able to tell you that those assumptions are unrealistic. Expose those assumptions and they become readily apparent. If the narrator had told his friends how long he thought it would take AND the conditions under which he was making his estimate, I'm guessing the friends push back on the estimate as unrealistic. At the very least, you've given yourself (and them) the ability to evaluate your assumptions and test them.

Finally, though I could probably go on, if you give an estimate that allows you to mark a particular day on the calendar instead of range of days, you're treating your estimate as a measurement. Estimates should be ranges, not numbers. They should also be held loosely. Yes, go ahead and make plans based on your ranges, but expect to adjust those plans as you gain experience and improve your estimates on the remaining work. It should not come as a surprise that your plans have to change.

In my experience, there are many reasons why estimates are wrong - failing to account for the unexpected is only one of them and shouldn't be the primary reason. People with experience usually factor that into their estimate. The one unexpected that you can't account for and which will trip you up the most is when the destination changes mid-trip. When you are faced with a situation where you're not sure where you are going - and start-up land or new product development can be one of those places - avoid estimates for anything longer than the immediate work in front of you. If you can live without them entirely, do so. If you can't, then use experience, spikes, exposing assumptions, and ranges rather than numbers, to help make the necessary estimates and use them appropriately.

Twin Cities Code Camp: Presentation

Links to information used in my presentation on Automated Testing: Patterns and Practices

Slides (SpeakerDeck)

Lightweight Integration Testing with Dapper

Dapper Example Repository

Convention-Driven Automatic Release Versioning from Your Git Repo

Introduction

I have a project where I wanted to automatically generate some information about what version has been deployed so I can ensure that, post-deploy, the correct version has been deployed to production easily.  For my purposes, I chose to use a mechanism that depends on semantic versioning of the Git release branches to avoid having to maintain version numbers inside my project. That is, assuming that my Git release branches follow a naming standard that incorporates the version of the code, I wanted to use the branch name and some other information to identify the version in the deployed application.

I found an article, Unobtrusive MSBuild: Using Git information in your assemblies, that was slightly more complex than I wanted describing how to do this with MSBuild Community tasks. Using information from the article and simplifying it some I was able to develop a fairly straight-forward way of building an AssemblyInformationalVersionAttribute attribute with the Git branch name, commit hash, and the build time.

Using the "unobtrusive approach" described in the article, this is generated in a separate, unversioned file that is re-created on every build(Properties/AutoGeneratedAssemblyInfo.cs). When done for a release it will reflect the branch being used for the release deployment. The file is removed when the project is cleaned.  Note: it's important that this file be excluded from your Git repo as it changes on every build - no sense in checking in a file that is auto-generated anyway.

Process

1. Update your .gitignore file to exclude files named "GeneratedAssemblyInfo.cs". Commit this change so that the automatically generated files we will be creating will not be added to the repository.
   
2. Unload your project and open the project file in the editor.  Right-click on the project name in Visual Studio and select Unload Project. Right-click again on the (unloaded) project and select Edit <project-name>.
   
3. Install MSBuildTasks NuGet package into the project. The original author added a separate “dummy” build project to the solution. I found that adding it to an existing project works equally well. This will add a .build folder to your solution with the MSBuild Community tasks and targets.
   
   Note: you may need to set the PowerShell Execution Policy for this to work. I found that I needed to set the 32-bit PS Execution Policy to RemoteSigned on my 64-bit system as well using:

   start-job { Set-ExecutionPolicy Unrestricted } -RunAs32 | wait-job | Receive-Job 
   

   
   This needs to be executed from a PowerShell console window running as administrator.
   
4. Add a Project Group to your .csproj referencing the community tasks that you just added. This property group defines the path to the community tasks folder, relative to the solution directory, the path of the generated assembly info file, the assembly copyright condition and extends the build/clean item groups so that the targets that we will define later are called at the appropriate times.

   <PropertyGroup>
     <MSBuildCommunityTasksPath>$(SolutionDir)\.build</MSBuildCommunityTasksPath>
     <GeneratedAssemblyInfoFile Condition=" '$(GeneratedAssemblyInfoFile)' == '' ">$(MsBuildProjectDirectory)\Properties\GeneratedAssemblyInfo.cs</GeneratedAssemblyInfoFile>
     <AssemblyCopyright Condition="'$(AssemblyCopyright)' == ''">
     </AssemblyCopyright>
     <BuildDependsOn>
       SetAssemblyVersion
       $(BuildDependsOn)
     </BuildDependsOn>
     <CleanDependsOn>
       $(CleanDependsOn)
       SetAssemblyVersionClean
     </CleanDependsOn>
     <TargetFrameworkProfile />
   </PropertyGroup>
   

5. Import the MSBuild Community task targets. Note original article describes also having to define a UsingTask for the Git branch command. I found that this already existed in the version of the community targets I was using, 1.4.0.88. Add the following directive with your other Import directives.

   <Import Project="$(MSBuildCommunityTasksPath)\MSBuild.Community.Tasks.Targets" />
   

6. Add the SetAssemblyVersion target which actually builds the GeneratedAssemblyInfo.cs file with the AssemblyInformtionalVersion attribute applied to the assembly being built. Also add a BeforeBuild target that depends on the SetAssemblyVersion target you just added to make sure that it is invoked. Note: the build target below is simplified from the original article to include only those aspects of the Git information that are to be used and does not include any manually maintained version information or build computer information as that was not necessary for my requirements.

   <Target Name="BeforeBuild" DependsOnTargets="SetAssemblyVersion">
   </Target>
   <Target Name="SetAssemblyVersion">
   <PropertyGroup>
     <BuildTime>$([System.DateTime]::UtcNow.ToString("yyyy-MM-dd HH:mm:ss"))</BuildTime>
     <AssemblyCopyrightText Condition=" '$(AssemblyCopyright)' != '' ">$(AssemblyCopyright) $([System.DateTime]::UtcNow.Year)</AssemblyCopyrightText>
     <AssemblyCopyrightText Condition=" '$(AssemblyCopyrightText)' == '' ">
     </AssemblyCopyrightText>
   </PropertyGroup>
   <GitVersion LocalPath="$(SolutionDir)">
     <Output TaskParameter="CommitHash" PropertyName="CommitHash" />
   </GitVersion>
   <Message Importance="High" Text="Commit is $(CommitHash)" />
   <GitBranch LocalPath="$(SolutionDir)">
     <Output TaskParameter="Branch" PropertyName="GitBranch" />
   </GitBranch>
   <Message Importance="High" Text="Branch is $(GitBranch)" />
   <AssemblyInfo CodeLanguage="CS" OutputFile="$(GeneratedAssemblyInfoFile)" AssemblyInformationalVersion="$(GitBranch)-$(CommitHash), built $(BuildTime) UTC" AssemblyCopyright="$(AssemblyCopyrightText)" />
   </Target>
   

7. Add the SetAssemblyVersionClean target to delete the generated assembly info file when the project is cleaned.

   <Target Name="SetAssemblyVersionClean" Condition="Exists($(GeneratedAssemblyInfoFile))">
     <Delete Files="$(GeneratedAssemblyInfoFile)" />
   </Target>
   

8. Add a compile directive to the ItemGroup containing your other compile directives for the generated file.

   <Compile Include="$(GeneratedAssemblyInfoFile)" />
   

9. Save your changes to the project file, close, and reload the project in your solution. Build the solution. Observe in the build output that the commit hash and branch information messages specify the current values.

Usage

In my project I have a MaintenanceController that is restricted to administrators. This controller outputs a maintenance page with information about the current version and other system information. The version is obtained from the AssemblyInformationalVersionAttribute we created.

using System.Linq;
using System.Reflection;
using System.Web.Mvc;
using System.Web.UI;

using MyProject.Web.ViewModels;

namespace MyProject.Web.Controllers
{
    [Authorize(Roles = Role.ADMIN)]
    public class MaintenanceController : Controller
    {
        private static readonly string _currentProductionTag;
        private const int ONE_DAY = 60 * 60 * 24;

        static MaintenanceController()
        {
            try
            {
                _currentProductionTag = Assembly.GetExecutingAssembly()
                                               .GetCustomAttributes(typeof(AssemblyInformationalVersionAttribute), false)
                                               .OfType<AssemblyInformationalVersionAttribute>()
                                               .First()
                                               .InformationalVersion;
            }
            catch
            {
                _currentProductionTag = "Unknown";
            }
        }

        [HttpGet]
        [OutputCache(Location = OutputCacheLocation.Server, Duration = ONE_DAY)]
        public ActionResult Index()
        {
            return View(new MaintenanceIndexViewModel()
            {
                CurrentProductionTag = _currentProductionTag,
                // other information
            });
        }

    }
}

Converting a Legacy LINQ-to-SQL Data Context to Entity Framework

Recently I worked with a client to upgrade their web application to use the latest technology available for ASP.NET MVC. This included upgrading the .NET framework, upgrading existing NuGet packages (and using packages to replace libraries tracked in SCM), and replacing custom code that duplicated current framework features. The largest change, however, was to upgrade the data access layer to use a combination of Entity Framework and Dapper instead of LINQ-to-SQL (L2S). Because this is client code I'm only including a few isolated snippets and not a link to the repository in this article.

While there are many similarities between L2S's DataContext and Entity Framework's DbContext implementations, there are enough gotchas that it became more of an ordeal than I had anticipated.

Reverse Engineering the Model

I started out by using the Entity Framework Power Tools to reverse engineer the existing database. I did this into a separate solution to avoid conflicts with the existing model classes and data context. Once you've added the power tools you will get an Entity Framework option with some new sub-options when you right-click on a project. I used the Entity Framework -> Reverse Engineer Code First power tool to generate the model classes, fluent mappings, and DbContext implementation.

Once I had the model reverse engineered, I manually adapted the reverse engineered classes to reflect the changes made to the L2S model over time.  This included adding some short cut properties – say to find the first item in a collection – and convenience methods.  I’ll note here that my goal was to do an “in-place” upgrade, not refactor the architecture. Even though there were places that refactoring would improve the code I felt that it was too risky to attempt much refactoring AND swap the underlying data access technology at the same time. There were only  a few places where there was some obvious code duplication that I ended up changing in this way; mostly I simply translated the code from the L2S classes to the EF classes. Because I was doing this in a separate solution this involved copying some code over from the existing solution where the existing models referenced types outside of the model.  Mostly these were enumerated types (enums) that were used for some model properties. By and large the model classes were well-isolated from the rest of the solution.  The fact that the previous developers followed good principles made the transition much easier.

Even so, there were a few areas of “gotchas” where the design choices in EF didn’t match those in L2S that caused me problems.

Configuring Relations

This was probably the largest area of difficulty that I faced. There were 3 distinct types of mapping issues that I had with relations. First, not all of the relations in the previous model had been mapped. In those cases I simply removed the mapped relation from the reverse-engineered models. Since my goal was to replicate existing behavior this seemed like the obvious choice. Later, when we start doing optimizations on the EF code, some of these may be added back if needed to improve query performance.

Second, some relations were mapped as one-to-many using the EF tooling that had been mapped one-to-one in L2S. These were trickier to deal with because of some missing capabilities in the fluent mapping methods.  With a one-to-many mapping, the fluent EF tooling allows you to include both the navigation property and the column housing the foreign key in the model. That’s not true – or at least I could not find a way to do it – for one-to-one/zero. In those cases, the methods that allow you to specify the foreign key correlation to an model property do not exist and if you try to include the FK property, you get a column exists conflict on the model.
I handled these in two ways. If the FK property wasn’t used outside the model class, I defaulted to removing the FK property and setting the property up as one-to-one/zero in the model using WithRequired(t => t.NavProperty).WithOptional(t => t.InverseNavProperty), specifying a custom key as necessary with .Map().  If the FK property was used outside the class – which you might do in order to avoid a subquery to get the related data – I left the property mapped one-to-many and added a convenience property to find the FirstOrDefault() in the collection to replace the existing one-to-one/zero mapping. This necessitated some code changes to set up the relations but these seemed straightforward and involved the least amount of disruption to existing code.  This introduced some properties that needed to be ignored explicitly on the model, but more on that later.

Lastly, I ran into some issues with many-to-many mappings. These included many-to-many mappings that did not follow the Entity Framework convention for foreign keys. Entity Framework expects foreign keys on join tables and join tables that included extra data. These seemed particularly difficult to deal with, though I came to find eventually that what I thought were join table mapping issues were really caused by some navigation properties that had not been ignored.   L2S uses a paradigm where the mappings have to be manually added to the model. EF uses a convention-based mechanism where it applies a default mapping unless you’ve explicitly overridden or ignored it.  This mismatch caused a lot of confusion at first as the reverse engineering tool had set up all of the mappings. In the process of resolving the issues above I had inadvertently introduced some model errors with the convenience properties.

With respect to those places where the keys did not match the expected conventions, I ended up explicitly mapping the foreign keys using .Map() and specifying the appropriate key values. This also involved some code changes where I had to remove the join table model that had existed in the L2S model since L2S did not handle many-to-many mappings directly but only as bidirectional one-to-many mappings from each side to the join table.  When there was additional data in the join table, I left the join table model in the solution and set it up as the bidirectional one-to-many association.

Sample Many-to-Many Modeled as Two One-to-Many Relations

public class UserClientMap : EntityTypeConfiguration<UserClient>
{
    // model simplified

    HasKey(t => t.Id);

    Property(t => t.Id).HasColumn("Id");
    Property(t => t.IsPrimaryContact).HasColumn("IsPrimaryContact");
    Property(t => t.UserId).HasColumn("UserId");
    Property(t => t.ClientId).HasColumn("ClientId");

    HasRequired(t => t.Client)
        .WithMany(t => t.UserClients)
        .HasForeignKey(t => t.ClientId);

    HasRequired(t => t.User)
        .WithMany(t => t.UserClients)
        .HasForeignKey(t => t.UserId);
}

Excluding Properties

The existing model included some properties that were not persisted. I had made an initial assumption that if these weren't mapped in the fluent mappings, they would be automatically excluded since all of the properties in the database had been mapped in the reverse engineered mapping files. This turned out to be a false assumption; I haven't used the fluent mappings much and haven't internalized their idiosyncrasies. I had also made the assumption that properties without setters, the convenience properties from above, would also be ignored. That also turned out to be false.

I began to set up the properties causing model errors manually using .Ignore(). This would work, but given the size of the model my laziness prevailed and I tried to find a way that I could use conventions that would allow me to use less code per mapping class and help prevent errors caused by failing to update the mapping file when a model changed to add a property that should be ignored.

I landed on an extension method to the EntityTypeConfiguration class coupled with the use of NotMappedAttribute found in the System.ComponentModel.DataAnnotations.Schema namespace.

The extension class has a method, ExludeIgnoredProperties, that enforces two conventions by adding an Ignore() for the property if it contains no setter method or if it is marked with the NotMappedAttribute. Each mapping class adds a call to this.ExcludeIgnoredProperties(); and both of these conventions are enforced for this model. Because each property has a different type I ended up using "dynamic" to allow the correctly typed lambda expression to be returned when constructing the argument for the Ignore() method.

internal static class EntitytTypeConfigurationExtensions
{
    public static void ExcludeIgnoredProperties<T>(this EntityTypeConfiguration<T> configuration)
        where T : class
    {
        foreach (var expression in GetIgnoredProperties<T>())
        {
            configuration.Ignore(expression);
        }
    }

    private static IEnumerable<dynamic> GetIgnoredProperties<T>()
        where T : class
    {
        var type = typeof(T);
        var properties = type.GetProperties(BindingFlags.Instance | BindingFlags.Public);


        foreach (var expression in GetPropertiesWithoutSetters<T>(properties))
        {
            yield return expression;
        }

        foreach (var expression in GetExcludedProperties<T>(properties))
        {
            yield return expression;
        }
    }

    private static IEnumerable<dynamic> GetPropertiesWithoutSetters<T>(IEnumerable<PropertyInfo> properties)
        where T : class
    {
        var type = typeof(T);
        foreach (var propertyInfo in properties)
        {
            var setter = propertyInfo.GetSetMethod();
            if (setter == null)
            {
                var parameter = Expression.Parameter(type, "t");
                var body = Expression.Property(parameter, propertyInfo);
                var lambda = Expression.Lambda(body, parameter);
                yield return lambda;
            }
        }
    }

    private static IEnumerable<dynamic> GetExcludedProperties<T>(IEnumerable<PropertyInfo> properties)
        where T : class
    {
        var type = typeof(T);
        foreach (var propertyInfo in properties)
        {
            var ignoreAttribute = propertyInfo.GetCustomAttribute<NotMappedAttribute>();
            if (ignoreAttribute != null)
            {
                var parameter = Expression.Parameter(type, "t");
                var body = Expression.Property(parameter, propertyInfo);
                var lambda = Expression.Lambda(body, parameter);
                yield return lambda;
            }
        }
    }
}

Replicating Default LoadWith Behavior

The last major challenge I had was to replicate L2S's default LoadWith behavior. In L2S, you need to set up all of the included properties before making any queries using the data context. If you don't an InvalidOperationException is thrown. As a result, the repository implementation defined some properties that automatically got loaded with each model type when the data context was instantiated.

Entity Framework is much more flexible and allows the included relations to be defined per-query, however, I wanted to retain as much of the existing code and behavior as I could. In particular, I needed to retain as much of the efficiency of using included relations as I could without having to re-visit every query performed against the repository

This is probably the area in which my in-place upgrade resulted in the most compromises from what I would normally have implemented. Generally I use a repository per root pattern, but the existing repository was a single generic repository and the "roots" where defined via generic constraints on the model type. Each "root" model type extended an abstract (empty) Root model class and the constraints only allowed a Root class to be the base of a query.

Using a repository-per-root would have allowed me to to easily defined which related entities were loaded by default for each class. With a single, generic repository I would need to use some reflection to achieve the same effect. In addition I wanted to be able to easily override the defaults for specific queries to make them more efficient as needed. Lastly, I wanted to avoid using if/else or switch statements to define the logic. In my mind the included properties should be more closely associated with the model than the repository so I ended up opting for attribute-based definition of the included relations.

LoadWithAttribute

[AttributeUsage(AttributeTargets.Class, AllowMultiple = false, Inherited = false)]
public class LoadWithAttribute : Attribute
{
    internal IEnumerable<string> Includes { get; private set; }

    private string _include;

    public string Include
    {
        get { return _include; }
        set
        {
            _include = value;
            Includes = (value ?? "").Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries)
            .Select(s => s.Trim());
        }
    }

    public LoadWithAttribute(string include)
    {
        Include = include;
    }
}

This attribute is added to each class that needs to have default included properties defined. It takes a comma-separated string of property names for the related data to include. In the repository, the attribute is obtained by reflection but only if the developer doesn't supply a set of included properties specific to that query. Note that the signatures include both one without any included properties and one that takes a number of property names via a params. Both of these defer the behavior to an implementation method that takes an IEnumerable of property names. There's also a signature (not shown) that takes, as params, any number of Func parameters if the developer wants to specify using strongly-typed properties rather than property names.

Sample Repository Methods

public class Repository : IRepository
{
    // class simplified
    private static readonly ConcurrentDictionary<Type, IEnumerable<string>> _defaultIncludeMap = new ConcurrentDictionary<Type, IEnumerable<string>>();

    public IQueryable<T> All<T>(Expression<Func<T, bool>> filter)
        where T : Root, new()
    {
        return AllImplementation(filter, GetDefaultIncludes<T>());
    }

    public IQueryable<T> All<T>(Expression<Func<T, bool>> filter, params string[] includedProperties)
        where T : Root, new()
    {
        return AllImplementation(filter, includedProperties);
    }


    private IQueryable<T> AllImplementation<T>(Expression<Func<T, bool>> filter, IEnumerable<string> includedProperties)
        where T : Root, new()
    {
        IQueryable<T> dbSet = _context.Set<T>();

        foreach (var property in includedProperties)
        {
            dbSet = dbSet.Include(property);
        }

        return dbSet.Where(filter);
    }

    private static IEnumerable<string> GetDefaultIncludes<T>()
    {
        var includes = _defaultIncludeMap.GetOrAdd(typeof(T), type =>
        {
            var attribute = type.GetCustomAttribute<LoadWithAttribute>();

            return attribute == null
                ? new List<string>()
                : attribute.Includes;
        });

        return includes;
    }
}

Now, with these things in place I only needed to exercise the code and find the places where L2S supported constructs were used that don't have translations to EF queries. I'm still working through this exercise, performing query optimizations as I can to take advantage of the greater flexibility that EF offers. Fortunately these are relatively easy to spot (they throw exceptions) and, so far, have been relatively easy to find alternative constructs for.

Action Verification Patterns

I was reminded again about a verification pattern that I use frequently when doing TDD/unit tests. It's really two different patterns. The first when you want to ensure that some action occurs, the other when you want to make sure it hasn't.

My examples will be specific to FakeItEasy, but you should be able to extrapolate them to any mocking framework.

The use case for my specific example is the case when you're creating a new entity via WebAPI and you may be connecting that entity to an existing related entity or creating a new entity for the other end of the relationship as well. For example, you want to sign up a new user but may be associating that new user with an existing company or, if there is no existing company, create a new one at the same time as you create the user. In this case, among other tests, you'll want to test that the new company either was or was not created as appropriate.

The patterns come into play in the Assert step of the test. With the mocking framework you ensure that the call across the system boundary, in this case to the database, either did or did not happen. When testing that call was made, use the verification signature that specifies exactly which entity was to have been added.

For example,

A.CallTo(() => _testContext.MockCompanies
                           .Add(A<Company>.That
                                          .Matches(c => c.Name == expectedCompanyName))
 .MustHaveHappened();

The other pattern, when you want to exclude an action, is similar except you use the verification signature that accepts any matching entity.

A.CallTo(() => _testContext.MockCompanies.Add(A<Company>.Ignored))
 .MustNotHaveHappened();

In the first, case you make sure that the exact expected instance has really been added to the database. In the second, you ensure that no entity of that type has been added to the database. By matching on all entities in the second case you avoid an over-specification in which an error in the spec might give you a false positive by failing to match a call because of an error in the condition.

Autofac Extensions: It’s the Little Things

I get tired of reading, and writing, dependency registrations in Autofac. Very descriptive, but also very repetitive. Imagine a large application with a couple of dozen small, focused services each with a registration like:

builder.RegisterType<DescriptiveService>()
       .As<IDescriptiveService>()
       .InstancePerHttpRequest();

It's not a pretty sight.

To make this easy to read and, especially with syntax highlighting, relatively easy to spot errors I created a small extension method as part of my dependency configuration class.

public static class DependencyConfig
{
    private static void RegisterPerRequest<TImpl, T>(this ContainerBuilder builder)
        where TImpl : class, T
    {
         builder.RegisterType<TImpl>()
                .As<T>()
                .InstancePerHttpRequest();
    }

    public static void Configure(Action<IDependencyResolver> setResolver)
    {
         var builder = new ContainerBuilder();
         builder.RegisterPerRequest<DescriptiveService,IDescriptiveService>();

         // ...
    }
}

I'll grant you that it works better in a wide window or with shorter class names but I'm happy to not be repeating so much code.

Introducing TelegraphJS

Long ago, on a very different internet, we used IFrames extensively to layout content where that content might come from many different components on the same system. IFrames are still used but have a very different purpose these days, frequently to include code from various different servers into a seamless experience for the user rather than simply layout various components from the same system.

Recently on a project I had a need to use IFrames to provide the ability to embed and interact with content from a different server on a web page. The basic idea is that by including some JavaScript on the page, you create an IFrame “gadget” whose source lives on a different server and a button which allows you to open and close the “gadget” both from a button on the parent page and from inside the “gadget.”  I also had a need to trigger some complex behavior within the “gadget” based on the URL parameters found on the request to the parent window.  To do this I developed some custom JavaScript that uses postMessage to communicate between the parent window and the IFrame.

After doing this I decided to abstract that code an make a more generalizable framework, which I’ve called TelegraphJS, that makes it relatively easy to communicate between windows and trigger actions in the other window via callbacks.  Partly this was an exercise in creating a module that can be optionally used with or without RequireJS, partly it was because I hadn’t found anything existing that did this. I confess that I didn’t look too hard.

The basic idea is that you would import the telegraph module (or the the globally available Telegraph module) to create a new message handler for a specific window.  You would optionally register a set of handlers for events (messages) that get passed to the window. Events consist of a unique string, I used the convention “APP:msg” for my events, and a data object for additional information.

Simple Example

This is a simple example using the global Telegraph module.  An example using RequireJS can be found at https://github.com/tvanfosson/telegraphjs. This bit creates the parent end of a connection to an IFrame with id create-frame, sets up a handler to receive APP:opened messages from the IFrame (well, really any IFrame since we haven’t added any security). It then sends an APP:open message to the IFrame. It's really only intended to demonstrate the syntax (and I haven't tried it). To see how it really works, look at the RequireJS example on GitHub.

I’ll note here that on initialization you may need to repeatedly send any start up messages as the IFrame’s handlers may not yet have been applied. In my use I ended up setting up an interval timer that sent startup up messages until an acknowledgement was received from the child IFRAME. My examples omit the retry behavior.

var iframe = document.getElementById('client-frame');
var contentWindow = iframe.contentWindow;

var handlers = {
    'APP:opened' : function(e, data) {
        alert('opened');
    }
};

var telegraph = new Telegraph(contentWindow, handlers);

telegraph.send('APP:open');

The client end of this code is set up the reverse. It receives the APP:open message and then sends back the APP:opened message.

var telegraph;
var handlers = {
    'APP:open' : function(e, data) {
        telegraph.send('APP:opened');
    }
};

telegraph = new Telegraph(window.parent, handlers);

Additional Features

The code today is pretty simple, but it also includes the ability to add and remove handlers via on/off similar to how jQuery event handlers are added.  At present it doesn’t accept space-separated events but that’s on my TODO list.

TDD and Me

It will come as no surprise to anyone who knows me but I have a bit of a thing for Test Driven Development (TDD).  Recently @dhh stirred the developer waters with a provocative post, titled TDD is dead. Long live testing. That set off a firestorm on Twitter (what doesn’t these days) which engendered a response from @unclebobmartin, titled Monogamous TDD. Leaving the rhetoric about fundamentalism, from both, aside, I’m more inclined to agree with the latter. I’ve found TDD to be a practice which does, as Uncle Bob says, help me:

1. Spend less time debugging
2. Document the system at the lowest level in an “accurate, precise, and unambiguous” way.
3. Forces me to write higher quality, decoupled code.

I’ve even, occasionally, gotten enough confidence from my unit tests that I do deploy without fear based on passing tests and do feel the confidence to refactor that Uncle Bob describes.

But Uncle Bob left out one, perhaps glaringly obvious and important, benefit that TDD provides that other testing strategies don’t. I suspect this because both DHH and Uncle Bob agree on the need for tests.

TDD, because it insists on writing the tests first, ensures that the tests are written at all.

My fear, based on my experience with other developers, is that despite DHH’s careful phrasing, most developers will only hear “TDD is dead.” To many, if not most, developers the question isn’t, “Do we write the tests first or last?” but “Do we write tests at all?” In my experience, TDD was the practice that gave me enough of a reason to write the tests.

Though I would agree that tests were necessary and led to higher quality code, I always found writing tests afterwards something that I would rarely, if ever, do. Why? because the code “obviously worked.” Never mind the fact that it actually probably didn’t, or at least, didn’t always work. Even now, say when I’ve written some experimental code to try and figure out how something works, I find that when I go to write the production code I discover things about the experimental code that were broken when I write the tests (first) for the production version.

Even within the last week I’ve had conversations with developers who insist that TDD slows you down and “customers don’t pay us to write tests.” While true in part, these sorts of myths are drive people to omit tests entirely, not test last or test using alternative methods. I fear that they will only be reinforced by DHH’s post.

To borrow a concept, Shu Ha Ri, from Alistair Cockburn, I feel like the “conversation” between DHH and Uncle Bob is taking place between two people at the Ri, or fluent stage that will be misinterpreted by the majority of us in the Shu, or follower, stage. Because the practices at this stage are still being ingrained and can take effort to master, they are easily abandoned when another “expert” weighs in and discredits them. In our sound-bite culture, this is especially pernicious as DHH isn’t advocating the abandonment of testing (which I think many will hear) but an alternative practice of testing.

My Practice

Or How I Make TDD Work for Me

First and foremost I consider myself to be committed to high quality code. Secondarily, I consider myself to be a pragmatist. When I started doing TDD, I tested everything. I also found myself writing prescriptive, rather than descriptive tests. I can still tend to do this if I’m not careful.

What do I mean by that? I mean that the tests were describing how things got done rather than what was being done. I think this is one of the key pitfalls of poorly done TDD and leads to tests being brittle and hard to maintain (though it's true about all tests, it's just that - mostly - those other tests are never written). Since this is one of the common criticisms of TDD, I think it's important to note that it's also something that needs to change if you're going to be successful at TDD. These days I try to write tests that are the least constraining as possible and define behavior, not implementations as much as possible.

I also try to only test meaningful code that I write. "Meaningful" is a tricky concept as sometimes I find myself going back and adding a test that I didn't think initially was meaningful...until it broke in QA. Logging is one of those things that I don't often test unless there are specific requirements around it. IOC container configuration is another. YMMV. "Code I write" is much easier. I avoid testing that the framework or language works. So, for example, no testing of C# properties (i.e., getter/setter methods) unless there is complex logic in it.

The other thing that was crippling to me when I first started with TDD, was copy-paste disease or "DRY-rot". By that I mean that I often simply copy-pasted test set-up into each test rather than treating my test code as a first-class entity and applying the same craftsmanship principles to it that I did to the production code. It seems obvious in retrospect, but this was test code, not production code (or so my thinking went). Of course, when things changed that led to a cascading chain of test changes. Again, this is one of the key complaints I hear about TDD (and again all tests).

Now I make sure to refactor my tests to keep my code DRY and employ good architectural principles in my test code as well as my code under test. In my mind it's all production code. Refactoring to DRY code, just like in your code under test, makes changing the code much less painful and easier to maintain.

I'm also frequently confronted with code that hasn't been written with TDD or whose tests haven't been maintained. As a pragmatist, I don't find it valuable to go back and write tests for existing code, especially when I'm first coming to a project and don't have a complete understanding of how that code is supposed to work.

Aside: the lack of tests makes it harder to understand such code since there isn't an "executable specification" that shows me how the code differs from the, typically, out-of-date or non-existent documentation. At best you can get an incomplete oral history from the previous developer, but even they may not be the original author and only have a limited understanding.

If I'm implementing a new feature I will write tests around that feature and refactor as necessary to support testability in the feature. In many cases this means introducing either wrappers around hard to test (read: static) implementations or alternative constructors with suitable default implementations which allow me to introduce dependency injection where it wasn't before. Less frequently, while changing features, I'll write enough tests around the feature to help ensure that I'm not breaking anything. This is easier when there are exiting albeit out-of-date tests as at least the architecture supports testability. As a last resort I do manual testing.

Almost invariably, if TDD hasn't been used, there are no or only minimal developer tests. I don't count QA scripts as tests though they serve a valuable purpose, they don't given the developer enough feedback quickly enough to ensure safety during development. This is almost universal in my experience and leads to the fears I've detailed above.

The last thing I'll mention is, as I alluded to before, I don't typically write unit tests for experiments or spikes. If I do end up using that code as the basis for production code, I try to follow the practice of commenting out the pieces of it and writing the tests as if the code did not exist and then uncommenting the relevant bit of code to pass the test. Invariably, I find that my experimental code was broken in some significant way when I do this so I try to keep it to a minimum.

The Need for Speed

I want to say a word about one the common complaints that I hear about TDD: that TDD slows development. I think that this is both true (and good) and false (and still good). First, yes, writing the tests in addition to writing the code, particularly before you write the code can add additional effort to the task. If you otherwise don't write the test at all, by doing TDD you've incurred the entire cost of writing the test. If you otherwise write the test after, by doing TDD you've incurred any refactoring you've done while writing the code.

So, yes, doing TDD can take more time per unit of code under test than either not testing or writing tests after. BUT... that time isn't necessarily wasted because it also gives you more time and the impetus to think about the code you are going to write before you write it. In my experience, this leads to better code the first time and less re-writing and can actually save time even during initial development, though it typically does increase it somewhat.

Over the long term, I think the idea that TDD takes more time is patently false. There is a body of research into the topic that shows that testing, and specifically TDD, reduces code defects and re-work significantly (cf, Is There Hard Evidence of the ROI of Unit Testing).

Ancedotally, I've also found that on projects where I've used TDD and other developers haven't tested at all, we've spent a lot more effort around the parts of the code that haven't been tested. While I'd like to take credit for my superior development skills, the reality is that I just prevented a lot of the defects that I otherwise would have made using TDD. When I'm not able to write tests, I make those same errors.

On projects where they have unit tests, I've found that when I come in later, I'm much more quickly able to get up to speed and I'm able to develop with more confidence, increasing my productivity, when I have the safety of a unit test suite. Again, in my experience, if you don't do TDD, you probably also don't do testing, period. At the very least, if you wait and write the tests afterwards, you're still incurring the costs of finding and fixing defects while the code is under development that TDD could have saved you.

Always a Newb

Finally, while I feel like I've learned a lot about testing, in general, and TDD, in particular, over the years by no means do I consider myself as knowing everything. I'm still discovering how to make it work better. I don't pretend that there aren't alternative ways that might work as well. Based on my experience, it's the best thing that I've found and I heartily recommend it to anyone who asks. I will happily abandon TDD for any practice that works better. So far I haven't found one.

I will say, though, that I know enough to be able to safely say that if you're not doing tests, unit or otherwise, don't bother weighing in on whether TDD or some other means of test is better...just start testing. If you are testing but haven't tried TDD, give it a chance. Study the pitfalls (maybe start with TDD, Where Did It All Go Wrong (Ian Cooper) and avoid them but don't blindly dismiss it based on someone else's experience because there are plenty of us that have a different, markedly better one.

Edited to correct misspellings and grammar. Now, if I could only find a way of doing TDD on my blog posts.

Runtime-typed Generic Collection Extensions

Another post inspired by a Stack Overflow question (see my answer).

The specific situation encountered by the asker of the question was prompted by a situation in which the asker knew the specific type that was being returned, but only at runtime. The asker wanted to be able to invoke a method delegate that accepted a IEnumerable<Foo> or IEnumerable<Bar> that was, presumably, passed as an argument along with the type, Foo or Bar, to a framework method that is unaware of the specific type at compile type. The problem is that when the ToList() method is used, it was returning List<object> instead of List<Foo> or List<Bar> as required as required by the delegate that the asker was attempting to invoke. This resulted in an ArgumentException for the delegate's Invoke method as the underlying type was not convertible.

While I suspect that there is probably a better way to create the framework there wasn’t enough information in the question for me to comment on that. Instead I decided to try and construct some extension methods that would produce a collection of the specific type that could be used by the delegate using ideas from the accepted answer and an answer on a similar question by the indubitable Jon Skeet.

To provide a test case for the solution I mocked up some classes similar to those in the question. First, there is a Dog, which knows how to Bark(), and a Bird, which knows how to Sing().

public class Dog
{
    private readonly int _id;

    public Dog(int id)
    {
        _id = id;
    }

    public string Bark()
    {
        return string.Format("Woof...{0}", _id);
    }
}

public class Bird
{
    private readonly int _id;

    public Bird(int id)
    {
        _id = id;
    }

    public string Sing()
    {
        return string.Format("Squawk...{0}", _id);
    }
}

Then we have a Framework class that does a query that returns a collection of IEntry objects, each being an Entry object with a Data property that is either a Dog or a Bird.

public class Framework
{
    public IEnumerable<IEntry> QueryOfAllType(Type type)
    {
        var range = Enumerable.Range(0, 10);

        if (type.IsAssignableFrom(typeof(Bird)))
        {
            return range.Select(i => new Entry
                        {
                            Data = new Bird(i)
                        })
                        .ToList();
        }

        return range.Select(i => new Entry
                    {
                        Data = new Dog(i)
                    })
                    .ToList();
    }
}

public interface IEntry
{
    object Data { get; }
}

public class Entry : IEntry
{
    public object Data { get; set; }
} 

Lastly, we have our program which will query the Framework for IEntry objects of the proper type, select the Data property, and use the new extensions to convert the collection to a collection of the proper type to be used by a delegate to perform the appropriate action for that object. Note: the code below has been updated from the previous version to more accurately replicate the original problem - that is, that the type information can't be inferred from the delegate or the invoking method.

class Program
{
    static void Main(string[] args)
    {
        DoThatThing(typeof(Bird) , Vocalize);
        DoThatThing(typeof(Dog), Vocalize);
    }

    private static void DoThatThing(Type type, Action<IEnumerable> thingToDo)
    {
        var framework = new Framework();

        var result = framework.QueryOfAllType(type)
                              .Select(e => e.Data)
                              .ToListOfType(type);

        thingToDo.DynamicInvoke(new [] { result });

    }

    private static void Vocalize(IEnumerable animals)
    {
        foreach (var animal in animals)
        {
            if (animal is Dog)
            {
               Console.WriteLine(((Dog)animal).Bark());
            }
            else if (animal is Bird)
            {
                Console.WriteLine(((Bird)animal).Sing());
            }
        }
    }
}

Below is my solution, using reflection to cast the returned collection to the appropriate type and convert the IEnumerable to a list, retaining the behavior that the actual collection rather than an iterator is produced.

public static class EnumerableExtensions
{
    private static readonly Type _enumerableType = typeof(Enumerable);

    public static IEnumerable CastAsType(this IEnumerable source, Type targetType)
    {
        if (source == null)
        {
            throw new ArgumentNullException("source");
        }

        var castMethod = _enumerableType.GetMethod("Cast").MakeGenericMethod(targetType);

        return (IEnumerable)castMethod.Invoke(null, new object[] { source });
    } 

    public static IList ToListOfType(this IEnumerable source, Type targetType)
    {
        var enumerable = CastAsType(source, targetType);

        var listMethod = _enumerableType.GetMethod("ToList").MakeGenericMethod(targetType);

        try
        {
            return (IList)listMethod.Invoke(null, new object[] { enumerable });
        }
        catch (TargetInvocationException e)
        {
            ExceptionDispatchInfo.Capture(e.InnerException).Throw();
            return null; // to satisfy the compiler, never reached
        }
    } 
}

This is what I do for fun on Saturdays. The extension code and tests are available on GitHub.

These Aren’t The API Models You’re Looking For

I recently answered a question on Stack Overflow that started me thinking about the differences between domain models and the models you expose through your Web API. While the question was only tangentially related, I thought it would be helpful to think through why I believe that domain models shouldn’t be exposed directly through your API.

I’ve long argued for model-per-view in MVC and I’ve carried that thinking over to WebAPI as well. In an MVC context, I think that separation of concerns is the strongest argument in favor of using separate view models. Views have specific data needs for display that don’t belong in your domain models. For example, they may contain auxiliary information containing localization or paging. Your views may not need or you may not desire your view models (which can also used as the arguments for POST actions) to contain all domain properties – for example, whether a user is in a role. Or you may expose this as part of the model but validate it in a completely different way, leading to differing validation requirements. Lastly, your views may flatten multiple models into a single, composite model, omitting unimportant details from that particular view’s point of view.

Many or most of these also apply to API models, even though they are or seem to be strictly about the data. While I don't claim the following to be an exhaustive list, here the reasons I think that you should use separate models for your API and map between your domain and your API models rather than directly expose your domain models.

Hiding Implementation Details

There are two reasons why you might want to hide the implementation details of your model. As in MVC, there are likely some record-keeping attributes on your domain models that are not necessary for the proper operation of the API. For example, you may track who created the item and when, who updated it and when - in a HIPPA world you may even track who viewed it, when, and, perhaps, for what purpose. You may also have actual domain information that needs to remain private even if some of the information is exposed; or it may only conditionally be available based on the API consumer's privileges. This, in itself, is probably enough of a reason to separate your API models from your domain models. Doing this allows you to keep your domain model simpler and cleaner, more appropriate to the data rather than the usage of the data. The alternative would, perhaps, be a multiplication of domain models, representing purpose-specific views and an accompanying complication of your API (or additional APIs) to support those models.

Transforming Implementation Details

Similarly, there are aspects of your model to represent relationships in the data that may be irrelevant to the API consumer. You may model a relationship as many-to-many in your domain model. For example, a product may have many accessories while an accessory may be available for more than one product. From a catalog management perspective this makes perfect sense, but for an API that allows an external consumer to access your catalog you may only want to expose it as a one-to-many relationship. This product has these accessories and not expose that the accessories belong to more than one product. You may not even want to model the reverse relationship from accessory to product in the API at all. It might simply be immaterial.

Also you may want to provide localization for product attributes so that your API can be language aware. While your domain model must account for the relationships to alternative text for attributes, you're likely to perform substitution for those attributes appropriate to the language requested rather than provide all translations, forcing the consumer to pick the correct one for their purpose.

Navigation

Navigation in your domain model is established through foreign key relationships in the database, which are translated to object references and collections of object references in your object model. Navigation in APIs is represented by URLs. "RESTful", "REST-like", "Hypermedia" - that's a discussion for a different post - but modern thinking is that your API on the web should leverage web protocols (the verbs and request patterns). This can be a challenge in designing your API as you trade off between model purity and efficiency.

In your object model the cost of navigation is very low (an object reference) but on the web it is exponentially higher - a full web request. The thing that's clear, though, unless your data set is very small, is that you will have to make compromises to keep your requests from ballooning in size. While you may be able to keep your entire product catalog in memory, it's not likely that you're going to want to serialize all of it and deliver it in a single web request. Instead, you're going to break your API up into multiple methods, each perhaps corresponding to a aggregate root. You may choose to expose navigation between related elements via explicit navigation properties or simply ids that the client uses in constucting the appropriate URLs to request more information.

If you embrace web protocols in your API, you'll need to account for the differences in navigation properties between your object model and your API.

Versioning

Another strong argument in favor of using separate view models is versioning. Because APIs represent a "public" product, tying disparate systems together, it's rarely possible to simply discard an older API when the underlying domain model is modified. In response API designers need to account for this by introducing some sort of versioning into their API design. If you directly expose your domain models, this could also mean that you would need to introduce versioning into your domain model to support older versions of an API. Decoupling your domain models from your API through the use of purpose-specific API models insulates your domain model from this need. While you may need to main multiple versions of your API models to support multiple versions of your API, your domain model can evolve to support new features without compromise for the sake of the API.

Summary

In short, you should use separate API models. Yes, it might seem simpler to directly use your domain models and there are a lot of examples that take this expedient. In the long run and with a more complex API than typically used for tutorials, separate API models are going to serve you well.

Unit testing JsonResult with Nested Anonymous Types

Recently I had a need to test some controller actions that returned nested anonymous types. Since the Data property on the JsonResult is of type object this can involve jumping through some hoops with reflection to get access to the properties in the anonymous types.  My first attempt was to simply serialize the object, then deserialize it to an anonymous type. This worked, but I found that my nested types were JObjects and I needed to use cast and convert these to the actual type.

public class TestContext
{
    // remainder omitted
    
    public dynamic ToDynamicObject(object obj, object anonymousTemplate)
    {
        var serialized = JsonConvert.SerializeObject(obj);
        return JsonConvert.DeserializeAnonymousType(serialized, anonymousTemplate);
    }
}

Used as

var model = _c.ToDynamicObject(result.Data, new { success = true, data = new SubModel() });

var subModel = ((JObject)model.data).ToObject<SubModel>();

I found one solution at http://blog.m0sa.net/2011/02/mvc-unit-testing-jsonresult-actions.html which seemed to improve upon this by using a wrapper class that implements DynamicObject. This got me very close to what I wanted except, again, the nested anonymous objects themselves had to be wrapped individually. Here using my initial implementation of an extension method that used it.

var model = result.Data.AsDynamic();

var cardModel = ((object)model.data).AsDynamic() as SubModel;

To fix this I added a bit of code to the wrapper class so that the result of TryGetMember was an anonymous type, it created and returned a DynamicObject wrapper around it. To test if the type was anonymous I used the code found at http://www.liensberger.it/web/blog/?p=191 (referenced from this Stack Overflow question, http://stackoverflow.com/questions/2483023/how-to-test-if-a-type-is-anonymous)

Here’s the final implementation as extensions to object.  Note: these are internal to my test project so the restriction to object doesn’t bother me too much. JsonResult.Data has that type and that was the problem I was trying to solve.

internal static class ObjectExtensions
{
    public class DynamicWrapper : DynamicObject
    {
        private readonly Type _subjectType;
        private readonly object _subject;

        public static dynamic Create(object subject)
        {
            return new DynamicWrapper(subject);
        }

        private DynamicWrapper(object subject)
        {
            _subject = subject;
            _subjectType = subject.GetType();
        }

        public override bool TryGetMember(GetMemberBinder binder, out object result)
        {
            try
            {
                var propertyInfo = _subjectType.GetProperty(binder.Name);

                var getter = propertyInfo.GetGetMethod();

                result = getter.Invoke(_subject, null);

                if (result.IsAnonymous())
                {
                    result = Create(result);
                }
                return true;
            }
            catch
            {
                result = null;
                return false;
            }
        }
    }

    public static bool IsAnonymous(this object obj)
    {
        if (obj == null)
        {
            return false;
        }

        var type = obj.GetType();

        return Attribute.IsDefined(type, typeof(CompilerGeneratedAttribute), false)
                  && type.IsGenericType && type.Name.Contains("AnonymousType")
                  && (type.Name.StartsWith("<>") || type.Name.StartsWith("VB$"))
                  && (type.Attributes & TypeAttributes.NotPublic) == TypeAttributes.NotPublic;
    }

    public static dynamic AsDynamic(this object obj)
    {
        return DynamicWrapper.Create(obj);
    }
}

And the above code sample now becomes much cleaner and I am much happier:

var model = result.Data.AsDynamic();

var subModel = model.data as SubModel;