Testing Your Unity Configuration when you Build or Deploy .Net Apps

Dependency injection is great, but it does come with a risk that errors with the container configuration don’t show up until runtime (eg. forgetting to specify which implementation should be used for a given interface). In an MVC app, this can mean broken controllers, fatal errors with service classes, and parts of your site/application being unavailable. So a safeguard against this is to check your IOC configuration, either at deployment time, or even better, at build time. Then you can fail the build or back out of a deployment if the various services or controllers can’t be resolved.

To solve this problem on a recent project, I wrote some custom code to do a “get me one of everything” test. So if I have a service requiring other services/repositories, or a controller that uses those services, I know about any missing Unity configuration entries. This post covers .Net, MVC and Unity, but the same approaches should work for other IOC setups.

I’ve listed a couple of different approaches to Unity configuration testing in the following code snippets – applying a simple strategy to a large production codebase threw up a few issues, and prompted some experimentation and refactoring.

Basic Approach

First, I try and resolve everything I can find registered in the Unity container. Then I check that I can get an instance of each controller. This should ensure that all the constructor-injection is tested. Any errors with Unity get collected up and reported (and hopefully other errors are left to a different test). If the checks return no errors we’re good, otherwise we need to go fix the configuration. There are limits to this approach – I’m still trying to figure out a way to deal with individual calls to directly resolve implementations from the container – but checking the listed container contents and all the controllers cuts down on a lot of potential problems.

Running a Unity Check

Once you have a basic Unity checker class setup, you have options for running it. I have an API controller run this, so I can poll it from a command-line app/script on deployment. Our build pipeline does a “deploy to a development environment and run some tests” following a successful CI build, so we know soon after a code commit whether we have a shippable build. Getting the tests running at the local/CI build stage (eg. via unit tests) is even better.

Example Code

The first thing I need is a simple app with Unity setup (your production codebase will no doubt require some tweaks), and some very basic/stupid test interfaces/implementations. And then I can add some Unity configuration. I just grabbed the Unity.Mvc Nuget package, used the out-of-the-box UnityConfig.cs, and edited it to set up a load of config entries eg.

container.RegisterType();

I then made sure I had service implementations and controllers constructed with dependency injection so they get whatever’s configured in Unity as the implementation, eg.

public StuffController(IDoStuffService doStuffService) { ... }

I start by building a basic Unity checking class like this:


public class UnityConfigurationChecker
{
    public static IList GetUnityErrors()
    {
        List errors = new List();
        var container = UnityConfig.GetConfiguredContainer();

        foreach (ContainerRegistration registration in container.Registrations)
        {
            try
            {
                Type registrationType = registration.RegisteredType;
                var instance = container.Resolve(registrationType);
            }
            catch (Exception e)
            {
                errors.Add(e.Message);
            }
        }

        //Note: going to add extra checks for controllers in here!

        return errors;
    }
}

I can then call it to get a list of (hopefully zero) errors with

var messages = UnityConfigurationChecker.GetUnityErrors();

This class isn’t intended to be hit by production code or user action, it’s just a helpful utility that tells us all the configuration errors we currently have. No messages is good, otherwise we need to investigate the issues (so I just catch all the exceptions and make a note of them).

Running Unity Tests

You could start with a basic controller/action that just runs the Unity checks and reports any errors to a view. Expand this with an API method and a script that runs during your build process to call the API and get a list of (hopefully zero) issues.

Now the fun begins as we start commenting-out Unity entries and watch those errors eg. “The current type, UnityChecker.Services.IDoStuffService, is an interface and cannot be constructed. Are you missing a type mapping?”

So far this only tests the registered container entries. It will only pick up issues with registered types that require other registered types.

Assume you have this setup of three services, only one of which requires constructor injection:

container.RegisterType();
container.RegisterType();
container.RegisterType();

public DoStuffService() { ... }
public DoThingsService() { ... }
public DoStuffAndThingsService(IDoStuffService doStuffService, IDoThingsService doThingsService) { ... }

If you forgot to register either the DoStuffService or DoThingsService, the DoStuffAndThingsService can’t be resolved.

Testing Unity With Unit Tests

So this gives you a Unity-checking utility that you can use on a running web application, but that means you have to wait until deployment time to check your config. Better than a fatal runtime exception days after deployment, but checking this during the CI process (or even before check-in) would be awesome. So let’s try building container-checking into a unit test.

Add a unit test project to the solution (I’m using MsTest, but the NUnit approach should be the same) and create a simple unit test class:


[TestClass]
public class UnityConfigurationCheckerTest
{
    [TestMethod]
    public void GetUnityErrors_ExpectZeroErrors()
    {
        var messages = UnityConfigurationChecker.GetUnityErrors();
        Assert.AreEqual(0, messages.Count);
    }
}

Verifying Controllers (Version 1)

So far, this example only checks the container registrations, not the controllers. We can amend our method with extra code that also checks the controllers:


    //Need reference to the main MVC web application assembly
    //May need to check for typeof Controller and ApiController
    //Note: Prevent false-positives of abstract controllers
    var assembly= typeof(UnityConfig).Assembly;
    var controllerTypes = assembly.GetTypes().Where(t => t.IsSubclassOf(typeof(Controller)));

    foreach (var controllerType in controllerTypes)
    {
        if (!controllerType.IsAbstract)
        {
            try
            {
                var controllerInstance = container.Resolve(controllerType);
            }
            catch (Exception e)
            {
                errors.Add(e.Message);
            }
        }
    }
    return errors;

Note that searching the correct assembly for controllers is vital. Amend as necessary if you want to use the Unity checker with multiple different web application assemblies.

If the controller constructor expects something that’s not registered in the container, we’ll now get an error about it. Actually, I did get an error regarding one of the controllers in the demo MVC solution I used as a starting point, so some care may need to be taken about what constitutes an actual error in your configuration…

Better Exception Checking

Treating all the exceptions you hit as “fatal” is unhelpful, we want to distinguish between container registration errors and other runtime issues (eg. a service fails because of some other database or configuration issue). Be mindful of the testing context you run the Unity checks in. This is especially important if you’re going to use container checking as a build-breaking condition.

An improvement is to catch “System.InvalidOperationException” and “Microsoft.Practices.Unity.ResolutionFailedException”, and then log the exception type and message. For controllers that throw ResolutionFailedException, if there’s an InnerException then use that instead.

As a test, you could make sure your container registers everything you need, then deliberately throw an exception in a service/controller constructor – differentiate between being unable to construct the controller (missing Unity configuration entry) and a subsequent runtime exception.

Deciding on the correct set of exceptions to catch / ignore is still a judgement call and might not catch every scenario of “IOC configuration vs other runtime error” problem you run into…

Verifying Controllers (Version 2)

At this point, it’s maybe worth re-thinking the controller checks. While we’re making sure we can get controller instances, we’re not too worried about whether we can construct an instance of a controller so much as whether the controller has access to everything it needs to be constructed. So what if, instead of exercising Unity by constructing an instance of the controller, we just check that everything injected into the controller’s constructor is available in Unity?

So my refactored UnityConfigurationChecker class now contains this:


public class UnityConfigurationChecker
{
    private static void ResolveImplementationFromIoc(IUnityContainer container, Type registrationType)
    {
        try
        {
            var instance = container.Resolve(registrationType);
        }
        catch (InvalidOperationException e)
        {
            throw e;
        }
        catch (ResolutionFailedException e)
        {
            throw e;
        }
        catch (Exception)
        {
            //Ignore
        }
    }

    public static IList GetUnityErrors()
    {
        List errors = new List();

        var container = UnityConfig.GetConfiguredContainer();


        foreach (ContainerRegistration registration in container.Registrations)
        {
            Type registrationType = registration.RegisteredType;
            try
            {
                ResolveImplementationFromIoc(container, registrationType);
            }
            catch (Exception e)
            {
                errors.Add(e.GetType() + " - " + e.Message);
            }
        }
        //Need reference to the main MVC web application assembly
        //May need to check for typeof Controller and ApiController
        var assembly = typeof(UnityConfig).Assembly;
        var controllerTypes = assembly.GetTypes().Where(t => t.IsSubclassOf(typeof(Controller)));

        foreach (var controllerType in controllerTypes)
        {
            if (!controllerType.IsAbstract)
            {
                var constructors = controllerType.GetConstructors();
                foreach (var constructor in constructors)
                {
                    System.Reflection.ParameterInfo[] parameters = constructor.GetParameters();
                    foreach (var parameter in parameters)
                    {
                        var parameterType = parameter.ParameterType;
                        if (parameterType.IsInterface)
                        {
                            try
                            {
                                ResolveImplementationFromIoc(container, parameterType);
                            }
                            catch (Exception e)
                            {
                                errors.Add("Registration error with controller " + controllerType.FullName + ". " + e.GetType() + " - " + e.Message);
                            }
                        }
                    }
                }
            }
        }

        return errors;
    }
}

And then a quick test – get the controller to deliberately fail eg. by throwing a new NotImplementedException in the constructor – and as long as the Unity configuration is fine, the Unity checks will pass, because you’re not actually constructing an instance of the controller. However, throw an exception in one of the concrete implementations you resolve from the container, and you will get an error.

Check again that this is all working by editing the UnityConfig.cs and hiding a few necessary registrations.

Gotchas and Other Issues

I encountered a few issues when applying this strategy to an existing codebase. The Unity configuration works fine in its natural web application context, but running in an unfamiliar unit testing context causes issues. Because you’re actually running the Unity registration and creating instances of those registered types during your tests, you may encounter errors in the setup before the test even runs and starts testing your newly-configured container.

Firstly, I had a few types that wouldn’t resolve themselves because they fell over at construction time looking for non-existent configuration settings. A few extra “appSettings” entries in the unit test project’s “app.config” file fixed that.

Next, I had our old friend the “Could not load file or assembly” exception. Because the unit test project is running the same code as your target application, it’s going to need to reference and load the same assemblies. Good luck debugging that one.

I was running the Unity configuration in application startup with a “UnityConfig.RegisterComponents()” call to run all my “RegisterType” calls. I also had the unit test run that as a setup step to mimic a clean application startup.

You can wrap a try-catch around your setup call, or just step-through in the debugger to find the problem registration, but it might take some effort to get your container to work correctly for your unit tests.

An Alternative Approach Using Reflection (Version 3)

There’s an alternate approach worth considering at this point. The aim of this exercise is not to actually resolve and construct any concrete instances (we won’t be using them). That’s just one way to test the container (it’s probably the best way). All we actually care about is one thing: if you have a service/controller with a concrete instance of another service injected at run-time, will it work? Or will the container say “I don’t know how to resolve that” and let the app fall over?

I already used reflection to inspect my controller constructor parameters, I can do the same trick for my service classes.

So here’s code for an alternative approach. I also took the opportunity to wrap any returned errors in a class that can easily show errors in the test runner just by overriding ToString().


public class UnityConfigurationChecker
{
    private static Type GetMappedRegisteredConcreteType(IUnityContainer container, Type registrationType)
    {
        if (container.IsRegistered(registrationType))
        {
            //Don't resolve, just ask the container what it might resolve to
            ContainerRegistration registration = container.Registrations.FirstOrDefault(r => r.RegisteredType == registrationType);
            if (registration != null)
            {
                return registration.MappedToType;
            }
        }
        throw new ArgumentException("Could not map to concrete type for " + registrationType);
    }

    private static bool InspectConcreteTypeEnsureConstructorParametersValid(IUnityContainer container, Type concreteType)
    {
        //Don't care about abstract controllers / services
        if (!concreteType.IsAbstract)
        {
            //Look for constructor parameters and resolve all injected interface implementations
            var constructors = concreteType.GetConstructors();
            foreach (var constructor in constructors)
            {
                System.Reflection.ParameterInfo[] parameters = constructor.GetParameters();
                foreach (var parameter in parameters)
                {
                    var parameterType = parameter.ParameterType;
                    if (parameterType.IsInterface)
                    {
                        try
                        {
                            GetMappedRegisteredConcreteType(container, parameterType);
                        }
                        catch (Exception e)
                        {
                            throw new ArgumentException("Registration error with type " + concreteType.FullName + ". " + e.GetType() + " - " + e.Message);
                        }
                    }
                }
            }
        }
        return true;
    }

    public static UnityConfigurationCheckSummary GetUnityErrorsContainerConfiguration()
    {
        UnityConfigurationCheckSummary summary = new UnityConfigurationCheckSummary();
        var container = UnityConfig.GetConfiguredContainer();

        foreach (ContainerRegistration registration in container.Registrations)
        {
            Type registrationType = registration.RegisteredType;
            var concreteType = registration.MappedToType;
            try
            {
                InspectConcreteTypeEnsureConstructorParametersValid(container, concreteType);
            }
            catch (Exception e)
            {
                summary.Errors.Add(e.Message);
            }
        }
        return summary;
    }

    public static UnityConfigurationCheckSummary GetUnityErrorsControllerConfiguration()
    {
        UnityConfigurationCheckSummary summary = new UnityConfigurationCheckSummary();
        var container = UnityConfig.GetConfiguredContainer();

        var assembly = typeof(UnityConfig).Assembly;
        var controllerTypes = assembly.GetTypes().Where(t => t.IsSubclassOf(typeof(Controller)));

        foreach (var controllerType in controllerTypes)
        {
            try
            {
                InspectConcreteTypeEnsureConstructorParametersValid(container, controllerType);
            }
            catch (Exception e)
            {
                summary.Errors.Add(e.Message);
            }
        }
        return summary;
    }
}


public class UnityConfigurationCheckSummary
{
    public IList Errors { get; set; }
    public int Count { get { return Errors.Count(); } }

    public UnityConfigurationCheckSummary()
    {
        Errors = new List();
    }

    public override string ToString()
    {
        if (Errors.Any())
        {
            return string.Join(", ", Errors);
        }
        return string.Empty;
    }
}

Here’s the unit test code, checking that we get no errors, but giving a nice summary if we do. You might need to add a “Setup” method to reset/recreate your container for each test.


[TestClass]
public class UnityConfigurationCheckerTest
{
    [TestMethod]
    public void GetUnityErrorsContainerConfiguration_ExpectZeroErrors()
    {
        var summary = UnityConfigurationChecker.GetUnityErrorsContainerConfiguration();
        Assert.AreEqual(string.Empty, summary.ToString());
        Assert.AreEqual(0, summary.Count);
    }

    [TestMethod]
    public void GetUnityErrorsControllerConfiguration_ExpectZeroErrors()
    {
        var summary = UnityConfigurationChecker.GetUnityErrorsControllerConfiguration();
        Assert.AreEqual(string.Empty, summary.ToString());
        Assert.AreEqual(0, summary.Count);
    }
}

Summary – Verifying Unity Configuration

With some simple code like this, every time you add a service or change a controller, your Unity configuration should be tested for correctness with a regular unit test run. And you can always put the tests behind an API method and poll after a deployment to be sure that your configuration is correct.

Versioning .Net Applications With Team City and Git/SVN/TFS

Some notes on setting up version numbering for .Net applications and having your build server manage everything (I’m using Team City). I’ll cover some differences between centralised (TFS, SVN) and decentralised (Git) version control. These notes are based on a couple of “single-branch, build-once-and-redeploy” continuous delivery pipelines I set up for projects in TFS and Git.

Why Version?

We’ve all seen software shipped from developer machines, hoped we shipped the code we actually tested, and had to troubleshoot live issues where all we know about the code is it’s “probably whatever Bob had on his laptop three weeks ago”. Want the easy life? Bake in a version number that:

  1. Confirms the code was built and released through an approved CI/CD process – the presence of a default “1.0.0.0” version indicates worrying “laptop” deployments *
  2. Confirms the production code matches the tested code (and hasn’t had any bonus unpredictability merged in)
  3. Tells you where to find the exact code revision in source control
  4. Tells you which build job created it
  5. Gives you a changing version number to display to users that says “new stuff”

Display that version somewhere in your code and have it available on your build server.

* OK, you could cheat and fake a version number, but the aim is to make things so easy that doing things properly is the only option you consider.

What Version Number Format Should I Use?

This one is up to you. .Net projects allow a basic four-segment numbering system.

A recommended format is: “<major version>.<minor version>.<build number>.<revision>”.

Major/minor components are typically your “marketing version”, but you might want to use them to signal build time (eg. calendar date or sprint number). They probably won’t change often, and updates may be manually controlled. You definitely want to have build and revision/changeset numbers automatically applied so you can trace where the code came from. If your source control doesn’t use sequential version numbers (eg. Git) then you need to make a few adjustments, typically by using the revision checksum/snapshot instead of a sequential code revision.

Displaying the Version Number In Your Application

Your application should display the version number somewhere – on a help/about dialog, footer, via an “info” API call, etc.

Within your code, each project has an “AssemblyInfo.cs” file under “properties”. This can contain versioning attributes for:

  • AssemblyVersion
  • AssemblyFileVersion
  • AssemblyInformationalVersion

Version and file-version components consist of four numbers (you might be able to squeeze letters into the file version, but you’ll get a warning). The informational version allows more freedom, so you could hide a checksum, like a Git revision (yes, I really did ship an application with version number “1.3.29.300bcd5309dacf3897fc41ba11dff56409b136db”).

These numbers are baked into the compiled code (inspect the properties of a DLL file).

Your default AssemblyInfo.cs entries look like this:

[assembly: AssemblyVersion(“1.0.0.0”)]
[assembly: AssemblyFileVersion(“1.0.0.0”)]

You can manually edit these and rebuild, but much better to get your build process to apply them automatically. To display, just use:

System.Reflection.Assembly.GetExecutingAssembly().GetName().Version.ToString()

For the AssemblyInformationalVersion, use

Attribute.GetCustomAttributes(Assembly.GetExecutingAssembly(), 
typeof(AssemblyInformationalVersionAttribute)) as AssemblyInformationalVersionAttribute[]

And then display the first item in the array.

If you’re using Git, the AssemblyInformationalVersion is the place to store your revision.

Build Pipelines for Continuous Delivery / Continuous Deployment

I’m assuming you have some sort of pipeline whereby code passes through an initial CI (build on code change) stage to Test (deployment) to Live (deployment). You can control the flow of this pipeline in a couple of different ways:

  • Single branch / “trunk-based” deployment – everyone commits from local workspace (or their own development branches) to a single main branch, and you build once (at the Continuous Integration stage) then redeploy the same package/artifact at subsequent Test/Live stages
  • Branch-based – you merge code from Dev to Test to Live branches, and rebuild at each stage before deploying

You can use either approach, but always “ship what you test”. Because centralised version control systems (eg. SVN and TFS) assign a new sequential revision number when you merge code to a different branch, it’s difficult to track code changes across branches, so I wouldn’t recommend using a branch-based approach for SVN or TFS unless your process involves full continuous deployment at every step and you trust to full automated test coverage. With Git’s decentralised model, you have a checksum built into to the revision, so you can tell whether the merge introduced any changes, and a branch-based continuous delivery approach becomes viable.

 

Versioning in Team City

Each build in Team City is assigned a build number. You edit this in the configuration’s General Settings panel, just enter a value for “Build number format”. Typically this is built from a number of properties, the default being the “%build.counter%” (Team City’s internal count of builds for that configuration, also editable on the General Settings panel).

While the build counter is managed by Team City and the revision tracks back to source control, other build numbers can be applied. So to manage “marketing” versions for your product:

  1. Edit Project->Parameters
  2. Create two “configuration” parameters – “Project.Major.version” and “Project.Minor.version”

You could set up parameters for an individual build if you want, but project-wide parameters make sense. These values are now available to use in components of other build numbers as “%Project.Major.version%” and “%Project.Minor.version%”.

Some clarification – there’s a few different build numbers you’ll have to manage here:

  • The Team City build counter – increments each time Team City runs a specific build configuration job eg. “build code for project X” (this is a single number)
  • The Team City build number – applied to each build run, displayed on Team City dashboard and build reports (this might be a four-digit version number)
  • The version number(s) baked into the code at build time

Team City makes setting version numbers really easy, using the “Assembly Info Patcher” build feature – use this for every build configuration that is actually building code (CI, or any branch-based “re-build and re-deploy” stage). Just edit a build configuration and go to Build Features->Assembly Info Patcher. Set the required “Assembly version”, “Assembly file version” and “Assembly informational version” values to edit the
corresponding attributes in the AssemblyInfo.cs files.

Build Number Format (TFS)

For the initial CI build configuration use a build number format of

%Project.Major.version%.%Project.Minor.Version%.%build.vcs.number%.%build.counter%

I just set this as the version and file version in the assembly info patcher dialog, and ignore the informational version.

If you’re taking the single-branch/build-once-and-redeploy approach, then for a chained/dependent build configuration (eg. Deployments to Test/Live) set the build number of subsequent builds in the chain to be

%dep.CIProjectName.build.number%

You can use this approach to assign the same build number to the same code built/deployed by different build configurations – I do this to track a release candidate from the initial CI stage (build and package) through the various Test and Live deployment pipeline stages.

Build Number Format (Git)

For Git, I set the build number format to

%Project.Major.version%.%Project.Minor.Version%.%build.counter%

This means I only get 3-digit version numbers displayed on Team City, and no direct reference to the code.

I then set the assembly info patch values to be (on Build Features->Assembly Info Patcher)

Version and file version:

%Project.Major.version%.%Project.Minor.Version%.%build.counter%.0

Informational version:

%Project.Major.version%.%Project.Minor.Version%.%build.counter%.%build.vcs.number%

Passing Team City Version to Build Scripts

A quick note: Team City sets the current build number (the full 4-digit one) in a “BUILD_NUMBER” environment variable. So you could have an MsBuild script in your project use this if you need it (eg. to name files, or append to release notes and emails).

Example MsBuild usage:

<Message Text="Build number=$(BUILD_NUMBER)"/>

 

Summary

So you can use your build server to generate meaningful version numbers, track the same code as it works through the various chained builds of your build pipeline from initial commit to production deployment, and bake the appropriate version into your code. With a version number available, you should have confidence that the production release is known and tested software, and not some mystery release you can’t recreate locally.

 

Choosing What Gets Deployed When Publishing A .Net Web Application

When you use MsBuild to package up a web application for publishing, you typically rely on the publishing profile, to manage either a “build and publish” or “build a package for later deployment” process (a .zip package is created either way). By default, everything in your project may be copied to the output and included in the deployment package .zip file, unless you make adjustments.

You already have some control over whether files get copied into the output and packaged up, using file properties in the solution explorer, as set by the “Build Action” (set to “none” rather than “content”) and “copy to output directory” properties. However, it may be easier to just use some MsBuild to avoid packaging several files (or whole folders) based on a wildcard pattern.

Example MsBuild syntax for generating the deployment package would be:

msbuild WebApp\WebApp.csproj /t:Build /p:VisualStudioVersion=11.0;DeployOnBuild=True;PublishProfile="DeploymentPackage.pubxml";ProfileTransformWebConfigEnabled=False;Configuration=Release;

 

Including Extra Files In The Package

I previously covered ways to include extra files that your project doesn’t know about:

https://ajmburns.wordpress.com/2016/04/10/including-additional-generated-files-when-publishing-a-net-web-application/

Removing Files From The Package

If you wanted your deployment to omit selected files or folders, you can get the publish profile to that as well. I have to do this to hide deployment parameter config files and selected custom scripts that are part of my solution, but should not be deployed.

To test, add the excludes to the main “PropertyGroup” element of your publish profile, then build the package and check the generated package .zip file.

Example – excluding a list of custom scripts:

<ExcludeFilesFromDeployment>
 compile-this.bat;compile-that.bat
</ExcludeFilesFromDeployment>

Example – excluding a wildcard list of configs:

<ExcludeFilesFromDeployment>
 parameters*.xml
</ExcludeFilesFromDeployment>

Example – excluding a folder:

<ExcludeFoldersFromDeployment>
 myConfigs
</ExcludeFoldersFromDeployment>

Including Additional / Generated Files When Publishing a .Net Web Application

When you publish a .Net web app, you typically set up a publish profile, either to do an immediate build-and-publish, or to publish to a package for later deployment (eg. single-trunk / build-once-deploy-many-times scenarios).

So for building a package, the MsBuild syntax would look like:

msbuild WebApp\WebApp.csproj /t:Build /p:VisualStudioVersion=11.0;DeployOnBuild=True;PublishProfile="DeploymentPackage.pubxml";ProfileTransformWebConfigEnabled=False;Configuration=Release;

You just use MsBuild and ask your project to build itself. The good bit is that your project knows what it contains, and therefore what needs deployed. The bad bit is that your project knows what it contains, and therefore what needs deployed… Any files generated by an external tool, or any other files you didn’t explicitly tell Visual Studio (listed as content in the .csproj file) are excluded. Thankfully, you can include extra stuff into your deployment package with a few lines of MsBuild commands. While you can probably hack the .csproj, the easiest way I’ve found is just to plug some MsBuild directly into the publish profile.

This location does make sense – you’re specifying what gets published after all. Note that if you have several publish profiles (maybe you have branch-based deployment and per-environment build-and-publish) then things get more complicated. I’m going to stick with the simple case – a single publish profile for creating a generic deployment package that we can then deploy (with appropriate configuration) to any target environment.

I recently had to solve this problem for handling some generated CSS, but you can use it for any non-solution files you want to include as part of the web application deployment package zip.

A quick warning – this does involve editing a generated file (the .pubxml file that you probably only create once).

Tweaking The Publish Profile

So assume I have some tools generating CSS (in “/webproj/content/styles”) and extra text files (in “/webproj/content/documentation”). I don’t even need to set up tools to generate these files – I can actually just simulate by manually adding some dummy files. I’m assuming that the files get generated within the project directory (because you want to deploy them to IIS or Azure when you deploy your site).

Anyway, create the extra files within the project directory, don’t tell Visual Studio or your project about them, and then build the deployment package. You’ll see they get omitted. We can fix that.

Go edit your publish profile, and add these lines inside the “PropertyGroup” element – we need to add a “CopyAllFilesToSingleFolderForMsdeployDependsOn” property.

 <CopyAllFilesToSingleFolderForMsdeployDependsOn>
 IncludeCustomFilesInPackage;
 $(CopyAllFilesToSingleFolderForMsdeployDependsOn);
 </CopyAllFilesToSingleFolderForMsdeployDependsOn>
 </PropertyGroup>

This goes off and calls a target called “IncludeCustomFilesInPackage” that we’ll create in a minute. The name of the “CopyAllFilesToSingleFolderForMsdeployDependsOn” property is important (MsBuild will look for it), the name of the custom “IncludeCustomFilesInPackage” target can be whatever we want (change the names and see what happens).

There is also a “CopyAllFilesToSingleFolderForPackageDependsOn” property, but this seems to get ignored even when building in package mode. For completeness, it would look like this:

<CopyAllFilesToSingleFolderForPackageDependsOn>
 IncludeCustomFilesInPackage;
 $(CopyAllFilesToSingleFolderForPackageDependsOn);
</CopyAllFilesToSingleFolderForPackageDependsOn>

But you don’t seem to need it – just add the “CopyAllFilesToSingleFolderForMsdeployDependsOn”.

Now we can go define our “IncludeCustomFilesInPackage” target, so add this to the .pubxml file, inside the “Project” element:

<Target Name="IncludeCustomFilesInPackage">
 <Message Text="Collecting custom files..."/>
 <ItemGroup>
  <CustomFiles Include="Content\styles\**\*" />
  <CustomFiles Include="Content\documentation\**\*" />
  <FilesForPackagingFromProject Include="%(CustomFiles.Identity)">
   <DestinationRelativePath>%(CustomFiles.Identity)</DestinationRelativePath>
  </FilesForPackagingFromProject>
 </ItemGroup>
 <Message Text="Add to package %(FilesForPackagingFromProject.Identity)"/>
</Target>

Add as many entries as you need into the ItemGroup. I added a couple of “message” calls to output progress.

So when you run your package build and inspect the final .zip package that gets created (check in the “obj” directory eg. “projectName/obj/projectName/projectName.zip”) then it should contain all those extra files that your .csproj didn’t know about.

Troubleshooting

When you’re setting this up, you might run into problems getting the paths correct. You can always add in a load of Message calls, and redirect the output of your MsBuild run to a text file.

You can add the following in your custom target for debugging purposes:

<ItemGroup>
 <GeneratedIncludeFiles Include="Content\documentation\**\*" />
</ItemGroup>
<Message Text="Generated files to include: %(GeneratedIncludeFiles.Identity)"/>

A note about “DestinationRelativePath” – I have seen this specified as “%(RecursiveDir)%(Filename)%(Extension)” instead of using the “CustomFiles” item group, but I had trouble getting this to actually include the custom files.

.Net Web App Configuration with Parameters – Moving beyond web.config files

For any web application, you’ll typically want to configure your application for the different local, Dev, Test and Production environments. And you want all your building and deployment to be handled by your build server, which means anything you do needs to run using MsBuild, Powershell, command-line .exes, or whatever nice wrappers your build server provides.

Web.Config files

When you first start building .Net web applications, you probably use web.configs. And then when you need more than the out-of-the-box “debug” and “release” configurations, you resort to adding more build configurations and web.config transforms. And these are fine when you’re selecting a configuration to build and run locally, or publishing from Visual Studio, but this doesn’t scale beyond a quick experimental project or hackathon to an actual production site.

So if you want “test” and “production” configs, the temptation is to use web.config files and transforms, where local dev uses the “debug” configuration, and everything else uses an optimised “release” configuration. Actually adding another configuration involves first adding a build configuration, then adding the new web.config transforms. If you’ve ever done web publishing through Visual Studio, you’ll probably have spent a lot of time setting up build configurations for each environment (that should be a warning), then creating the web.config transforms so you can use them when publishing.

However, web.configs have a couple of issues:

  1. They hold different configurations for different purposes – your application “build” configuration for debug/release configurations, and your environment “deployment” configuration (local, test, production).
  2. Web.configs are designed to be used in a “configure, build, deploy” process, and for a “build once, deploy often” continuous delivery pipeline, we want to “build, deploy, configure”.

Whether your branching-and-deployment strategy is “single trunk” (build package once, deploy to each environment) or “branch-based” (rebuild from a known code snapshot/revision and configure before each deployment), it pays to start thinking of “build” and “deployment” configurations as separate.

You can cheat (I did at first) and run all your web.config transforms at build-time, bundle them all up into your release package, and ship the correct config at deployment time (by configuring your deployment, or by deploying a generic package followed by a specific config file). My original web.config approach is covered in this epic post.

Web.config files and the Continuous Deployment Pipeline

It makes sense to have developers configure your core application “build” configuration at build time, but you don’t necessarily want to restrict the “business preference” or environment-specific configurations to requiring a complete application rebuild as you make a code change to inject those changes into the start of the continuous deployment pipeline. Having configuration settings that can be changed outside of the commit, build, deploy process is really useful – you should still manage these configuration values in source control or other change management system.

Using SetParameter.xml Files

So instead of putting our environment config values in web.config files, we’re going to put them in a separate file – the SetParameters.xml file. This makes things cleaner, and is also necessary when using certain deployment tools that only work on whole packages not individual files (eg. Azure Powershell cmdlets).

We need a web app that we can build into a deployment package, and somewhere we can deploy to. I’m using a new Azure web app, but parameters files also work with IIS.

Creating the Deployment Package

First we need an application that can be built into a deployable web package – skip this section if you already have a project you want to deploy.

As a demo, I set up a new basic MVC web app (just add a single HomeController and Index view). I also create appSettings entries for “EnvironmentName” and “EnvironmentDescription”, so I have some values to set. I can read these from the ConfigurationManager (eg. ConfigurationManager.AppSettings[“EnvironmentName”]) and display on my view. The plan is that I have different parameter files for my dev, test and live environments, and when I deploy with the appropriate configuration, an on-screen message shows that the appropriate configuration has been applied. Web.config transform files already exist for the debug and release build configurations (we could add more), but we’re not going to use these for configuring our environments.

Now I’m going to publish my site to a deployment package, so I can later have a two-stage build process:

  1. Build into a deployment (zip) package
  2. Deploy to dev/test/live environment and apply appropriate configuration

So in Visual Studio, go to build->publish, select profile type “custom” and name it “DeploymentPackage”. Connection method is “Web Deploy Package”, select a location (eg. “obj/appName”) and site name. . Go to solution explorer, look in yourAppName/Properties/PublishProfiles and see the newly-created publishing profile
(you can now edit the publish profile and change these values if you want).

Build the application with

msbuild appName.csproj /t:Build /p:DeployOnBuild=True;PublishProfile=DeploymentPackage;ProfileTransformWebConfigEnabled=False;Configuration=Release

Or just run the publish from Visual Studio.

This builds your site (in the “Release” configuration), creates your deployment package zip and a couple of extra files – a manifest, .cmd, readm, and a SetParameters.xml file. The parameters file is really basic:

<?xml version="1.0" encoding="utf-8"?>
<parameters>
 <setParameter name="IIS Web Application Name" value="MyWebAppName" />
</parameters>

But we can expand this with our own settings.

Deploying the Web App

OK, so I’m going to deploy this package to an Azure site.

Using Powershell, the syntax is:

Publish-AzureWebsiteProject -Name appname -Package myapp.zip -SetParametersFile appname.SetParameters.xml

For MsDeploy, the syntax looks like this:

msdeploy.exe -source:package="appname.zip"
-dest:auto,ComputerName="https://appname.scm.azurewebsites.net/msdeploy.axd?site=appname",UserName="$user",Password="extractFromPublishProfile",IncludeAcls="False",AuthType="Basic" 
-verb:sync -setParam:name="IIS Web Application Name",value="appname" -disableLink:AppPoolExtension 
-disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose 
-userAgent="VS12.0:PublishDialog:WTE12.3.51016.0" -setParamFile:"appname.SetParameters.xml"

When I run this, I find that this hasn’t actually done anything to override the settings with the values in the SetParameters file, because I didn’t set the parameter value. Also, I have to tell our app which parameters actually exist.

So go back to Visual Studio and add a “parameters.xml” file to the root of your web project. It should look like this:

<?xml version="1.0" encoding="utf-8" ?>
<parameters>
 <parameter name="EnvironmentName"
 description="blah blah blah"
 defaultValue="notsure">
 <parameterEntry kind="XmlFile" scope="Web.config"
 match="/configuration/appSettings/add[@key='EnvironmentName']/@value" />
 </parameter>
</parameters>

This creates a named parameter with a default value, and specifies how this should affect the web.config at deployment time. Rebuild and re-package the application, and the generated “SetParameters.xml” file now has a “setParameter” entry for “EnvironmentName”. Awesome. You can go edit that file, change the value, and run
MsDeploy / Publish-AzureWebsiteProject again to deploy the changes (and only the changes get shipped). This is basically doing a web.config transform at deployment time. And of course, you can expand your parameters.xml file for more settings. Update parameters.xml, rebuild the package, and you’ll see new entries in SetParameters.xml.

If you only want to change a single setting, you can do that with parameters added to your MsDeploy call. In fact, you could have a load of settings in a parameters.xml file, and then apply those as a base while overriding selected settings at deploy time. This might prove useful if you have to deploy multiple copies of your app to production, yet tag each one differently (eg. for A/B testing, “developer” API deployments, etc). Anyway, just add this to the end of your MsDeploy call:

-setParam:SettingName=value

This is obviously a really simple example, just changing appSettings. You can also use parameterization to set connection strings and other configuration settings. This post details the parameters syntax: http://vishaljoshi.blogspot.co.uk/2010/07/web-deploy-parameterization-in-action.html

Configuring for Different Environments

So far, all we have is a single appname.SetParameters.xml file, and it gets generated at build time, so we don’t want to be editing that. What we actually need is a SetParameters.xml file for each environment, and we need to make sure their values survive rebuilding the project. So you can either create additional SetParameters.xml files within your project and include them at build time (basically the equivalent of web.config transforms), or hold your SetParameters.xml files somewhere else, and just consume them at deploy time.

Note that if you do just include the parameters.xml files in your code, it might seem like you’ve gained nothing over hacky web.config transforms – you’re still configuring at commit time and shipping all the possible configurations to use at deployment time. However, you have de-coupled the code configuation (debug/release build config) from the environment configuration, and removed the need to have all those near-duplicate build configurations cluttering up your solution just to enable web.config transforms.

Summary

So now we have a scripted process to

  1. Build our application into a deployment package, including a specification for how the application can be configured
  2. Deploy our application to our chosen environment, applying the necessary configuration at deployment time

This allows you to implement a continuous delivery pipeline with whatever build/deployment server you use, and have “build once, deploy repeatedly” behaviour at each dev/test/live stage of your pipeline.

Gotcha: Don’t Deploy All The Parameters!

You’ll want to have the various parameters available at deployment time – read on for notes on including them with your build artifact so you have a “deployable zip package plus a set of parameter config files” – but you don’t actually want them included inside the “site.zip” package, because then they’ll end up getting deployed, and you don’t want people being able to go to http://www.yourawesomesite.com/parameters.test.xml and seeing all your secrets.

So the answer is to omit them from the build, either using the file properties, to set “Build Action” to be “none” rather than “content”, and “Copy to output directory” to “do not copy” on individual files, or by using some scripting (eg. MsBuild within your publish profile) to decide what ends up in the deployment package.

It’s best not to rely on humans setting the build properties and ensure that your build process omits the parameters (or have a CI/post-build check inspecting the deployment package for contraband).

You can add the following entry to your “DeploymentPackage.pubxml” publish profile to omit them:

<ExcludeFilesFromDeployment>
 parameters*.xml
</ExcludeFilesFromDeployment>

This is used by MsBuild when generating the package:

msbuild WebApp.csproj /p:DeployOnBuild=True;PublishProfile=DeploymentPackage.pubxml

When the deployment package is built this should exclude the matching parameter files.

Summary – what you should have is:

  1. Access to the appropriate “parameters-environment.xml” file at deployment time (either a release artifact that contains your site.zip and parameter files, or a separate repository of configuration files)
  2. The “parameters.xml” template in your “site.zip” deployment package
  3. Absolutely no “parameters-env.xml” files in the actual “site.zip” deployment package (expand the zip archive and navigate down to the “PackageTmp” folder to verify)

Bonus: Shipping Environment Configurations with the Deployment Package

One approach to including SetParameters files within the codebase might look like this.

Add a set of “parameters.dev.xml”, “parameters.test.xml”, “parameters.live.xml” files to your solution, clone them from the generated SetParameters.xml file eg:

<?xml version="1.0" encoding="utf-8"?>
<parameters>
 <setParameter name="IIS Web Application Name" value="MyAppName" />
 <setParameter name="EnvironmentName" value="Test" />
 <setParameter name="EnvironmentDescription" value="Testing" />
</parameters>

Now I need to handle building the package and copying of all the necessary release files into a “BuildArtifacts” folder. This could be achieved in different ways:

  • Set the parameters.*.xml properties to “copy to output directory=copy always”
  • Use an MsBuild or other scripted step to copy them into your final “artifacts” folder

For the MsBuild approach, I have a “WebPackage.msbuild” file in my solution with the following fragment:

<Target Name="BuildPackage">
 <Exec Command='$(MSBuildExe) $(WebApp)\$(WebApp).csproj /t:Build /p:VisualStudioVersion=11.0 
/p:DeployOnBuild=True;PublishProfile="$(PublishProfilePackage)";ProfileTransformWebConfigEnabled=False;Configuration=$(Configuration)'/>
 <CallTarget Targets="_CopyPackageToDeploymentDirectory"/>
 <CallTarget Targets="_CopyParametersToDeploymentDirectory"/>
</Target>

<Target Name="_CopyPackageToDeploymentDirectory">
 <ItemGroup>
 <PackageFiles Include="$(PackageAssemblyDir)\**"/>
 </ItemGroup>
 <Copy SourceFiles="@(PackageFiles)" DestinationFolder="$(BuildArtifactsDir)\$(WebApp)\%(RecursiveDir)"></Copy>
</Target>

<Target Name="_CopyParametersToDeploymentDirectory">
 <ItemGroup>
 <ParameterFiles Include="$(WebApp)\parameters.*.xml"/>
 </ItemGroup>
 <Copy SourceFiles="@(ParameterFiles)" DestinationFolder="$(BuildArtifactsDir)\$(WebApp)"></Copy>
</Target>

This is part of a “WebPackage.msbuild” script within my solution, as long as I have all the properties defined (in a PropertyGroup within the MsBuild script) I can just call

msbuild WebPackage.msbuild /t:BuildPackage

to generate a nice “BuildArtifacts” folder with everything I need to deploy my app.

Scripting Azure Activity and Authenticating with Publish Profiles

If you want to do anything with Azure, the tools Microsoft provide for humans – Visual Studio and the Azure Portal – are great. Unfortunately, if you want to automate any of your Azure activity – for example, deploying applications – then things get a bit tricky. Some Azure commands are available as simple command-line tools (eg. MsDeploy), but the preferred (and sometimes only) way is to use Powershell and the Azure cmdlets. And
with Powershell comes a few tricky authentication issues.

A warning: what I’m about to demonstrate is an approach, possibly not the best approach. I’m still trying to figure this stuff out, and this may not be the most secure method of doing things.

So the problem is this: My real-world deployment pipeline currently has automated Azure deployments via msdeploy.exe (all sorts of hassles and risks with the authentication credentials and the scripts are ugly, but it works). But as we want to use deployment slots for “zero-downtime deployments” (ship to staging slot, instantly switch slots to live), we need a smarter mechanism than manually swapping slots via the Portal. Swapping deployment slots is a one-liner in Powershell (“Set-AzureWebsite -Slot”), but as a proof of concept, I’m just going to get my script to connect to my Azure account and bring back some info. Then we can do some real damage and actually deploy code…

Setting Up Azure Powershell

So you’ll need to install Powershell and the Azure cmdlets. I had to upgrade to Powershell 4 to get this to work (also, I needed to re-install Azure Powershell after upgrading Powershell).

Fire up Powershell (you may also have an “Azure Powershell” shortcut available).

Check your Powershell version with this command:

$PSVersionTable.PSVersion

Get Powershell from http://social.technet.microsoft.com/wiki/contents/articles/21016.how-to-install-windows-powershell-4-0.aspx

Get the web Platform Installer and use it to install Azure Powershell –
https://www.microsoft.com/web/downloads/platform.aspx

You can run a test Azure command, for example:

Get-AzureSubscription

Doesn’t matter if you get an error about your credentials, at this point only worry if you’re getting instant errors about unrecognised commands because you haven’t installed everything.

The Manual Process

So you fire up an Azure Powershell console session, and start doing stuff. And of course, you have to supply authentication credentials. So do

Add-AzureAccount

and log in to your Microsoft account at the prompt. Once logged in, you can run some commands. To see your details:

Get-AzureSubscription

To see all your webapps:

Get-AzureWebsite

To “log out” – either:

Remove-AzureAccount -Name yourname@somewhere.com

Or

Remove-AzureSubscription

So all of this is great if you’re a human, not so good if you’re a robot, because “login prompts”. We need a better way to authenticate.

Publish profiles

If you’ve used MsDeploy to ship to Azure through either Visual Studio or the msdeploy.exe, you’ll have used publish profiles. These are per-application files that Azure provide containing authentication details (go to the portal and download the publish profile for each web app). However, you can’t really use these with Azure cmdlets, they’re “per-application”, which is an issue if you want to do something like swap between staging and live slots (actually, two apps with two publish profiles).

Thankfully, there’s another “account” publish profile you can use. Go to your Powershell console, make sure you’re logged in, and do:

Get-AzurePublishSettingsFile

There’s probably also a link to download this file on the portal somewhere, but I’ve yet to see it. Anyway, you should get a simple XML file called “SubscriptionName-date-credentials.publishsettings”. Save this somewhere safe.

Now you can do

Import-AzurePublishSettingsFile [.publishsettings file path]

And this imports the certificate, essentially “logging you in” as the trusted deployment user specified in the publish settings. Go run some “Get-AzureSubscription” and “Get-AzureWebsite” calls.

Scripting Azure Activity

You just did Azure stuff without having to deal with login prompts. Great. Now you can script all that up.

Create a simple script (text file) called “Connect.ps1” that contains this:

param([string]$publishSettingsPath)

#Connect to account
Import-AzurePublishSettingsFile -PublishSettingsFile $publishSettingsPath

#Show account status
echo "run Get-AzureSubscription"
Get-AzureSubscription
echo "run Get-AzureAccount"
Get-AzureAccount

#List all websites
echo "run Get-AzureWebsite"
Get-AzureWebsite

You can run this from a standard command prompt, with this call:

powershell -File Connect.ps1 C:\pathTo\myAzure.publishsettings

(The script takes a publish settings file path as an argument).

Adding Azure Powershell to an Automated Build Process

If you can script your Azure activity – and we just did – then you can add that to a build process. As a quick test, I dumped that Connect.ps1 script into an existing project and created a simple Team City build config to run it, so whenever I commit project changes to Git, I can have Team City connect to Azure.

All I had to do was create a Team City build step with the following configuration.

Runner type: Powershell
Powershell run mode: version 4
Script: file (with path to .ps1 relative to build agent checkout/working directory)
Script execution mode: external file
Script arguments: C:\pathTo\myAzure.publishsettings

Run this build and go look in the build logs, and you should see the script output, same as when you ran it from a command prompt.

For reference – Team City’s PowerShell arguments are [-Version, 4.0, -NonInteractive, -ExecutionPolicy, ByPass,
-File, \TeamCity\buildAgent\work\etc\PowershellScripts\Connect.ps1, \AzurePublishProfiles\my-
credentials.publishsettings]

Summary

So we now have scripts accessing our Azure account and running commands without hitting login prompts. This allows for automated deployments to Azure, including all the activities that were previously only available when using the portal. However, there’s a few security issues that you’ll likely want to address.

Additional Notes

Security Issues and “logging out”

The recommended advice seems to be to remove all accounts before importing the publish settings file, to prevent confusion over which Azure account is used by the Powershell session. Given that you’re logging in with a text file instead of a password, it’s good security practice to delete the publish settings file once you’ve set up access. However, Azure sessions have a tendency to time-out, so you probably can’t just do a one-time download of this file to set up a build machine then delete it – you’re going to need access to this file whenever you want to run an automated deployment (hence my adding the import command to the powershell script). I’m still working on this one – I suspect that Azure Resource Manager is ultimately the way to go.

Outstanding security issues:

  1. Where to keep the publish settings file? For security, source control would be a bad move, as would a network share. Probably storing a copy on the (locked down) build machine is best, but for a distributed build setup (eg. Team City with multiple agents) your build configuration needs to look in a consistent place on each agent. Getting around importing a publish settings file would be the ultimate aim.
  2. How to “log out” at the end of the session? Remove-AzureAccount seems to want an account name you don’t have, but it can apparently be “smart” about choosing the correct account to remove. Remove-AzureSubscription would work. Also, these commands don’t “remove” anything from your Azure account, they just kill the cached authentication credentials for your Powershell session.
  3. Can I rename the publish settings file? Yes. No complaints when I did just that.
  4. I’m paranoid. How do I revoke and reset my publishing credentials? Um, TBC. You might be able to use the “Reset deployment credentials” option in the portal.

Links for more info

I borrowed various bits of info from various sources.

Microsoft’s Azure cmdlet reference: https://msdn.microsoft.com/library/azure/jj554330.aspx

Resetting deployment credentials (in any webapp “publish your app” section): https://blogs.msdn.microsoft.com/kaushal/2013/05/29/windows-azure-web-sites-reset-your-deployment-credentials-vs-reset-your-publish-profile-credentials/

Cached credentials and removing accounts: http://itproguru.com/expert/2015/03/how-to-remove-azure-accounts-cached-credentials-from-powershell-remove-azureaccount-for-all-accounts-step-by-step/

Azure Web App Deployment With MsDeploy and Powershell

For my first attempt at automating an Azure deployment process, I had some MsBuild wrap the calls to the MsDeploy.exe. This lets me have the build server (eg. Team City) run the MsBuild script and call a simple “DeployX” target for the relevant Dev/Test/Production deployment. The MsBuild script is responsible for supplying MsDeploy with the correct settings. Problem is, the MsDeploy call passes Azure credentials (for the web app, not a user) as clear-text command-line parameters.

If we have those publish settings stored in the MsBuild script, we have to copy those parameter values from the Azure publish profile into a script (the build script) that lives in source control. Not so secure, and it involves some tedious copy-and-pasting.

While your application should know “how to deploy itself”, a big problem with locking your publishing credentials into your deployment packages is what happens if those settings should change. If you have a continuous delivery pipeline (you should), and if code takes a few days to travel through the pipeline from check-in to deployment, any code in the pipeline with outdated publish settings is hard to deploy (you may need to just grab the deployment packages and modify them). It would be better if the credentials lived outside of the build pipeline. You could have your build server pass these in via parameters, but that only works if you’re deploying one or two apps (I’m currently working on a project that deploys a set of eight apps together, whether that’s the correct long-term strategy is another question).

So a first step would be, rather than storing the Azure credentials in a script, just include the publish profile files and read them at deployment time. Then you can move them out of your code (source control and deployment packages) and store them on disk or in a different repository – anywhere that every developer or build server agent machine can access – and read the appropriate profile at deployment time.

You could do this with a custom application, script or MsBuild target that reads and parses the publish profile Xml file. Or just use some Powershell, it has some built-in XML hacking cpabilities. Instead of knowing the publish settings for the MsDeploy call, our MsBuild script can run a Powershell script instead.

Note that this isn’t the only way to deploy with Powershell. In fact, there’s probably better ways to use Powershell to deploy directly as a trusted Azure user, rather than all the hassle of downloading publish profiles. All we’re doing is wrapping the MsDeploy calls that actually ship our code.

A Powershell Script That Deploys Webapps

So create a Powershell script like this (yes the funny quotes to escape characters are weird but necesssary):

#Usage: powershell .\deployWebApp.ps1 [pathToWebAppPackageFolder] [webAppPackageZipFile] [pathToPublishSettingsFile] [configuration]
param([string]$packageFolderPath,[string]$packageZip,[string]$publishProfilePath,[string]$configuration)

#MsDeploy.exe location (if not in path)
[string]$msDeployExe="C:\Program Files (x86)\IIS\Microsoft Web Deploy V3\msdeploy.exe"

#Get publish settings from file
[xml]$xml=Get-Content($publishProfilePath)
[string]$azureSite=$xml.publishData.publishProfile.msDeploySite.get(0)
[string]$azureUrl=$xml.publishData.publishProfile.publishUrl.get(0)
[string]$azureUsername=$xml.publishData.publishProfile.userName.get(0)
[string]$azurePassword=$xml.publishData.publishProfile.userPWD.get(0)

#MsDeploy "white label" webapp package (additional configs to follow)
$msDeployPackageCommand = "`"$msDeployExe`" -source:package=`"$packageFolderPath\$packageZip`" -dest:auto,ComputerName=`"https://$azureUrl/msdeploy.axd?site=$azureSite`",UserName=`"$azureUsername`",Password=`"$azurePassword`",IncludeAcls=`"False`",AuthType=`"Basic`" -verb:sync -setParam:name=`"IIS Web Application Name`",value=`"$azureSite`" -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose -userAgent=`"VS12.0:PublishDialog:WTE12.3.51016.0`"";
Write-Output ("MsDeploy Command: " + $msDeployPackageCommand);
cmd.exe /c "`"$msDeployPackageCommand`"";

#MsDeploy config file "web.configuration.config" eg. "web.TEST.config"
$msDeployConfigCommand = "`"$msDeployExe`" -source:contentPath=`"$packageFolderPath\Configs\Web.$configuration.config`" -dest:contentPath=`"$azureSite\web.config`",ComputerName=`"https://$azureUrl/msdeploy.axd?site=$azureSite`",UserName=`"$azureUsername`",Password=`"$azurePassword`",IncludeAcls=`"False`",AuthType=`"Basic`" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose -userAgent=`"VS12.0:PublishDialog:WTE12.3.51016.0`"";
Write-Output ("MsDeploy Command: " + $msDeployConfigCommand);
cmd.exe /c "`"$msDeployConfigCommand`"";

For the configs, I run web.config transforms on all my configs during the CI/packaging build stages, and ship all the possible configs with the deployment package.

OK, try running the script with:

powershell powershellScripts\deployWebApp.ps1 BuildArtifacts\DeployableWebApp DeployableWebApp.zip C:\Development\AzurePublishProfiles\myappname.azurewebsites.net.PublishSettings Debug

This works, so we go and add this to our MsBuild script with this target:

<Target Name="PsDeployToDev">
 <Exec Command="$(PowerShellExe) powershellScripts\deployWebApp.ps1 $(PackageDeploymentPath)$(WebApp) $(WebApp).zip $(AzurePublishProfilesPath)\myappname.azurewebsites.net.PublishSettings Debug"/>
</Target>

Run it with:

msbuild WebDeploy.msbuild /t:PsDeployToDev

…and it fails, because of an execution policy issue. See http://go.microsoft.com/fwlink/?LinkId=135170

Yes, within this command prompt, I have the rights to run a powershell script, but not to run a script that runs my powershell script. Go Microsoft! Let’s add a few parameters to our MsBuild Powershell call to handle the execution policy:

<Target Name="PsDeployToDev">
 <Exec Command="$(PowerShellExe) -NonInteractive -ExecutionPolicy Unrestricted -File powershellScripts\deployWebApp.ps1 $(PackageDeploymentPath)$(WebApp) $(WebApp).zip $(AzurePublishProfilesPath)\myappname.azurewebsites.net.PublishSettings Debug"/>
</Target>

Success! You can test this by editing, re-building and re-deploying your web app.

I have this running as a Team City build step (MsBuild runner calls the appropriate target in my MsBuild script).

This is a starting point – a basic “ship a white-label package and a config file”, you’ll need to adapt it for deploying your applications.

Some next steps:

  • You’ll probably want to turn off verbose logging and outputting of the actual deployment command (with the publishing credentials arguments) once you’re happy things are working
  • You might need the “msdeploy.exe -useCheckSum” argument to prevent files getting redeployed after your build server does a clean checkout / rebuild

Discovering Web.configs for Deployments with MsBuild Custom Targets

My current job is working on a web application that needs a lot of configuration, and is deployed to multiple environments. We’re using web.config transforms to apply the configurations, but our build pipeline (Team City and MsBuild scripts) builds a deployment “package” as a release candidate, then a subsequent build step deploys (or redeploys) that package to test, staging and production environments (or doesn’t, if we don’t like the release candidate at any point). When we deploy, we ship the site plus one of around 20 transformed web.configs.

So while getting automated deployments up and running with MsDeploy, one of the things I needed to do was create a portable “white label” deployment package, then set the configuration to whatever environment I deployed to. As I’m using web.configs, I need to do the following:

  1. Generate all possible web.configs when I’m building the package to get a set of build artefacts consisting of zipped web application package plus transformed web.configs.
  2. Publish the site, then apply the appropriate web.config (from build artefacts) at deployment time.

So what I’m about to show is half “quick and dirty solution to my deployment problem”, and half “crazy things you can do with MsBuild”.

Publishing The Chosen Config

Using an MsBuild file to wrap my build and publish commands, I have this big chunk of MsDeploy for deploying my web.config to my Azure or IIS site:

<Target Name="_DeployWebConfig">
 <Exec Command='$(MsDeployExe) -source:contentPath="$(WorkingDir)$(PackageDeploymentDir)\Web. 
$(Configuration).config" -dest:contentPath="$(AzureSite)\web.config",ComputerName="https:// 
$(AzureUrl)/msdeploy.axd?site= 
$(AzureSite)",UserName="$(AzureUsername)",Password="$(AzurePassword)",IncludeAcls="False",AuthType="Basic" - 
verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension - 
retryAttempts=2 -verbose -userAgent="VS12.0:PublishDialog:WTE12.3.51016.0"'/>
</Target>

That’s all the MsDeploy for this post. We know we can ship the configs. First we need to get them.

Create All The Configs

My first attempt at transforming all the configs (from web project directory into the deployable package directory) is to include the “TransformXml” task at the top of the build script:

<UsingTask TaskName="TransformXml" AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web 
\Microsoft.Web.Publishing.Tasks.dll"/>

And then have a target I can call to transform the configs.

<Target Name="_TransformWebConfigs1">
 <TransformXml Source="$(WebApp)\Web.config" Transform="$(WebApp)\Web.Debug.config" 
Destination="$(PackageAssemblyDir)\Web.Debug.config" StackTrace="true" />
 <TransformXml Source="$(WebApp)\Web.config" Transform="$(WebApp)\Web.Test.config" 
Destination="$(PackageAssemblyDir)\Web.Test.config" StackTrace="true" />
 <TransformXml Source="$(WebApp)\Web.config" Transform="$(WebApp)\Web.Release.config" 
Destination="$(PackageAssemblyDir)\Web.Release.config" StackTrace="true" />
</Target>

Which is OK if you only have a few configurations. But what if you had to manage several environments? Let’s expand this target:

<Target Name="_TransformWebConfigs2">
 <ItemGroup>
 <Config Include="Debug"/>
 <Config Include="Test"/>
 <Config Include="Release"/>
 </ItemGroup>
 <TransformXml Source="$(WebApp)\Web.config"
 Transform="$(WebApp)\Web.%(Config.Identity).config"
 Destination="$(PackageAssemblyDir)\Web.%(Config.Identity).config"
 StackTrace="true" />
</Target>

Not much better, except now what we’re doing is creating an item group that lists all our configs, then using MsBuild’s “%” operator to iterate though all the items in our item group. Yes, that “TransformXml” call is basically a “foreach” loop.

Find All The Configs

What we really want is to find all the configs automatically. This is where we start to hit the limits of MsBuild (or at least my knowledge, I might have missed something in the reference), because we want to get a flattened directory listing of the configs.

Consider discovering the configs and loading them into an ItemGroup:

<Target Name="_DiscoverConfigs1">
 <!-- Finds all config files, BUT keeps webapp path pre-pended -->
 <ItemGroup>
 <Resources Include="$(WebApp)\Web.*.config"/>
 </ItemGroup>
 <Message Text="[@(Resources)]"/>
 <Message Text="All configs:" />
 <Message Text="[%(Resources.Identity)]"/>
</Target>

What we’d like is an ItemGroup with just “web.Debug.config”, “web.Test.config”, etc. I can’t find an easy way to do this using MsBuild syntax. But hey, MsBuild can run anything. So we could run this through a little script or even a tiny console app. Or just write C# directly in our build file. Yeah, let’s do that

First, add another property. We’re going to be creating our own MsBuild task.

<MSBuildPath Condition=" '$(MSBuildPath)'=='' ">C:\Windows\Microsoft.NET\Framework 
\v4.0.30319</MSBuildPath>

Now, let’s rewrite our discovery target:

<Target Name="_DiscoverConfigs2">
 <!-- Finds all config files, BUT keeps webappPath pre-pended -->
 <ItemGroup>
 <Resources Include="$(WebApp)\Web.*.config"/>
 </ItemGroup>
 <Message Text="[%(Resources.Identity)]"/>
 <_FlattenPathsTakeFilenamesOnly Filepaths="@(Resources)">
 <Output TaskParameter="Filenames" ItemName="WebConfigs"/>
 </_FlattenPathsTakeFilenamesOnly>
 <Message Text="[@(WebConfigs)]"/>
 <Message Text="All configs:" />
 <Message Text="[%(WebConfigs.Identity)]"/>
</Target>

And that “_FlattenPathsTakeFilenamesOnly” task? Let’s create it. Time to write some basic C# directly in our build script.

<!-- Create custom task to get file containing a version string in a directory -->
 <UsingTask TaskName="_FlattenPathsTakeFilenamesOnly" TaskFactory="CodeTaskFactory" 
AssemblyFile="$(MSBuildPath)\Microsoft.Build.Tasks.v4.0.dll">
 <ParameterGroup>
 <Filepaths ParameterType="System.String[]" Required="true"/>
 <Filenames ParameterType="Microsoft.Build.Framework.ITaskItem[]" Output="true"/>
 </ParameterGroup>
 <Task>
 <Using Namespace="System.IO"/>
 <Using Namespace="System.Linq"/>
 <Code Type="Fragment" Language="cs">
 <![CDATA[
 Filenames=new TaskItem[Filepaths.Length];
 for(int i=0;i < Filepaths.Length;i++)
 {
 Filenames[i]=new TaskItem(Filepaths[i].Substring(Filepaths[i].LastIndexOf("\\") + 1));
 //Console.WriteLine("Split " + Filepaths[i] + " > " + Filenames[i]);
 }
 ]]>
 </Code>
 </Task>
</UsingTask>

This is a trivial example, if you were building a custom task that really did something, write and compile it elsewhere (eg. as part of your solution).

So our final config discovery target looks like:

<Target Name="_TransformWebConfigs3">
 <ItemGroup>
 <Resources Include="$(WebApp)\Web.*.config"/>
 </ItemGroup>
 <Message Text="[%(Resources.Identity)]"/>
 <_FlattenPathsTakeFilenamesOnly Filepaths="@(Resources)">
 <Output TaskParameter="Filenames" ItemName="Configs"/>
 </_FlattenPathsTakeFilenamesOnly>
 <TransformXml Source="$(WebApp)\Web.config"
 Transform="$(WebApp)\%(Configs.Identity)"
 Destination="$(PackageAssemblyDir)\%(Configs.Identity)"
 StackTrace="true" />
</Target>

This just trawls through your web app source directory, finds all the source web.configs, runs each in turn through the transformer and dumps into your chosen directory. Which for our requirements, gives us all the possible web.configs ready in case we want to deploy one of them.

Pre-Transforming Web.Configs For Deployment

When your approach to automated deployment is to build a deployment package, then later deploy that to successive test/staging/live environments, you need to manage configurations. Unfortunately, Microsoft’s web.config approach is designed to work only as part of a configure-build-deploy process. For configuring at deployment time, you need a better approach (parameters files). Or a hack… There are better ways to manage configurations, but you might not have the ability to move away from web.configs.

A quick note: I originally included this section as part of a larger post on MsDeploy, but really this is a separate subject.

Transforming Web.Configs at Build Time

I’m going to manually add a target to my MsBuild file to transform my web.configs. I don’t know at build time where my deployment will end up, so I just include all of the possible configs.

We need a solution with an MsBuild file, and the necessary build configurations and web.config transform files setup. When building the deployment package, we also transform all the possible configs (and include in the deployment artifact). When deploying, we MsDeploy the entire site (from zip package), then do an additional MsDeploy of a single web.config file.

Now add a “_TransformWebConfigs” target and a reference to the “TransformXml” task that I’m going to borrow.

Right at the top of your build file, inside the Project element and before the PropertyGroup, add this:

<UsingTask TaskName="TransformXml" AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll"/>

For now I’ll just go process each configuration to create all possible web.configs in our package, transformed and ready to deploy. I’ve named the files “web.configurationName.config”, you could store them in “ConfigurationName\web.config” instead.

<Target Name="_TransformWebConfigs">
<TransformXml Source="$(WebApp)\Web.config" Transform="$(WebApp)\Web.Debug.config" Destination="$(PackageAssemblyDir)\Web.Debug.config" StackTrace="true" />
<TransformXml Source="$(WebApp)\Web.config" Transform="$(WebApp)\Web.Test.config" Destination="$(PackageAssemblyDir)\Web.Test.config" StackTrace="true" />
<TransformXml Source="$(WebApp)\Web.config" Transform="$(WebApp)\Web.Release.config" Destination="$(PackageAssemblyDir)\Web.Release.config" StackTrace="true" />
</Target>

And to call this, extend your “BuildPackage”, between building the package and copying to the deployment directory, add this:

<CallTarget Targets="_TransformWebConfigs" />

Now we add a “_DeployWebConfig” target that applies the correct config based on the $(Configuration) property (call MsBuild script with “/t:DeployPackage /p:Configuration=Test”). This is almost like the previous MsDeploy command, except:

    1. For the source, it really doesn’t like relative paths, so I grab the current working directory
    2. I’m explicitly setting the destination, because we’re renaming the file as we deploy
    3. You don’t set the “IIS Web Application Name”, MsDeploy complains if you do
<Target Name="_DeployWebConfig">
<Exec Command='$(MsDeployExe) -source:contentPath="$(WorkingDir)$(PackageDeploymentDir)\Web.$(Configuration).config" -dest:contentPath="$(AzureSite)\web.config",ComputerName="https://$(AzureUrl)/msdeploy.axd?site=$(AzureSite)",UserName="$(AzureUsername)",Password="$(AzurePassword)",IncludeAcls="False",AuthType="Basic" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose -userAgent="VS12.0:PublishDialog:WTE12.3.51016.0"'/>
</Target>

And you just insert the call to deploy the config at the end of your “DeployPackage” target:

<CallTarget Targets="_DeployWebConfig"/>

Note that your “web.X.config” gets renamed to just “web.config” on the target server during the deployment.

Automating Web Application Deployments to Azure with MsDeploy

We all know that having developers manually deploy code to test or production from their laptops is a really bad idea (wastes time, leads to unrepeatable “mystery” builds, hard to ensure you ship the approved and tested code, no traceability when things go wrong, etc.). So unless you’re willing to endure the three-monthly waterfall pain of spending days building and deploying the one true release version of your code, one of the first things you should do with a new project is set up a continuous integration (CI) and automated deployment system.

Microsoft make some great tools, but sometimes they make them really hard to use. A great example is their application deployment tool “MsDeploy”. Especially when it comes to Azure websites. When you’re manually deploying from Visual Studio then it’s easy, you just tweak the publish profile in a wizard and then hit “publish”, but as soon as you try to script and automate things, you run into all sorts of fun trying to figure out the MsDeploy syntax and error messages.

When my current project kicked off, we made the decision to host with Azure, and then found that setting up automated deployments to Azure is complicated. Because “MsDeploy”. So this is a guide, based on my experiences, on how you might go about automating your deployment process. It’s written with the aim of deploying a .Net MVC web application to Azure, but you can probably adapt for other purposes. I’ll also try and highlight any differences if you’re deploying to IIS instead.

Warning: this is the “works on my machine” version of this guide. I can’t guarantee that you won’t run into problems with your site, but I have successfully repeated this experiment on a couple of applications (including the one I actually get paid to build). So my aim here is not just to give you a bunch of MsDeploy command syntax, but to show how you can start to build and troubleshoot your own deployment process.

The Goal – Automated Build and Deployment

So this was all driven by the needs of our build and deployment process, which we want to be fully automated, and have an initial “continuous integration” build generate a versioned “release candidate” package that we can then use to deploy to successive test, staging and production environments (with the appropriate configuration). We’re agile, and we want to be able to deploy frequently to test and production, so we need an automated process.

In the real world, our product ships as multiple applications, to multiple production environments (different configurations). But we’ll start with the assumption that you just have a single web application to be deployed to “dev”, “test” and “production” environments.

So my inital aim is to eliminate Visual Studio from the process and get everything running on the command line. Those commands can then be scripted or otherwise run by the build server.

The Tools

First you’ll need Visual Studio (I built these examples with Community 2013). Thoughout this tutorial, I’m also going to be using MsBuild and MsDeploy on the command line. Oh, and you’ll need an Azure account (they do free trials). I’ll assume familiarity with VS and Azure, but MsBuild and especially MsDeploy might be new.

I’m not going to cover build servers, but instead just cover the process of escaping from Visual Studio to the command line. You should be able to get any build server to run these commands. And while I’m using MsBuild scripts to simplify some of the commands, you could do this another way. For the actual MsBuild and MsDeploy commands, I’m just going to call the .exes directly from my script, as you would on the command line. (Note that MsBuild has a built-in MsBuild task you can use, there can be complications getting this to use a suitable version of MsBuild for packaging and deploying).

By scripting deployments with MsBuild, I’m achieving a few things:

  1. Hiding command complexity (you tell the script to deploy and it handles all the MsDeploy arguments)
  2. Keeping complexity with the codebase – the build server calls a simple logical operation (“deploy to test”) and the build script (stored in source control with the code it builds) handles the details, making adding applications/artefacts to the build fairly easy
  3. Giving me an easy way to create and debug deployment operations locally without having to edit live on the build server

Finally, any version numbering is between you and your build server, I’m just assuming that we want one so we can track different releases (and track the current test/staging/production back to the code in source control) by setting the version number in AssemblyInfo files (at build time) then displaying somewhere (footer or about/diagnostics pages) as a quick check that the build and deployment actually worked.

The Approach

My basic approach to this has been to:

  1. Take the reference (manual) Visual Studio action, then script it
  2. Wrap commands in an easy-to-use format (MsBuild targets)
  3. Get a command working for hard-coded values before making it configurable
  4. Study errors carefully, check for stupid syntax errors, go slowly, be prepared to re-visit previous steps

This post steps through the process of getting a new web application to the point where it can be assembled into a deployable package, then deploying that package, without requiring Visual Studio. There’s some chunks of sample code (expect about 80 lines of MsBuild) for you to copy-and-paste and customise.

The Manual Publishing Process

If you manually publish to Azure or IIS from Visual Studio, you’re already using MsBuild and MsDeploy. Your process is probably something like:

  1. Create new build configurations, web.config transforms and publish profiles as necessary
  2. Make changes, test locally (uses MsBuild to compile code)
  3. “Publish” your project straight to Azure (builds, assembles what needs to be deployed, then uses MsDeploy to ship code to Azure)

So we want to take this process and automate it with some kind of build script, calling MsDeploy from our fancy new script.

Note: you can also publish your site to a package within Visual Studio and then deploy that later. If you’re already doing that, you’ve probably solved the hard part of deployments (actually deploying things).

A quick introduction to MsBuild and MsDeploy

MsBuild is Microsoft’s build engine. It’s what runs when you compile code in Visual Studio. But you can also run it on the command line, and feed it your own build scripts. In fact, you’re already doing that – a “.csproj” file contains MsBuild commands (XML).

MsDeploy is Microsoft’s deployment engine, and it’s what you’ll use to get your app deployed to Azure. In this scenario, it’s basically doing a “smart file copy” that allows the client (that would be us) to ship files (or at least changes) to the server (Azure), and set a few permissions along the way. MsDeploy can be run two ways – from Visual Studio, or from the command line (note that VS runs its own copy of the MsDeploy engine).

Warning: MsDeploy is painful to work with. Documentation is poor, the error messages are unhelpful, and taking a working set of commands for one solution and transplanting them to another may throw up all sorts of problems.

When you publish a site in Visual Studio, you’re running MsDeploy. You create a “publishing profile” (look in your web app project’s “Properties/PublishProfiles” folder), and then you publish with your chosen profile and configuration. There are basically two ways to publish an application in Visual Studio:

  1. The “Build and deploy” approach – compile the code, configure (web config transforms) and deploy as a single operation.
  2. The “Package” approach – compile the code and zip into a package, to be deployed at some later date.

We’re interested in creating and deploying packages, but we’ll be doing the “build and deploy” as an exercise.

I’m going to try and go through the steps that get you from a manual “build and deploy” to a fully automated system of building a package, then repeatedly deploying it to one environment after another.

Preparation

OK, so we need an application to deploy, an Azure site to deploy to, and an MsBuild script to hold the deployment commands. You can probably skip/skim these next sections.

Create a basic web app solution

You could go straight into automating your current project, but I’m going to use a really simple web app (called “DeployableWebApp”) for this example, you may want to do the same.

So open Visual Studio, and create a new Web Application. It doesn’t matter what you put in here, but I’d recommend you have the following:

  • A message or other output that requires compilation to change (to show it’s a new build)
  • A message on a view that you can change for each deployment (to show that the deployed code has changed)
  • A web.config setting representing the environment that you can display somewhere – we’ll use this to test applying the configuration at deployment time

Create an Azure webapp

Log into your Azure account and go create a new web application. Call it anything you want (mine is called “deployme-dev”). For now we only need one application for “dev”, but later we’ll try deployments to “test” and “production”. You’ll also want to go and download the publish profile file, which contains some settings we’ll steal later.

Create an MsBuild script

You can just put your MsBuild targets and properties into your “.csproj” file, but I prefer to keep my build scripts in separate files. Add a new solution item, an XML file called “WebDeploy.msbuild” (you can call it anything you want, including the file extension). And then create your most basic build script:

<?xml version="1.0" encoding="utf-8"?>
<!-- MsBuild script for using MS Deploy - must be run as admin! -->
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<!-- Add properties to configure stuff -->
<!-- Add targets to do stuff -->
</Project>

For this example, I’m using one MsBuild script for both the building/package and deployment operations. You might want to split these up, as they will ultimately run on different build/deployment processes.
As your MsBuild file grows, you’ll want to organise things. You can expect your build file to contain:

  • References to external tasks
  • Properties (for configuration)
  • Build targets
  • Deployment targets
  • Internal helper targets

Run a test command from an MsBuild target

Add this into the build file, in the “Project” element:

<Target Name="HelloWorld">
<Message Text="I am the thing that deploys other things" />
</Target>

To run this, you need to open a command prompt, navigate to your solution directory, then type:

msbuild WebDeploy.msbuild /t:HelloWorld

This passes your script to MsBuild and calls the “HelloWorld” target. But… unless you have “msbuild.exe” on your path, nothing will happen. So go edit your environment variables.

Or, try this trick – create a custom command prompt text file named “CommandPrompt.bat” in your solution directory, and make it look like this:

CD /D "%~dp0"
@%comspec% /k "SET PATH=C:\Windows\Microsoft.NET\Framework\v4.0.30319;%PATH%"

Tweak the paths, add your own extra tools etc. Now you can fire up your command prompt and msbuild.exe is on the custom path (which is used for commands run in this command prompt only).

Your general MsBuild-running syntax is

msbuild path-to-build-file /t:TargetName /p:Properties

You can specify multiple properties with either “/p:prop1=X /p:prop2=Y” or “/p:prop1=X;prop2=Y”.

Important note: for running MsDeploy you’ll need to be running the command prompt as Administrator.

Manual publishing with “build and deploy”

First we test that a manual publish in Visual Studio works and we can actually deploy to our Azure site.

Select your site and go to “build->Publish”, and create a new profile called “azure-publish-dev”. On the connection tab, set method to “web deploy”, and raid the publish profile to set server to the publish Url (ends “:443”), site name to msdeploySite, username and password to the generated values. For now, you can ignore the “settings” tab, just go and publish. Hopefully this works. Go change something and re-publish to prove it.

So what this actually did was the following:

    1. Built your application (look in the “project/obj/config” folder)
    2. Figured out what needs to be included (look in “project/obj/config/Package” folder, especially the “manifest.xml” file)
    3. Ran MsDeploy to copy the changes

Look in the build output window for more details. Something you’ll notice is a line like this:

Start Web Deploy Publish the Application/package to "https://deployme-dev.scm.azurewebsites.net/msdeploy.axd?site=deployme-dev ..."

And yes, that command is partially-hidden. Welcome to MsDeploy. It can be handy to know what Visual Studio is doing when it publishes. Unfortunately, Visual Studio has its own way in to the MsDeploy engine.

MsDeploy and command discovery

So by default, Visual Studio is accessing the MsDeploy assemblies directly. But you can switch it to use the command-line executable instead. Basically, you just create a “MyWebAppProjectName.wpp.targets” file in your project (it’s an MsBuild file that gets run during the build), and use it to set the “UseMsdeployExe” property to “true”. Best add to your solution so VS finds it. Full details here:

http://blogs.msdn.com/b/webdev/archive/2010/11/03/web-deploy-how-to-see-the-command-executed-in-visual-studio-during-publish.aspx

Anyway, you should now see the full msdeploy.exe call in the build output. That’s going to form the basis of everything else we do. Start by copying what Visual Studio does. Then we can adapt to other deployment scenarios.

You can re-run this command as a test. Either paste straight into a command prompt, or paste into a new text file (call it “deploy.cmd” and run by typing “deploy”). You’ll need to make sure anything with spaces (eg. the msdeploy.exe call) is wrapped in quotes, and you might need to swap some single quotes for double quotes.

A reminder – you’ve just told Visual Studio to use a different method of deploying your project. So if strange things suddenly occur when you publish with Visual Studio, be prepared to revert this change.

Deploying Without Visual Studio

The MsDeploy commands are long and complex, so we’re going to wrap them up in MsBuild targets for simplicity.

First, as we go, I’ll be replacing hard-coded values with properties. So in your MsBuild file, inside the “Project” element, add a PropertyGroup that looks like:

<PropertyGroup>
<Configuration Condition=" '$(Configuration)' == '' ">Release</Configuration>
</PropertyGroup>

This creates a property called “Configuration”, gives it a default value (“Release”), and lets you use it later within a Target by inserting “$(Configuration)” into a command. You can override the default with a “/p:Configuration=Test” argument when running the build. I’ll be using these properties throughout to make the various commands readable and configurable. The full set of properties we need looks like this, either add them all now or as you need them:

<PropertyGroup>
<!-- Exes need full path specified if not already available on %PATH% -->
<MSBuildExe Condition=" '$(MSBuildExe)'=='' ">msbuild.exe</MSBuildExe>
<MSDeployExe Condition=" '$(MSDeployExe)'=='' ">msdeploy.exe</MSDeployExe>
<Configuration Condition=" '$(Configuration)' == '' ">Release</Configuration>
<SolutionName Condition=" '$(SolutionName)' == '' ">DeployableWebApp</SolutionName>
<WebApp Condition=" '$(WebApp)' == '' ">DeployableWebApp</WebApp>
<WorkingDir Condition=" '$(WorkingDir)' == '' ">$(MSBuildThisFileDirectory)</WorkingDir>
<PackageAssemblyDir Condition=" '$(PackageAssemblyDir)' == '' ">DeployableWebApp\ReleasePackage</PackageAssemblyDir>
<PackageDeploymentDir Condition=" '$(PackageDeploymentDir)' == '' ">DeploymentPackage</PackageDeploymentDir>
<PublishProfileDeploy Condition=" '$(PublishProfileDeploy)' == '' ">azure-publish-dev</PublishProfileDeploy>
<PublishProfilePackage Condition=" '$(PublishProfilePackage)' == '' ">azure-package</PublishProfilePackage>

<!-- Azure settings from the .PublishSettings file -->
<AzureUsername Condition=" '$(AzureUsername)' == '' ">$john-doe</AzureUsername>
<AzurePassword Condition=" '$(AzurePassword)' == '' "></AzurePassword>
<AzureSite Condition=" '$(AzureSite)' == '' ">site-name</AzureSite>
<AzureUrl Condition=" '$(AzureUrl)' == '' ">site-name.scm.azurewebsites.net</AzureUrl>
</PropertyGroup>

Something that’s really important when figuring out and testing msdeploy commands is to always start from a “clean” state, so create yourself a “CleanAll” target:

<Target Name="CleanAll">
<PropertyGroup>
<BinDir>$(WebApp)\bin</BinDir>
<ObjDir>$(WebApp)\obj</ObjDir>
</PropertyGroup>
<RemoveDir Directories="$(binDir);$(ObjDir)"/>
<Exec Command="$(MSBuildExe) $(SolutionName).sln /t:clean"/>
</Target>

Run it with “msbuild WebDeploy.msbuild /t:CleanAll” and watch it clean.

If you want to just check your code builds as well, create yourself a “Build” target:

<Target Name="Build">
<Exec Command='$(MSBuildExe) $(SolutionName).sln /t:Build /p:Configuration=$(Configuration)'/>
</Target>

And a quick point about the MsBuild syntax: I’m calling MsBuild from within MsBuild, using the msbuild.exe like you would on the command line. MsBuild also has its own “MsBuild” task syntax, which looks like this:

<MSBuild Projects="$(SolutionName).sln" Targets="Build" Properties="Configuration=$(Configuration)"/>

Use either version, they both call MsBuild with the specified script (a “.sln”, “.csproj”, or your own custom MsBuild file), and run the specified target with the supplied properties.

Building and deploying in a single step

So a quick detour – our goal is to build a package and then deploy that package in a separate step. But first, let’s take the “build and deploy” example above and remove Visual Studio from the process.

We’re going to create a “BuildAndDeploy” target. And to make sure we do a “CleanAll” first, the “DependsOnTargets” lets you specify other targets that must be run first. My initial target looks like this:

<Target Name="BuildAndDeploy" DependsOnTargets="CleanAll">
<Exec Command='$(MSBuildExe) $(WebApp)\$(WebApp).csproj /t:Build /p:VisualStudioVersion=11.0 /p:DeployOnBuild=True /p:PublishProfile="$(PublishProfileDeploy)" /p:Configuration=$(Configuration) /p:Password="$(AzurePassword)"'/>
</Target>

Note: when setting up my publish profile, I checked “save password”. Inspecting the .pubxml shows a “_SavePWD” property but no “Password” property. Checking the generated msdeploy command shows no password, and deployment fails. So add your password to the publish profile (in “Password” property) and don’t override on your MsBuild call, or add a “/p:AzurePassword=letmein” argument to the msbuild call. And yes, having a property called “AzurePassword” that we then assign to “Password” looks strange. I prefixed all the Azure properties with “Azure” for readability. If we really did want to do a single-step “build and deploy”, using the expected “Password” property would make a lot more sense, but the whole point of this post is for those times when you don’t want to “build and deploy”.

You’ll notice that we’re not calling MsDeploy directly, just building the project, but that triggers calling MsDeploy. You’ll also notice from the msdeploy command that the “.SetParameters.xml” file is referenced to supply some of the deployment values. Also, setting the VisualStudioVersion is probably necessary (removing it can cause issues with directory paths).

Building a deployment package

Having to build the code at deployment time isn’t always very useful, because you end up rebuilding the same thing over and over as you promote the same code to different environments (also, it can screw up your versioning strategy). So we already saw how Visual Studio’s publish does a two-step “build/assemble the site, then deploy it”. We’d like to perform these as separate actions:

    1. Build and assemble into a package, environment-neutral (optionally stamped with a version number)
    2. Some time later deploy the package to test, applying “test” configuration
    3. Some time later deploy the package to production, applying “production” configuration

This is important: The package doesn’t know where it will eventually be deployed and configured.

So in Visual Studio, go to publish your site, but this time create a new publish profile of type “Web Deploy Package”. In settings, for the package location enter “ReleasePackage\SiteName.zip” and leave the site name blank. Publish and you get a zip of your site and a few XML files. You’ll also get a “.deploy.cmd” file, which contains everything you need for a one-click deploy. Except, it’s missing a few necessary parameters and values that you need to figure out. Thanks, Microsoft. Actually, all the deployment “.cmd” gives you is the MsDeploy command we’re about to build.

Time to recreate this in our script, so back to the MsBuild, and add some new targets:

<Target Name="BuildPackage" DependsOnTargets="CleanAll">
<Exec Command='$(MSBuildExe) $(WebApp)\$(WebApp).csproj /t:Build /p:VisualStudioVersion=11.0 /p:DeployOnBuild=True;PublishProfile="$(PublishProfilePackage)";ProfileTransformWebConfigEnabled=False;Configuration=$(Configuration)'/>
<CallTarget Targets="_CopyPackageToDeploymentDirectory"/>
</Target>

<Target Name="_CopyPackageToDeploymentDirectory">
<ItemGroup>
<PackageFiles Include="$(PackageAssemblyDir)\**"/>
</ItemGroup>
<Copy SourceFiles="@(PackageFiles)" DestinationFolder="$(PackageDeploymentDir)\%(RecursiveDir)"></Copy>
</Target>

So this should build a big ZIP archive of your site, with a few other generated “.cmd” and “.xml” files.

One of the things I’ve added here is the copying of the package from the original “assembly” directory to somewhere else. In real world scenarios, your build will end up generating a package that you then stash somewhere (eg. a “build artefacts” folder) then copy to whatever workspace directory your deployment process runs in for the publishing. So to avoid any reliance on anything other than what’s in the package, you can build the package, move it somewhere for deployment, then clean your workspace.

Note: The MsBuild call for building the package is a little strange – you “build” the project but specify that you want a package (deploy with the publish profile, which creates a package). There’s also a “Package” target, which skips building and compiles a .zip package, but may or may not contain everything. Publishing to a package from Visual Studio gives you a reference zip file.

Deploying the package

Right. Scary experiment time! Add the deployment target:

<Target Name="DeployPackage">
<Exec Command='$(MsDeployExe) -source:package="$(PackageDeploymentDir)\$(WebApp).zip" -dest:auto,ComputerName="https://$(AzureUrl)/msdeploy.axd?site=$(AzureSite)",UserName="$(AzureUsername)",Password="$(AzurePassword)",IncludeAcls="False",AuthType="Basic" -verb:sync -setParam:name="IIS Web Application Name",value="$(AzureSite)" -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose -userAgent="VS12.0:PublishDialog:WTE12.3.51016.0" -whatif'/>
</Target>

This is the MsDeploy call we stole from the Visual Studio publish output, tweaked with some readable/configurable parameter values. If msdeploy.exe isn’t already on your path, go to the PropertyGroup and set the value of $(MsDeployExe) to “C:\Program Files (x86)\IIS\Microsoft Web Deploy V3\msdeploy.exe” (including the quotes).

OK, try running this:

msbuild WebDeploy.msbuild /t:DeployPackage /p:AzureUsername="U";AzurePassword="P"

(If you haven’t already set the Azure username/password in your build file, make sure you pass the correct values in).

You should see a big long msbuild.exe call in the output, assuming there’s no missing quotes or other typos.

This is the point where it might all fail and spew red text and error messages, and you have to go check the MsDeploy syntax and figure out which parameters to set and what values to use. It’s essential to eliminate anything relating to the “build” environment where the package was created, so ensure no reference to hard-coded build location or deployment Urls in the “package.pubxml” and no use of a generated “parameters.xml” file (a source of many problems).

We haven’t actually published any changes, yet. See the final “-whatif” parameter? That runs MsDeploy in a demo mode where it doesn’t actually change anything (but will complain at the slightest syntax error or missing/incorrect parameter name/value). So remove the “-whatif” and try a real deployment.

Deploying Only What Has Changed

Usually, you only want to deploy the changes. After all, if your application consists of thousands of files, of which only 20 have actually changed in a given build, you want the process to be fast. MsDeploy will deploy only the changes, but by default this is based on file dates. If you do a clean checkout and full rebuild, your “unchanged” files may have a more recent date than their identical counterparts on the server. To prevent redeploying everything, add the “-useCheckSum” argument to your MsDeploy call:

msdeploy.exe -usechecksum

You’ll get a summary of the changes actually shipped when you run MsDeploy, so you can verify if it’s copying too many files.

Bonus: Deploying to IIS

So on my way to figuring out getting MsDeploy to work with Azure, I started with the most basic publish to a local IIS site. If you want to adapt this experiment for IIS instead of Azure, you’ll need the MsDeploy service running on your server.

My original command went something like this (site is named “DeployableWebApp” in IIS):

<Exec Command='$(MsDeployExe) -source:package="fullpathtopackage.zip" -dest:auto,includeAcls="False" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -setParamFile:"fullpathto.SetParameters.xml"' />
<Exec Command='$(MsDeployExe) -verb:sync -source:contentPath="fullpathtopackage\Web.$(Configuration).config" -dest:contentPath="DeployableWebApp\web.config",includeAcls="False" -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension' />

Note the use of the “SetParameters.xml” file. It didn’t “just work” when I added the step to move the package from an assembly to a deployment directory before deploying. It even complained about the app pool settings (which was probably actually more to do with not finding the dest site and assuming defaults). So let’s get rid of that hassle:

<Exec Command='$(MsDeployExe) -source:package="$(PackageDeploymentDir)\$(WebApp).zip" -dest:auto,includeAcls="False" -verb:sync -setParam:name="IIS Web Application Name",value="DeployableWebApp" -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose -userAgent="VS12.0:PublishDialog:WTE12.3.51016.0"' />
<Exec Command='$(MsDeployExe) -source:contentPath="$(WorkingDir)$(PackageDeploymentDir)\Web.$(Configuration).config" -dest:contentPath="DeployableWebApp\web.config",includeAcls="False" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose -userAgent="VS12.0:PublishDialog:WTE12.3.51016.0"' />

Again, the “package” and “selected file” deployments are different. “IIS Web Application Name” (your destination configuration) is not an option when the source is “contentPath”.

Build Once, Deploy Often

If this is somehow working, what you have is pretty awesome – you can call “BuildPackage” then “DeployPackage” for a two-step build-and-deploy process. But we’ve only deployed to a single location. And you only get the default configuration – add a “/p:Configuration=Test” onto the end of your deployment command and nothing happens.

Config transforms and parameters

Managing configuration options, especially when you don’t know at build time which environment(s) your build will get deployed to, is a topic for another post.

However, as a bonus step, it’s worth taking a quick detour into web.configs to illustrate the subtle ways in which MsDeploy can treat shipping deployment packages and individual files differently.

Bonus – Transforming and deploying a web.config

Web.configs aren’t really designed for use in a two-stage “build/package then deploy” process, but you can cheat for now. See this post on Pre-Transforming Web.Configs For Deployment.

Build once, deploy everywhere

What about taking our “dev” package and deploying it to “test”? We need to check that nothing in our generic package is tied to our current dev environment, because we want to deploy the package to “dev”, then “test”, then “production”. We can test this easily:

    1. Go back to your Azure dashboard and create another two web apps for “deployme-test” and “deployme-production”
    2. Grab the publish profile files
    3. Publish!

Now you could go through all the previous steps of creating a publishing profile, testing the build-and-deploy in Visual Studio then the command line, and then trying to publish the package. Or… you could just trust that your “DeployPackage” target actually uses the property values you supply. So as a quick test, you can edit the hard-coded values for your Azure site and credentials, or just skip the properties and create multiple “DeployPackageDev”, DeployPackageTest” etc. targets.

But really, you want to pass these values in on the command-line with the “/p:Param=Value” syntax, and specify your “AzureUrl”, “AzureSite”, “AzureUsername” and “AzurePassword”.

Remember, you have to run “BuildPackage” to get your changes into the package, but then you can deploy that to wherever you want. And after you’ve built the package, you can clean your solution, including the package assembly directory (the package has been copied elsewhere) and deploy that package again and again.

This is where having web.config transforms set up is awesome – you can transform all the configs, bundle them into the package, then have the deployment set whatever configuration works for your target environment.

So you could do

  • “BuildPackage”, “DeployPackage” (with dev settings and config), “DeployPackage” (with test settings and config)
  • “BuildPackage”, “CleanAll”, “DeployPackage” (to dev), “DeployPackage” (to test), etc

One final thing – you may still have your Azure credentials hidden in your build file, which is going to end up in source control with the rest of your project, and maybe even with the package. So you might want to remove these (and pass them in via parameter when you run the deployment script).

Shipping the build script

If you actually want to be deploying these packages you create (eg. from your build server), you’ll need access to the build script. Which probably means shipping the build script with the package (so the correct version of the script is used to create and then ship the package). You can do this with some simple file copying – just create yourself an MsBuild target to have the MsBuild script include itself alongside the package.

Summary

The things to look out for

Getting to this point took a lot of trial and error and searching and wondering if it was even possible to drive MsDeploy outside of Visual Studio. The main things that caused me problems:

  • Building the package – beware the project “Package” target, make sure you build and zip up everything (or your deployed app may complain about missing assemblies). Also, the permissions may differ between Visual Studio-generated packages and those your build script spits out (file sizes differ by a few bytes), shouldn’t actually cause any issues.
  • Setting the VisualStudioVersion on build package calls – fail to enforce this and strange things happen, like your published package not ending up in the expected folder, which can then kill subsequent actions (like those web.config transforms). The “DesktopBuildPackageLocation” path in your publish profile may be ignored in favour of just using the “app/obj/Release/Package” directory for assembling the release package.
  • The MsBuild.exe used on the local path may not be the same MsBuild used by your build server, build packaging behaviour differs between versions.
  • Reading the error messages – first make sure you know whether the error is just a typo with your MsBuild/MsDeploy syntax (a misplaced quote), a missing file, or an actual MsDeploy issue.
  • Parameters XML files – caused me endless pain, setting values based on local settings, refusing to let me override values for the remote server, erroneously complaining about ACL errors, AppPool settings (when deploying to IIS), these seem to be a real problem if you don’t publish from the directory you originally assembled the package in. I basically just tried to get rid of these and specify as arguments to my MsDeploy command instead. If you can see everything in one big long MsDeploy command, you might spot issues quicker.
  • Absolute paths and local workspace build paths getting passed to MsDeploy (check those parameters files, and your publish profile). When this ends up on a build server, not only will you be deploying from a different location to where you built your package, but you won’t even be building the package in the same location you did on your local workspace.
  • Relative paths and contentPath – when deploying files (source is “contentPath”) rather than a package, MsDeploy needs a full path to the file, relative paths give you “site does not exist” errors because your source files can’t be found.
  • Viewing a working MsDeploy command – capture the working Visual Studio version, use as a reference, dump the output of your command-line deployment run to a text file and inspect for differences.
  • The MsDeploy “source”/”dest” arguments – take care that you set source to “package” for your zipped package, “contentPath” for individual files, and match with a “dest” of “auto” or “contentPath” (especially if source and dest names change).
  • “IIS Web Application Name” is only applicable in certain deployment cases.
  • The package is not your average “zip” file – if you were to open up the generated .zip package, amend its contents, then zip it up again, you might find deploying it as a package won’t work (complaints about “IIS Web Application Name”, because MsDeploy doesn’t recognise it as a package). Just because a file is called “*.zip”, doesn’t mean it is one. I saw this once when cheating the scripted build process, and I think it’s a compression issue. In these examples, I’m deploying the generated package, then the configs. But you might want to be sneaky and hack the package you’re about to deploy so you can do something clever, like having automatic deployment to a “staging” slot that swaps that to “live” only if the single deployment step is successful.
  • Deploying all the files all the time – add the “useCheckSum” argument to prevent this.

What we have achieved

All this changing the site, doing a “BuildPackage” then a “DeployPackage” is fun, but what have we actually achieved? We should now have the following at our disposal:

  1. A strategy for mimicking Visual Studio’s “build and deploy” process (useful for getting a reference deploy command)
  2. A process for building our web app into an independent deployable package (without using Visual Studio)
  3. A process for deploying our generic web application package to a destination of our choice (without using Visual Studio), we don’t even have to choose between IIS and Azure until we deploy
  4. A process for applying a chosen web.config at deployment time (without using Visual Studio)

And once you replace Visual Studio with something that runs on the command line, suddenly all sorts of automated build and deployment scenarios open up. Imagine a scenario where your build and deployment pipeline is:

  1. Continuous integration builds a release candidate package on every build (and stamps a version number on it)
  2. Package is later deployed to test environment
  3. Later, the same package (that’s the tested code, not the current “head” of your development branch) is deployed to staging, production, etc.

There are a few things missing to get there, and I’ll cover these in other posts:

  1. Configuration – how web.config transforms, and an alternative “deployment parameters” approach, lets you tweak the configuration at deployment time (including shipping to environments that didn’t exist at build time)
  2. Managing the chaos of the MsBuild file when you want generic build and deploy targets to serve multiple apps and environments
  3. Adding these build steps to a build server for true automated deployment
  4. Adding database migrations to the process
  5. Adding other non-web-config-transform configurations (eg. theme assets) to your deployments