Choosing What Gets Deployed When Publishing A .Net Web Application

When you use MsBuild to package up a web application for publishing, you typically rely on the publishing profile, to manage either a “build and publish” or “build a package for later deployment” process (a .zip package is created either way). By default, everything in your project may be copied to the output and included in the deployment package .zip file, unless you make adjustments.

You already have some control over whether files get copied into the output and packaged up, using file properties in the solution explorer, as set by the “Build Action” (set to “none” rather than “content”) and “copy to output directory” properties. However, it may be easier to just use some MsBuild to avoid packaging several files (or whole folders) based on a wildcard pattern.

Example MsBuild syntax for generating the deployment package would be:

msbuild WebApp\WebApp.csproj /t:Build /p:VisualStudioVersion=11.0;DeployOnBuild=True;PublishProfile="DeploymentPackage.pubxml";ProfileTransformWebConfigEnabled=False;Configuration=Release;

 

Including Extra Files In The Package

I previously covered ways to include extra files that your project doesn’t know about:

https://ajmburns.wordpress.com/2016/04/10/including-additional-generated-files-when-publishing-a-net-web-application/

Removing Files From The Package

If you wanted your deployment to omit selected files or folders, you can get the publish profile to that as well. I have to do this to hide deployment parameter config files and selected custom scripts that are part of my solution, but should not be deployed.

To test, add the excludes to the main “PropertyGroup” element of your publish profile, then build the package and check the generated package .zip file.

Example – excluding a list of custom scripts:

<ExcludeFilesFromDeployment>
 compile-this.bat;compile-that.bat
</ExcludeFilesFromDeployment>

Example – excluding a wildcard list of configs:

<ExcludeFilesFromDeployment>
 parameters*.xml
</ExcludeFilesFromDeployment>

Example – excluding a folder:

<ExcludeFoldersFromDeployment>
 myConfigs
</ExcludeFoldersFromDeployment>

MsBuild Version Gotchas – Better Building with MsBuild scripts and Build Servers

I recently ran into issues when upgrading an old codebase from the original setup (Visual Studio 2013 and the MsBuild tools v4.0) to use Visual Studio 2015, building through Team City on a new machine that only had VS2015. The MsBuild scripts I had been using for building (mostly creating the deployment package) and actual deployment (calling MsDeploy) needed a few tweaks. Mostly, I had taken and adapted a boilerplate MsBuild script from an earlier project, left in some stupid hacks that worked well until they didn’t, and found my scripts needed a few tweaks to work with the current tools.

My build process was to use Team City to compile, package, and later deploy (via MsDeploy) a web app to Azure, using MsBuild scripts to manage the actual solution build/package/deploy specifics.

For the purposes of this post, Team City happens to be my build server of choice. If the detail of your build process is stored and versioned with the code (and not the build server configs) then switching build servers should be easy.

So this post is some note-to-future-self gotchas for the next time a tools upgrade breaks the build…

MsBuild.exe path and version

First issue was to check the location of the MsBuild.exe itself. As this wasn’t already in my path, I had been specifying it explicitly as a command prompt setup for locally debugging build scripts, using a “commandPrompt.bat” with the following line

@%comspec% /k "SET PATH=C:\Windows\Microsoft.NET\Framework\v4.0.30319;

So first issue is to fix the path and use the latest version – “C:\Program Files (x86)\MSBuild\14.0\Bin”.

Setting the local path to MsBuild isn’t strictly required – normally I’d build locally through Visual Studio, but I like being able to test the build scripts locally without having to push them to the build server, and wrapping the build specifics in an MsBuild script lets me do this. The important thing is that I want Visual Studio and Team City build to compile the code with the same MsBuild engine so there’s no surprises.

Whenever I run the MsBuild script eg. with a

msbuild WebPackage.msbuild /t:BuildPackage

I get the MsBuild engine version reported, but to help debug my scripts, I tend to put some kind of “Info” target in to report paths and settings, eg:

<Target Name="Info">
 <Message Text="MsBuildExtensionsPath=$(MsBuildExtensionsPath)" />
 <Message Text="MSBuildToolsVersion=$(MSBuildToolsVersion)"/>
 <Message Text="VisualStudioVersion=$(VisualStudioVersion)"/>
 <Message Text="VSToolsPath=$(VSToolsPath)"/>
</Target>

Now I can run:

msbuild WebPackage.msbuild /t:Info

If the latest MsBuild is on the path, I should get latest version reported (v14.0).

Also, in my .csproj file I found references in a PropertyGroup to “VisualStudioVersion” and “VSToolsPath” settings.

<PropertyGroup>
 <VisualStudioVersion Condition="'$(VisualStudioVersion)' == ''">10.0</VisualStudioVersion>
 <VSToolsPath Condition="'$(VSToolsPath)' == ''">$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v(VisualStudioVersion)</VSToolsPath>
</PropertyGroup>

Your build scripts shouldn’t really be tying themselves to specific tools versions, better to set these values and pass them in to any running MsBuild process.

MsBuild Scripts Calling MsBuild

Noticed another problem – I had my MsBuild script calling the MsBuild.exe directly:

<Exec Command="$(MSBuildExe) $(SolutionName).sln /t:clean"/>

…and because I couldn’t guarantee MsBuild.exe in the path, I cheated and had my MsBuild script setting the MsBuild.exe path to a default if it wasn’t overridden:

<MSBuildExe Condition=" '$(MSBuildExe)'=='' ">C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe</MSBuildExe>

So checking the output, I was using the old MsBuild engine version. Oops. I could probably have allowed the .exe location to be specified either on the path, as an argument to the script, or as a default in the script, but that’s a hassle. Luckily, to free myself from nasty tools path dependencies, I can rewrite my MsBuild calls as:

<MSBuild Projects="$(SolutionName).sln" Targets="clean"/>

The general format is:

<MSBuild Projects="filename" Targets="target" Properties="prop1=val1;prop2=val2"/>

This should mean that whatever MsBuild engine you use to kick off the script is the one that gets used throughout. I have had issues with a few quotes around property values (solution: don’t quote property values), but it should be possible to express all MsBuild calls using the MsBuild task.

Importing MsBuild Tasks

Ok, for some reason, probably relating to some hacky web.config transforming I had done on previous projects (I have done terrible things to get applications to deploy and workaround the “configure-build-deploy” workflow of web.configs), I had the following using statement in my MsBuild script:

<UsingTask TaskName="TransformXml" AssemblyFile="C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll"/>

This was actually referencing non-existent DLL

C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll

Which should be

C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.Tasks.dll

There’s another issue here – this will almost certainly require Visual Studio installed on the build server, or a cut down install eg. http://www.nuget.org/packages/MSBuild.Microsoft.VisualStudio.Web.targets/

Turns out, I actually no longer needed to use this task for building this project and I can just omit this line. But there will be times when your build script needs access to those extra tasks. If you can, try to avoid your build script knowing exactly where on the build server to look for stuff.

Team City Settings

So finally, I had to go into my Team City build configuration for all my build steps that used MsBuild and upgrade the MsBuild tools versions to latest. And check you’re using the same version throughout the build (I ran into problems compiling and packaging the same web app with different MsBuild versions in different Team City build steps).

Summary

Things to watch for:

  • Wrong MsBuild version on path – to debug, create an “Info” target in your build script and echo property values to console / build server log
  • Specifying the MsBuild version in your script – use the “MsBuild” task instead of direct MsBuild.exe calls to prevent specifying the .exe location on the path or in your script
  • References to a specific Visual Studio / tools version that may not be installed – eg. common MsBuild targets
  • Copying-and-pasting a previous build script and not cleaning out the unused stuff when you find a better way of doing things

Also – limit the use of MsBuild, try keep it simple, use it for building, packaging, including/excluding/copying files to be deployed. Handle deployment and other complex tasks through Powershell.

Including Additional / Generated Files When Publishing a .Net Web Application

When you publish a .Net web app, you typically set up a publish profile, either to do an immediate build-and-publish, or to publish to a package for later deployment (eg. single-trunk / build-once-deploy-many-times scenarios).

So for building a package, the MsBuild syntax would look like:

msbuild WebApp\WebApp.csproj /t:Build /p:VisualStudioVersion=11.0;DeployOnBuild=True;PublishProfile="DeploymentPackage.pubxml";ProfileTransformWebConfigEnabled=False;Configuration=Release;

You just use MsBuild and ask your project to build itself. The good bit is that your project knows what it contains, and therefore what needs deployed. The bad bit is that your project knows what it contains, and therefore what needs deployed… Any files generated by an external tool, or any other files you didn’t explicitly tell Visual Studio (listed as content in the .csproj file) are excluded. Thankfully, you can include extra stuff into your deployment package with a few lines of MsBuild commands. While you can probably hack the .csproj, the easiest way I’ve found is just to plug some MsBuild directly into the publish profile.

This location does make sense – you’re specifying what gets published after all. Note that if you have several publish profiles (maybe you have branch-based deployment and per-environment build-and-publish) then things get more complicated. I’m going to stick with the simple case – a single publish profile for creating a generic deployment package that we can then deploy (with appropriate configuration) to any target environment.

I recently had to solve this problem for handling some generated CSS, but you can use it for any non-solution files you want to include as part of the web application deployment package zip.

A quick warning – this does involve editing a generated file (the .pubxml file that you probably only create once).

Tweaking The Publish Profile

So assume I have some tools generating CSS (in “/webproj/content/styles”) and extra text files (in “/webproj/content/documentation”). I don’t even need to set up tools to generate these files – I can actually just simulate by manually adding some dummy files. I’m assuming that the files get generated within the project directory (because you want to deploy them to IIS or Azure when you deploy your site).

Anyway, create the extra files within the project directory, don’t tell Visual Studio or your project about them, and then build the deployment package. You’ll see they get omitted. We can fix that.

Go edit your publish profile, and add these lines inside the “PropertyGroup” element – we need to add a “CopyAllFilesToSingleFolderForMsdeployDependsOn” property.

 <CopyAllFilesToSingleFolderForMsdeployDependsOn>
 IncludeCustomFilesInPackage;
 $(CopyAllFilesToSingleFolderForMsdeployDependsOn);
 </CopyAllFilesToSingleFolderForMsdeployDependsOn>
 </PropertyGroup>

This goes off and calls a target called “IncludeCustomFilesInPackage” that we’ll create in a minute. The name of the “CopyAllFilesToSingleFolderForMsdeployDependsOn” property is important (MsBuild will look for it), the name of the custom “IncludeCustomFilesInPackage” target can be whatever we want (change the names and see what happens).

There is also a “CopyAllFilesToSingleFolderForPackageDependsOn” property, but this seems to get ignored even when building in package mode. For completeness, it would look like this:

<CopyAllFilesToSingleFolderForPackageDependsOn>
 IncludeCustomFilesInPackage;
 $(CopyAllFilesToSingleFolderForPackageDependsOn);
</CopyAllFilesToSingleFolderForPackageDependsOn>

But you don’t seem to need it – just add the “CopyAllFilesToSingleFolderForMsdeployDependsOn”.

Now we can go define our “IncludeCustomFilesInPackage” target, so add this to the .pubxml file, inside the “Project” element:

<Target Name="IncludeCustomFilesInPackage">
 <Message Text="Collecting custom files..."/>
 <ItemGroup>
  <CustomFiles Include="Content\styles\**\*" />
  <CustomFiles Include="Content\documentation\**\*" />
  <FilesForPackagingFromProject Include="%(CustomFiles.Identity)">
   <DestinationRelativePath>%(CustomFiles.Identity)</DestinationRelativePath>
  </FilesForPackagingFromProject>
 </ItemGroup>
 <Message Text="Add to package %(FilesForPackagingFromProject.Identity)"/>
</Target>

Add as many entries as you need into the ItemGroup. I added a couple of “message” calls to output progress.

So when you run your package build and inspect the final .zip package that gets created (check in the “obj” directory eg. “projectName/obj/projectName/projectName.zip”) then it should contain all those extra files that your .csproj didn’t know about.

Troubleshooting

When you’re setting this up, you might run into problems getting the paths correct. You can always add in a load of Message calls, and redirect the output of your MsBuild run to a text file.

You can add the following in your custom target for debugging purposes:

<ItemGroup>
 <GeneratedIncludeFiles Include="Content\documentation\**\*" />
</ItemGroup>
<Message Text="Generated files to include: %(GeneratedIncludeFiles.Identity)"/>

A note about “DestinationRelativePath” – I have seen this specified as “%(RecursiveDir)%(Filename)%(Extension)” instead of using the “CustomFiles” item group, but I had trouble getting this to actually include the custom files.

.Net Web App Configuration with Parameters – Moving beyond web.config files

For any web application, you’ll typically want to configure your application for the different local, Dev, Test and Production environments. And you want all your building and deployment to be handled by your build server, which means anything you do needs to run using MsBuild, Powershell, command-line .exes, or whatever nice wrappers your build server provides.

Web.Config files

When you first start building .Net web applications, you probably use web.configs. And then when you need more than the out-of-the-box “debug” and “release” configurations, you resort to adding more build configurations and web.config transforms. And these are fine when you’re selecting a configuration to build and run locally, or publishing from Visual Studio, but this doesn’t scale beyond a quick experimental project or hackathon to an actual production site.

So if you want “test” and “production” configs, the temptation is to use web.config files and transforms, where local dev uses the “debug” configuration, and everything else uses an optimised “release” configuration. Actually adding another configuration involves first adding a build configuration, then adding the new web.config transforms. If you’ve ever done web publishing through Visual Studio, you’ll probably have spent a lot of time setting up build configurations for each environment (that should be a warning), then creating the web.config transforms so you can use them when publishing.

However, web.configs have a couple of issues:

  1. They hold different configurations for different purposes – your application “build” configuration for debug/release configurations, and your environment “deployment” configuration (local, test, production).
  2. Web.configs are designed to be used in a “configure, build, deploy” process, and for a “build once, deploy often” continuous delivery pipeline, we want to “build, deploy, configure”.

Whether your branching-and-deployment strategy is “single trunk” (build package once, deploy to each environment) or “branch-based” (rebuild from a known code snapshot/revision and configure before each deployment), it pays to start thinking of “build” and “deployment” configurations as separate.

You can cheat (I did at first) and run all your web.config transforms at build-time, bundle them all up into your release package, and ship the correct config at deployment time (by configuring your deployment, or by deploying a generic package followed by a specific config file). My original web.config approach is covered in this epic post.

Web.config files and the Continuous Deployment Pipeline

It makes sense to have developers configure your core application “build” configuration at build time, but you don’t necessarily want to restrict the “business preference” or environment-specific configurations to requiring a complete application rebuild as you make a code change to inject those changes into the start of the continuous deployment pipeline. Having configuration settings that can be changed outside of the commit, build, deploy process is really useful – you should still manage these configuration values in source control or other change management system.

Using SetParameter.xml Files

So instead of putting our environment config values in web.config files, we’re going to put them in a separate file – the SetParameters.xml file. This makes things cleaner, and is also necessary when using certain deployment tools that only work on whole packages not individual files (eg. Azure Powershell cmdlets).

We need a web app that we can build into a deployment package, and somewhere we can deploy to. I’m using a new Azure web app, but parameters files also work with IIS.

Creating the Deployment Package

First we need an application that can be built into a deployable web package – skip this section if you already have a project you want to deploy.

As a demo, I set up a new basic MVC web app (just add a single HomeController and Index view). I also create appSettings entries for “EnvironmentName” and “EnvironmentDescription”, so I have some values to set. I can read these from the ConfigurationManager (eg. ConfigurationManager.AppSettings[“EnvironmentName”]) and display on my view. The plan is that I have different parameter files for my dev, test and live environments, and when I deploy with the appropriate configuration, an on-screen message shows that the appropriate configuration has been applied. Web.config transform files already exist for the debug and release build configurations (we could add more), but we’re not going to use these for configuring our environments.

Now I’m going to publish my site to a deployment package, so I can later have a two-stage build process:

  1. Build into a deployment (zip) package
  2. Deploy to dev/test/live environment and apply appropriate configuration

So in Visual Studio, go to build->publish, select profile type “custom” and name it “DeploymentPackage”. Connection method is “Web Deploy Package”, select a location (eg. “obj/appName”) and site name. . Go to solution explorer, look in yourAppName/Properties/PublishProfiles and see the newly-created publishing profile
(you can now edit the publish profile and change these values if you want).

Build the application with

msbuild appName.csproj /t:Build /p:DeployOnBuild=True;PublishProfile=DeploymentPackage;ProfileTransformWebConfigEnabled=False;Configuration=Release

Or just run the publish from Visual Studio.

This builds your site (in the “Release” configuration), creates your deployment package zip and a couple of extra files – a manifest, .cmd, readm, and a SetParameters.xml file. The parameters file is really basic:

<?xml version="1.0" encoding="utf-8"?>
<parameters>
 <setParameter name="IIS Web Application Name" value="MyWebAppName" />
</parameters>

But we can expand this with our own settings.

Deploying the Web App

OK, so I’m going to deploy this package to an Azure site.

Using Powershell, the syntax is:

Publish-AzureWebsiteProject -Name appname -Package myapp.zip -SetParametersFile appname.SetParameters.xml

For MsDeploy, the syntax looks like this:

msdeploy.exe -source:package="appname.zip"
-dest:auto,ComputerName="https://appname.scm.azurewebsites.net/msdeploy.axd?site=appname",UserName="$user",Password="extractFromPublishProfile",IncludeAcls="False",AuthType="Basic" 
-verb:sync -setParam:name="IIS Web Application Name",value="appname" -disableLink:AppPoolExtension 
-disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose 
-userAgent="VS12.0:PublishDialog:WTE12.3.51016.0" -setParamFile:"appname.SetParameters.xml"

When I run this, I find that this hasn’t actually done anything to override the settings with the values in the SetParameters file, because I didn’t set the parameter value. Also, I have to tell our app which parameters actually exist.

So go back to Visual Studio and add a “parameters.xml” file to the root of your web project. It should look like this:

<?xml version="1.0" encoding="utf-8" ?>
<parameters>
 <parameter name="EnvironmentName"
 description="blah blah blah"
 defaultValue="notsure">
 <parameterEntry kind="XmlFile" scope="Web.config"
 match="/configuration/appSettings/add[@key='EnvironmentName']/@value" />
 </parameter>
</parameters>

This creates a named parameter with a default value, and specifies how this should affect the web.config at deployment time. Rebuild and re-package the application, and the generated “SetParameters.xml” file now has a “setParameter” entry for “EnvironmentName”. Awesome. You can go edit that file, change the value, and run
MsDeploy / Publish-AzureWebsiteProject again to deploy the changes (and only the changes get shipped). This is basically doing a web.config transform at deployment time. And of course, you can expand your parameters.xml file for more settings. Update parameters.xml, rebuild the package, and you’ll see new entries in SetParameters.xml.

If you only want to change a single setting, you can do that with parameters added to your MsDeploy call. In fact, you could have a load of settings in a parameters.xml file, and then apply those as a base while overriding selected settings at deploy time. This might prove useful if you have to deploy multiple copies of your app to production, yet tag each one differently (eg. for A/B testing, “developer” API deployments, etc). Anyway, just add this to the end of your MsDeploy call:

-setParam:SettingName=value

This is obviously a really simple example, just changing appSettings. You can also use parameterization to set connection strings and other configuration settings. This post details the parameters syntax: http://vishaljoshi.blogspot.co.uk/2010/07/web-deploy-parameterization-in-action.html

Configuring for Different Environments

So far, all we have is a single appname.SetParameters.xml file, and it gets generated at build time, so we don’t want to be editing that. What we actually need is a SetParameters.xml file for each environment, and we need to make sure their values survive rebuilding the project. So you can either create additional SetParameters.xml files within your project and include them at build time (basically the equivalent of web.config transforms), or hold your SetParameters.xml files somewhere else, and just consume them at deploy time.

Note that if you do just include the parameters.xml files in your code, it might seem like you’ve gained nothing over hacky web.config transforms – you’re still configuring at commit time and shipping all the possible configurations to use at deployment time. However, you have de-coupled the code configuation (debug/release build config) from the environment configuration, and removed the need to have all those near-duplicate build configurations cluttering up your solution just to enable web.config transforms.

Summary

So now we have a scripted process to

  1. Build our application into a deployment package, including a specification for how the application can be configured
  2. Deploy our application to our chosen environment, applying the necessary configuration at deployment time

This allows you to implement a continuous delivery pipeline with whatever build/deployment server you use, and have “build once, deploy repeatedly” behaviour at each dev/test/live stage of your pipeline.

Gotcha: Don’t Deploy All The Parameters!

You’ll want to have the various parameters available at deployment time – read on for notes on including them with your build artifact so you have a “deployable zip package plus a set of parameter config files” – but you don’t actually want them included inside the “site.zip” package, because then they’ll end up getting deployed, and you don’t want people being able to go to http://www.yourawesomesite.com/parameters.test.xml and seeing all your secrets.

So the answer is to omit them from the build, either using the file properties, to set “Build Action” to be “none” rather than “content”, and “Copy to output directory” to “do not copy” on individual files, or by using some scripting (eg. MsBuild within your publish profile) to decide what ends up in the deployment package.

It’s best not to rely on humans setting the build properties and ensure that your build process omits the parameters (or have a CI/post-build check inspecting the deployment package for contraband).

You can add the following entry to your “DeploymentPackage.pubxml” publish profile to omit them:

<ExcludeFilesFromDeployment>
 parameters*.xml
</ExcludeFilesFromDeployment>

This is used by MsBuild when generating the package:

msbuild WebApp.csproj /p:DeployOnBuild=True;PublishProfile=DeploymentPackage.pubxml

When the deployment package is built this should exclude the matching parameter files.

Summary – what you should have is:

  1. Access to the appropriate “parameters-environment.xml” file at deployment time (either a release artifact that contains your site.zip and parameter files, or a separate repository of configuration files)
  2. The “parameters.xml” template in your “site.zip” deployment package
  3. Absolutely no “parameters-env.xml” files in the actual “site.zip” deployment package (expand the zip archive and navigate down to the “PackageTmp” folder to verify)

Bonus: Shipping Environment Configurations with the Deployment Package

One approach to including SetParameters files within the codebase might look like this.

Add a set of “parameters.dev.xml”, “parameters.test.xml”, “parameters.live.xml” files to your solution, clone them from the generated SetParameters.xml file eg:

<?xml version="1.0" encoding="utf-8"?>
<parameters>
 <setParameter name="IIS Web Application Name" value="MyAppName" />
 <setParameter name="EnvironmentName" value="Test" />
 <setParameter name="EnvironmentDescription" value="Testing" />
</parameters>

Now I need to handle building the package and copying of all the necessary release files into a “BuildArtifacts” folder. This could be achieved in different ways:

  • Set the parameters.*.xml properties to “copy to output directory=copy always”
  • Use an MsBuild or other scripted step to copy them into your final “artifacts” folder

For the MsBuild approach, I have a “WebPackage.msbuild” file in my solution with the following fragment:

<Target Name="BuildPackage">
 <Exec Command='$(MSBuildExe) $(WebApp)\$(WebApp).csproj /t:Build /p:VisualStudioVersion=11.0 
/p:DeployOnBuild=True;PublishProfile="$(PublishProfilePackage)";ProfileTransformWebConfigEnabled=False;Configuration=$(Configuration)'/>
 <CallTarget Targets="_CopyPackageToDeploymentDirectory"/>
 <CallTarget Targets="_CopyParametersToDeploymentDirectory"/>
</Target>

<Target Name="_CopyPackageToDeploymentDirectory">
 <ItemGroup>
 <PackageFiles Include="$(PackageAssemblyDir)\**"/>
 </ItemGroup>
 <Copy SourceFiles="@(PackageFiles)" DestinationFolder="$(BuildArtifactsDir)\$(WebApp)\%(RecursiveDir)"></Copy>
</Target>

<Target Name="_CopyParametersToDeploymentDirectory">
 <ItemGroup>
 <ParameterFiles Include="$(WebApp)\parameters.*.xml"/>
 </ItemGroup>
 <Copy SourceFiles="@(ParameterFiles)" DestinationFolder="$(BuildArtifactsDir)\$(WebApp)"></Copy>
</Target>

This is part of a “WebPackage.msbuild” script within my solution, as long as I have all the properties defined (in a PropertyGroup within the MsBuild script) I can just call

msbuild WebPackage.msbuild /t:BuildPackage

to generate a nice “BuildArtifacts” folder with everything I need to deploy my app.

Azure Web App Deployment With MsDeploy and Powershell

For my first attempt at automating an Azure deployment process, I had some MsBuild wrap the calls to the MsDeploy.exe. This lets me have the build server (eg. Team City) run the MsBuild script and call a simple “DeployX” target for the relevant Dev/Test/Production deployment. The MsBuild script is responsible for supplying MsDeploy with the correct settings. Problem is, the MsDeploy call passes Azure credentials (for the web app, not a user) as clear-text command-line parameters.

If we have those publish settings stored in the MsBuild script, we have to copy those parameter values from the Azure publish profile into a script (the build script) that lives in source control. Not so secure, and it involves some tedious copy-and-pasting.

While your application should know “how to deploy itself”, a big problem with locking your publishing credentials into your deployment packages is what happens if those settings should change. If you have a continuous delivery pipeline (you should), and if code takes a few days to travel through the pipeline from check-in to deployment, any code in the pipeline with outdated publish settings is hard to deploy (you may need to just grab the deployment packages and modify them). It would be better if the credentials lived outside of the build pipeline. You could have your build server pass these in via parameters, but that only works if you’re deploying one or two apps (I’m currently working on a project that deploys a set of eight apps together, whether that’s the correct long-term strategy is another question).

So a first step would be, rather than storing the Azure credentials in a script, just include the publish profile files and read them at deployment time. Then you can move them out of your code (source control and deployment packages) and store them on disk or in a different repository – anywhere that every developer or build server agent machine can access – and read the appropriate profile at deployment time.

You could do this with a custom application, script or MsBuild target that reads and parses the publish profile Xml file. Or just use some Powershell, it has some built-in XML hacking cpabilities. Instead of knowing the publish settings for the MsDeploy call, our MsBuild script can run a Powershell script instead.

Note that this isn’t the only way to deploy with Powershell. In fact, there’s probably better ways to use Powershell to deploy directly as a trusted Azure user, rather than all the hassle of downloading publish profiles. All we’re doing is wrapping the MsDeploy calls that actually ship our code.

A Powershell Script That Deploys Webapps

So create a Powershell script like this (yes the funny quotes to escape characters are weird but necesssary):

#Usage: powershell .\deployWebApp.ps1 [pathToWebAppPackageFolder] [webAppPackageZipFile] [pathToPublishSettingsFile] [configuration]
param([string]$packageFolderPath,[string]$packageZip,[string]$publishProfilePath,[string]$configuration)

#MsDeploy.exe location (if not in path)
[string]$msDeployExe="C:\Program Files (x86)\IIS\Microsoft Web Deploy V3\msdeploy.exe"

#Get publish settings from file
[xml]$xml=Get-Content($publishProfilePath)
[string]$azureSite=$xml.publishData.publishProfile.msDeploySite.get(0)
[string]$azureUrl=$xml.publishData.publishProfile.publishUrl.get(0)
[string]$azureUsername=$xml.publishData.publishProfile.userName.get(0)
[string]$azurePassword=$xml.publishData.publishProfile.userPWD.get(0)

#MsDeploy "white label" webapp package (additional configs to follow)
$msDeployPackageCommand = "`"$msDeployExe`" -source:package=`"$packageFolderPath\$packageZip`" -dest:auto,ComputerName=`"https://$azureUrl/msdeploy.axd?site=$azureSite`",UserName=`"$azureUsername`",Password=`"$azurePassword`",IncludeAcls=`"False`",AuthType=`"Basic`" -verb:sync -setParam:name=`"IIS Web Application Name`",value=`"$azureSite`" -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose -userAgent=`"VS12.0:PublishDialog:WTE12.3.51016.0`"";
Write-Output ("MsDeploy Command: " + $msDeployPackageCommand);
cmd.exe /c "`"$msDeployPackageCommand`"";

#MsDeploy config file "web.configuration.config" eg. "web.TEST.config"
$msDeployConfigCommand = "`"$msDeployExe`" -source:contentPath=`"$packageFolderPath\Configs\Web.$configuration.config`" -dest:contentPath=`"$azureSite\web.config`",ComputerName=`"https://$azureUrl/msdeploy.axd?site=$azureSite`",UserName=`"$azureUsername`",Password=`"$azurePassword`",IncludeAcls=`"False`",AuthType=`"Basic`" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose -userAgent=`"VS12.0:PublishDialog:WTE12.3.51016.0`"";
Write-Output ("MsDeploy Command: " + $msDeployConfigCommand);
cmd.exe /c "`"$msDeployConfigCommand`"";

For the configs, I run web.config transforms on all my configs during the CI/packaging build stages, and ship all the possible configs with the deployment package.

OK, try running the script with:

powershell powershellScripts\deployWebApp.ps1 BuildArtifacts\DeployableWebApp DeployableWebApp.zip C:\Development\AzurePublishProfiles\myappname.azurewebsites.net.PublishSettings Debug

This works, so we go and add this to our MsBuild script with this target:

<Target Name="PsDeployToDev">
 <Exec Command="$(PowerShellExe) powershellScripts\deployWebApp.ps1 $(PackageDeploymentPath)$(WebApp) $(WebApp).zip $(AzurePublishProfilesPath)\myappname.azurewebsites.net.PublishSettings Debug"/>
</Target>

Run it with:

msbuild WebDeploy.msbuild /t:PsDeployToDev

…and it fails, because of an execution policy issue. See http://go.microsoft.com/fwlink/?LinkId=135170

Yes, within this command prompt, I have the rights to run a powershell script, but not to run a script that runs my powershell script. Go Microsoft! Let’s add a few parameters to our MsBuild Powershell call to handle the execution policy:

<Target Name="PsDeployToDev">
 <Exec Command="$(PowerShellExe) -NonInteractive -ExecutionPolicy Unrestricted -File powershellScripts\deployWebApp.ps1 $(PackageDeploymentPath)$(WebApp) $(WebApp).zip $(AzurePublishProfilesPath)\myappname.azurewebsites.net.PublishSettings Debug"/>
</Target>

Success! You can test this by editing, re-building and re-deploying your web app.

I have this running as a Team City build step (MsBuild runner calls the appropriate target in my MsBuild script).

This is a starting point – a basic “ship a white-label package and a config file”, you’ll need to adapt it for deploying your applications.

Some next steps:

  • You’ll probably want to turn off verbose logging and outputting of the actual deployment command (with the publishing credentials arguments) once you’re happy things are working
  • You might need the “msdeploy.exe -useCheckSum” argument to prevent files getting redeployed after your build server does a clean checkout / rebuild

Discovering Web.configs for Deployments with MsBuild Custom Targets

My current job is working on a web application that needs a lot of configuration, and is deployed to multiple environments. We’re using web.config transforms to apply the configurations, but our build pipeline (Team City and MsBuild scripts) builds a deployment “package” as a release candidate, then a subsequent build step deploys (or redeploys) that package to test, staging and production environments (or doesn’t, if we don’t like the release candidate at any point). When we deploy, we ship the site plus one of around 20 transformed web.configs.

So while getting automated deployments up and running with MsDeploy, one of the things I needed to do was create a portable “white label” deployment package, then set the configuration to whatever environment I deployed to. As I’m using web.configs, I need to do the following:

  1. Generate all possible web.configs when I’m building the package to get a set of build artefacts consisting of zipped web application package plus transformed web.configs.
  2. Publish the site, then apply the appropriate web.config (from build artefacts) at deployment time.

So what I’m about to show is half “quick and dirty solution to my deployment problem”, and half “crazy things you can do with MsBuild”.

Publishing The Chosen Config

Using an MsBuild file to wrap my build and publish commands, I have this big chunk of MsDeploy for deploying my web.config to my Azure or IIS site:

<Target Name="_DeployWebConfig">
 <Exec Command='$(MsDeployExe) -source:contentPath="$(WorkingDir)$(PackageDeploymentDir)\Web. 
$(Configuration).config" -dest:contentPath="$(AzureSite)\web.config",ComputerName="https:// 
$(AzureUrl)/msdeploy.axd?site= 
$(AzureSite)",UserName="$(AzureUsername)",Password="$(AzurePassword)",IncludeAcls="False",AuthType="Basic" - 
verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension - 
retryAttempts=2 -verbose -userAgent="VS12.0:PublishDialog:WTE12.3.51016.0"'/>
</Target>

That’s all the MsDeploy for this post. We know we can ship the configs. First we need to get them.

Create All The Configs

My first attempt at transforming all the configs (from web project directory into the deployable package directory) is to include the “TransformXml” task at the top of the build script:

<UsingTask TaskName="TransformXml" AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web 
\Microsoft.Web.Publishing.Tasks.dll"/>

And then have a target I can call to transform the configs.

<Target Name="_TransformWebConfigs1">
 <TransformXml Source="$(WebApp)\Web.config" Transform="$(WebApp)\Web.Debug.config" 
Destination="$(PackageAssemblyDir)\Web.Debug.config" StackTrace="true" />
 <TransformXml Source="$(WebApp)\Web.config" Transform="$(WebApp)\Web.Test.config" 
Destination="$(PackageAssemblyDir)\Web.Test.config" StackTrace="true" />
 <TransformXml Source="$(WebApp)\Web.config" Transform="$(WebApp)\Web.Release.config" 
Destination="$(PackageAssemblyDir)\Web.Release.config" StackTrace="true" />
</Target>

Which is OK if you only have a few configurations. But what if you had to manage several environments? Let’s expand this target:

<Target Name="_TransformWebConfigs2">
 <ItemGroup>
 <Config Include="Debug"/>
 <Config Include="Test"/>
 <Config Include="Release"/>
 </ItemGroup>
 <TransformXml Source="$(WebApp)\Web.config"
 Transform="$(WebApp)\Web.%(Config.Identity).config"
 Destination="$(PackageAssemblyDir)\Web.%(Config.Identity).config"
 StackTrace="true" />
</Target>

Not much better, except now what we’re doing is creating an item group that lists all our configs, then using MsBuild’s “%” operator to iterate though all the items in our item group. Yes, that “TransformXml” call is basically a “foreach” loop.

Find All The Configs

What we really want is to find all the configs automatically. This is where we start to hit the limits of MsBuild (or at least my knowledge, I might have missed something in the reference), because we want to get a flattened directory listing of the configs.

Consider discovering the configs and loading them into an ItemGroup:

<Target Name="_DiscoverConfigs1">
 <!-- Finds all config files, BUT keeps webapp path pre-pended -->
 <ItemGroup>
 <Resources Include="$(WebApp)\Web.*.config"/>
 </ItemGroup>
 <Message Text="[@(Resources)]"/>
 <Message Text="All configs:" />
 <Message Text="[%(Resources.Identity)]"/>
</Target>

What we’d like is an ItemGroup with just “web.Debug.config”, “web.Test.config”, etc. I can’t find an easy way to do this using MsBuild syntax. But hey, MsBuild can run anything. So we could run this through a little script or even a tiny console app. Or just write C# directly in our build file. Yeah, let’s do that

First, add another property. We’re going to be creating our own MsBuild task.

<MSBuildPath Condition=" '$(MSBuildPath)'=='' ">C:\Windows\Microsoft.NET\Framework 
\v4.0.30319</MSBuildPath>

Now, let’s rewrite our discovery target:

<Target Name="_DiscoverConfigs2">
 <!-- Finds all config files, BUT keeps webappPath pre-pended -->
 <ItemGroup>
 <Resources Include="$(WebApp)\Web.*.config"/>
 </ItemGroup>
 <Message Text="[%(Resources.Identity)]"/>
 <_FlattenPathsTakeFilenamesOnly Filepaths="@(Resources)">
 <Output TaskParameter="Filenames" ItemName="WebConfigs"/>
 </_FlattenPathsTakeFilenamesOnly>
 <Message Text="[@(WebConfigs)]"/>
 <Message Text="All configs:" />
 <Message Text="[%(WebConfigs.Identity)]"/>
</Target>

And that “_FlattenPathsTakeFilenamesOnly” task? Let’s create it. Time to write some basic C# directly in our build script.

<!-- Create custom task to get file containing a version string in a directory -->
 <UsingTask TaskName="_FlattenPathsTakeFilenamesOnly" TaskFactory="CodeTaskFactory" 
AssemblyFile="$(MSBuildPath)\Microsoft.Build.Tasks.v4.0.dll">
 <ParameterGroup>
 <Filepaths ParameterType="System.String[]" Required="true"/>
 <Filenames ParameterType="Microsoft.Build.Framework.ITaskItem[]" Output="true"/>
 </ParameterGroup>
 <Task>
 <Using Namespace="System.IO"/>
 <Using Namespace="System.Linq"/>
 <Code Type="Fragment" Language="cs">
 <![CDATA[
 Filenames=new TaskItem[Filepaths.Length];
 for(int i=0;i < Filepaths.Length;i++)
 {
 Filenames[i]=new TaskItem(Filepaths[i].Substring(Filepaths[i].LastIndexOf("\\") + 1));
 //Console.WriteLine("Split " + Filepaths[i] + " > " + Filenames[i]);
 }
 ]]>
 </Code>
 </Task>
</UsingTask>

This is a trivial example, if you were building a custom task that really did something, write and compile it elsewhere (eg. as part of your solution).

So our final config discovery target looks like:

<Target Name="_TransformWebConfigs3">
 <ItemGroup>
 <Resources Include="$(WebApp)\Web.*.config"/>
 </ItemGroup>
 <Message Text="[%(Resources.Identity)]"/>
 <_FlattenPathsTakeFilenamesOnly Filepaths="@(Resources)">
 <Output TaskParameter="Filenames" ItemName="Configs"/>
 </_FlattenPathsTakeFilenamesOnly>
 <TransformXml Source="$(WebApp)\Web.config"
 Transform="$(WebApp)\%(Configs.Identity)"
 Destination="$(PackageAssemblyDir)\%(Configs.Identity)"
 StackTrace="true" />
</Target>

This just trawls through your web app source directory, finds all the source web.configs, runs each in turn through the transformer and dumps into your chosen directory. Which for our requirements, gives us all the possible web.configs ready in case we want to deploy one of them.

Pre-Transforming Web.Configs For Deployment

When your approach to automated deployment is to build a deployment package, then later deploy that to successive test/staging/live environments, you need to manage configurations. Unfortunately, Microsoft’s web.config approach is designed to work only as part of a configure-build-deploy process. For configuring at deployment time, you need a better approach (parameters files). Or a hack… There are better ways to manage configurations, but you might not have the ability to move away from web.configs.

A quick note: I originally included this section as part of a larger post on MsDeploy, but really this is a separate subject.

Transforming Web.Configs at Build Time

I’m going to manually add a target to my MsBuild file to transform my web.configs. I don’t know at build time where my deployment will end up, so I just include all of the possible configs.

We need a solution with an MsBuild file, and the necessary build configurations and web.config transform files setup. When building the deployment package, we also transform all the possible configs (and include in the deployment artifact). When deploying, we MsDeploy the entire site (from zip package), then do an additional MsDeploy of a single web.config file.

Now add a “_TransformWebConfigs” target and a reference to the “TransformXml” task that I’m going to borrow.

Right at the top of your build file, inside the Project element and before the PropertyGroup, add this:

<UsingTask TaskName="TransformXml" AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll"/>

For now I’ll just go process each configuration to create all possible web.configs in our package, transformed and ready to deploy. I’ve named the files “web.configurationName.config”, you could store them in “ConfigurationName\web.config” instead.

<Target Name="_TransformWebConfigs">
<TransformXml Source="$(WebApp)\Web.config" Transform="$(WebApp)\Web.Debug.config" Destination="$(PackageAssemblyDir)\Web.Debug.config" StackTrace="true" />
<TransformXml Source="$(WebApp)\Web.config" Transform="$(WebApp)\Web.Test.config" Destination="$(PackageAssemblyDir)\Web.Test.config" StackTrace="true" />
<TransformXml Source="$(WebApp)\Web.config" Transform="$(WebApp)\Web.Release.config" Destination="$(PackageAssemblyDir)\Web.Release.config" StackTrace="true" />
</Target>

And to call this, extend your “BuildPackage”, between building the package and copying to the deployment directory, add this:

<CallTarget Targets="_TransformWebConfigs" />

Now we add a “_DeployWebConfig” target that applies the correct config based on the $(Configuration) property (call MsBuild script with “/t:DeployPackage /p:Configuration=Test”). This is almost like the previous MsDeploy command, except:

    1. For the source, it really doesn’t like relative paths, so I grab the current working directory
    2. I’m explicitly setting the destination, because we’re renaming the file as we deploy
    3. You don’t set the “IIS Web Application Name”, MsDeploy complains if you do
<Target Name="_DeployWebConfig">
<Exec Command='$(MsDeployExe) -source:contentPath="$(WorkingDir)$(PackageDeploymentDir)\Web.$(Configuration).config" -dest:contentPath="$(AzureSite)\web.config",ComputerName="https://$(AzureUrl)/msdeploy.axd?site=$(AzureSite)",UserName="$(AzureUsername)",Password="$(AzurePassword)",IncludeAcls="False",AuthType="Basic" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose -userAgent="VS12.0:PublishDialog:WTE12.3.51016.0"'/>
</Target>

And you just insert the call to deploy the config at the end of your “DeployPackage” target:

<CallTarget Targets="_DeployWebConfig"/>

Note that your “web.X.config” gets renamed to just “web.config” on the target server during the deployment.

Automating Web Application Deployments to Azure with MsDeploy

We all know that having developers manually deploy code to test or production from their laptops is a really bad idea (wastes time, leads to unrepeatable “mystery” builds, hard to ensure you ship the approved and tested code, no traceability when things go wrong, etc.). So unless you’re willing to endure the three-monthly waterfall pain of spending days building and deploying the one true release version of your code, one of the first things you should do with a new project is set up a continuous integration (CI) and automated deployment system.

Microsoft make some great tools, but sometimes they make them really hard to use. A great example is their application deployment tool “MsDeploy”. Especially when it comes to Azure websites. When you’re manually deploying from Visual Studio then it’s easy, you just tweak the publish profile in a wizard and then hit “publish”, but as soon as you try to script and automate things, you run into all sorts of fun trying to figure out the MsDeploy syntax and error messages.

When my current project kicked off, we made the decision to host with Azure, and then found that setting up automated deployments to Azure is complicated. Because “MsDeploy”. So this is a guide, based on my experiences, on how you might go about automating your deployment process. It’s written with the aim of deploying a .Net MVC web application to Azure, but you can probably adapt for other purposes. I’ll also try and highlight any differences if you’re deploying to IIS instead.

Warning: this is the “works on my machine” version of this guide. I can’t guarantee that you won’t run into problems with your site, but I have successfully repeated this experiment on a couple of applications (including the one I actually get paid to build). So my aim here is not just to give you a bunch of MsDeploy command syntax, but to show how you can start to build and troubleshoot your own deployment process.

The Goal – Automated Build and Deployment

So this was all driven by the needs of our build and deployment process, which we want to be fully automated, and have an initial “continuous integration” build generate a versioned “release candidate” package that we can then use to deploy to successive test, staging and production environments (with the appropriate configuration). We’re agile, and we want to be able to deploy frequently to test and production, so we need an automated process.

In the real world, our product ships as multiple applications, to multiple production environments (different configurations). But we’ll start with the assumption that you just have a single web application to be deployed to “dev”, “test” and “production” environments.

So my inital aim is to eliminate Visual Studio from the process and get everything running on the command line. Those commands can then be scripted or otherwise run by the build server.

The Tools

First you’ll need Visual Studio (I built these examples with Community 2013). Thoughout this tutorial, I’m also going to be using MsBuild and MsDeploy on the command line. Oh, and you’ll need an Azure account (they do free trials). I’ll assume familiarity with VS and Azure, but MsBuild and especially MsDeploy might be new.

I’m not going to cover build servers, but instead just cover the process of escaping from Visual Studio to the command line. You should be able to get any build server to run these commands. And while I’m using MsBuild scripts to simplify some of the commands, you could do this another way. For the actual MsBuild and MsDeploy commands, I’m just going to call the .exes directly from my script, as you would on the command line. (Note that MsBuild has a built-in MsBuild task you can use, there can be complications getting this to use a suitable version of MsBuild for packaging and deploying).

By scripting deployments with MsBuild, I’m achieving a few things:

  1. Hiding command complexity (you tell the script to deploy and it handles all the MsDeploy arguments)
  2. Keeping complexity with the codebase – the build server calls a simple logical operation (“deploy to test”) and the build script (stored in source control with the code it builds) handles the details, making adding applications/artefacts to the build fairly easy
  3. Giving me an easy way to create and debug deployment operations locally without having to edit live on the build server

Finally, any version numbering is between you and your build server, I’m just assuming that we want one so we can track different releases (and track the current test/staging/production back to the code in source control) by setting the version number in AssemblyInfo files (at build time) then displaying somewhere (footer or about/diagnostics pages) as a quick check that the build and deployment actually worked.

The Approach

My basic approach to this has been to:

  1. Take the reference (manual) Visual Studio action, then script it
  2. Wrap commands in an easy-to-use format (MsBuild targets)
  3. Get a command working for hard-coded values before making it configurable
  4. Study errors carefully, check for stupid syntax errors, go slowly, be prepared to re-visit previous steps

This post steps through the process of getting a new web application to the point where it can be assembled into a deployable package, then deploying that package, without requiring Visual Studio. There’s some chunks of sample code (expect about 80 lines of MsBuild) for you to copy-and-paste and customise.

The Manual Publishing Process

If you manually publish to Azure or IIS from Visual Studio, you’re already using MsBuild and MsDeploy. Your process is probably something like:

  1. Create new build configurations, web.config transforms and publish profiles as necessary
  2. Make changes, test locally (uses MsBuild to compile code)
  3. “Publish” your project straight to Azure (builds, assembles what needs to be deployed, then uses MsDeploy to ship code to Azure)

So we want to take this process and automate it with some kind of build script, calling MsDeploy from our fancy new script.

Note: you can also publish your site to a package within Visual Studio and then deploy that later. If you’re already doing that, you’ve probably solved the hard part of deployments (actually deploying things).

A quick introduction to MsBuild and MsDeploy

MsBuild is Microsoft’s build engine. It’s what runs when you compile code in Visual Studio. But you can also run it on the command line, and feed it your own build scripts. In fact, you’re already doing that – a “.csproj” file contains MsBuild commands (XML).

MsDeploy is Microsoft’s deployment engine, and it’s what you’ll use to get your app deployed to Azure. In this scenario, it’s basically doing a “smart file copy” that allows the client (that would be us) to ship files (or at least changes) to the server (Azure), and set a few permissions along the way. MsDeploy can be run two ways – from Visual Studio, or from the command line (note that VS runs its own copy of the MsDeploy engine).

Warning: MsDeploy is painful to work with. Documentation is poor, the error messages are unhelpful, and taking a working set of commands for one solution and transplanting them to another may throw up all sorts of problems.

When you publish a site in Visual Studio, you’re running MsDeploy. You create a “publishing profile” (look in your web app project’s “Properties/PublishProfiles” folder), and then you publish with your chosen profile and configuration. There are basically two ways to publish an application in Visual Studio:

  1. The “Build and deploy” approach – compile the code, configure (web config transforms) and deploy as a single operation.
  2. The “Package” approach – compile the code and zip into a package, to be deployed at some later date.

We’re interested in creating and deploying packages, but we’ll be doing the “build and deploy” as an exercise.

I’m going to try and go through the steps that get you from a manual “build and deploy” to a fully automated system of building a package, then repeatedly deploying it to one environment after another.

Preparation

OK, so we need an application to deploy, an Azure site to deploy to, and an MsBuild script to hold the deployment commands. You can probably skip/skim these next sections.

Create a basic web app solution

You could go straight into automating your current project, but I’m going to use a really simple web app (called “DeployableWebApp”) for this example, you may want to do the same.

So open Visual Studio, and create a new Web Application. It doesn’t matter what you put in here, but I’d recommend you have the following:

  • A message or other output that requires compilation to change (to show it’s a new build)
  • A message on a view that you can change for each deployment (to show that the deployed code has changed)
  • A web.config setting representing the environment that you can display somewhere – we’ll use this to test applying the configuration at deployment time

Create an Azure webapp

Log into your Azure account and go create a new web application. Call it anything you want (mine is called “deployme-dev”). For now we only need one application for “dev”, but later we’ll try deployments to “test” and “production”. You’ll also want to go and download the publish profile file, which contains some settings we’ll steal later.

Create an MsBuild script

You can just put your MsBuild targets and properties into your “.csproj” file, but I prefer to keep my build scripts in separate files. Add a new solution item, an XML file called “WebDeploy.msbuild” (you can call it anything you want, including the file extension). And then create your most basic build script:

<?xml version="1.0" encoding="utf-8"?>
<!-- MsBuild script for using MS Deploy - must be run as admin! -->
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<!-- Add properties to configure stuff -->
<!-- Add targets to do stuff -->
</Project>

For this example, I’m using one MsBuild script for both the building/package and deployment operations. You might want to split these up, as they will ultimately run on different build/deployment processes.
As your MsBuild file grows, you’ll want to organise things. You can expect your build file to contain:

  • References to external tasks
  • Properties (for configuration)
  • Build targets
  • Deployment targets
  • Internal helper targets

Run a test command from an MsBuild target

Add this into the build file, in the “Project” element:

<Target Name="HelloWorld">
<Message Text="I am the thing that deploys other things" />
</Target>

To run this, you need to open a command prompt, navigate to your solution directory, then type:

msbuild WebDeploy.msbuild /t:HelloWorld

This passes your script to MsBuild and calls the “HelloWorld” target. But… unless you have “msbuild.exe” on your path, nothing will happen. So go edit your environment variables.

Or, try this trick – create a custom command prompt text file named “CommandPrompt.bat” in your solution directory, and make it look like this:

CD /D "%~dp0"
@%comspec% /k "SET PATH=C:\Windows\Microsoft.NET\Framework\v4.0.30319;%PATH%"

Tweak the paths, add your own extra tools etc. Now you can fire up your command prompt and msbuild.exe is on the custom path (which is used for commands run in this command prompt only).

Your general MsBuild-running syntax is

msbuild path-to-build-file /t:TargetName /p:Properties

You can specify multiple properties with either “/p:prop1=X /p:prop2=Y” or “/p:prop1=X;prop2=Y”.

Important note: for running MsDeploy you’ll need to be running the command prompt as Administrator.

Manual publishing with “build and deploy”

First we test that a manual publish in Visual Studio works and we can actually deploy to our Azure site.

Select your site and go to “build->Publish”, and create a new profile called “azure-publish-dev”. On the connection tab, set method to “web deploy”, and raid the publish profile to set server to the publish Url (ends “:443”), site name to msdeploySite, username and password to the generated values. For now, you can ignore the “settings” tab, just go and publish. Hopefully this works. Go change something and re-publish to prove it.

So what this actually did was the following:

    1. Built your application (look in the “project/obj/config” folder)
    2. Figured out what needs to be included (look in “project/obj/config/Package” folder, especially the “manifest.xml” file)
    3. Ran MsDeploy to copy the changes

Look in the build output window for more details. Something you’ll notice is a line like this:

Start Web Deploy Publish the Application/package to "https://deployme-dev.scm.azurewebsites.net/msdeploy.axd?site=deployme-dev ..."

And yes, that command is partially-hidden. Welcome to MsDeploy. It can be handy to know what Visual Studio is doing when it publishes. Unfortunately, Visual Studio has its own way in to the MsDeploy engine.

MsDeploy and command discovery

So by default, Visual Studio is accessing the MsDeploy assemblies directly. But you can switch it to use the command-line executable instead. Basically, you just create a “MyWebAppProjectName.wpp.targets” file in your project (it’s an MsBuild file that gets run during the build), and use it to set the “UseMsdeployExe” property to “true”. Best add to your solution so VS finds it. Full details here:

http://blogs.msdn.com/b/webdev/archive/2010/11/03/web-deploy-how-to-see-the-command-executed-in-visual-studio-during-publish.aspx

Anyway, you should now see the full msdeploy.exe call in the build output. That’s going to form the basis of everything else we do. Start by copying what Visual Studio does. Then we can adapt to other deployment scenarios.

You can re-run this command as a test. Either paste straight into a command prompt, or paste into a new text file (call it “deploy.cmd” and run by typing “deploy”). You’ll need to make sure anything with spaces (eg. the msdeploy.exe call) is wrapped in quotes, and you might need to swap some single quotes for double quotes.

A reminder – you’ve just told Visual Studio to use a different method of deploying your project. So if strange things suddenly occur when you publish with Visual Studio, be prepared to revert this change.

Deploying Without Visual Studio

The MsDeploy commands are long and complex, so we’re going to wrap them up in MsBuild targets for simplicity.

First, as we go, I’ll be replacing hard-coded values with properties. So in your MsBuild file, inside the “Project” element, add a PropertyGroup that looks like:

<PropertyGroup>
<Configuration Condition=" '$(Configuration)' == '' ">Release</Configuration>
</PropertyGroup>

This creates a property called “Configuration”, gives it a default value (“Release”), and lets you use it later within a Target by inserting “$(Configuration)” into a command. You can override the default with a “/p:Configuration=Test” argument when running the build. I’ll be using these properties throughout to make the various commands readable and configurable. The full set of properties we need looks like this, either add them all now or as you need them:

<PropertyGroup>
<!-- Exes need full path specified if not already available on %PATH% -->
<MSBuildExe Condition=" '$(MSBuildExe)'=='' ">msbuild.exe</MSBuildExe>
<MSDeployExe Condition=" '$(MSDeployExe)'=='' ">msdeploy.exe</MSDeployExe>
<Configuration Condition=" '$(Configuration)' == '' ">Release</Configuration>
<SolutionName Condition=" '$(SolutionName)' == '' ">DeployableWebApp</SolutionName>
<WebApp Condition=" '$(WebApp)' == '' ">DeployableWebApp</WebApp>
<WorkingDir Condition=" '$(WorkingDir)' == '' ">$(MSBuildThisFileDirectory)</WorkingDir>
<PackageAssemblyDir Condition=" '$(PackageAssemblyDir)' == '' ">DeployableWebApp\ReleasePackage</PackageAssemblyDir>
<PackageDeploymentDir Condition=" '$(PackageDeploymentDir)' == '' ">DeploymentPackage</PackageDeploymentDir>
<PublishProfileDeploy Condition=" '$(PublishProfileDeploy)' == '' ">azure-publish-dev</PublishProfileDeploy>
<PublishProfilePackage Condition=" '$(PublishProfilePackage)' == '' ">azure-package</PublishProfilePackage>

<!-- Azure settings from the .PublishSettings file -->
<AzureUsername Condition=" '$(AzureUsername)' == '' ">$john-doe</AzureUsername>
<AzurePassword Condition=" '$(AzurePassword)' == '' "></AzurePassword>
<AzureSite Condition=" '$(AzureSite)' == '' ">site-name</AzureSite>
<AzureUrl Condition=" '$(AzureUrl)' == '' ">site-name.scm.azurewebsites.net</AzureUrl>
</PropertyGroup>

Something that’s really important when figuring out and testing msdeploy commands is to always start from a “clean” state, so create yourself a “CleanAll” target:

<Target Name="CleanAll">
<PropertyGroup>
<BinDir>$(WebApp)\bin</BinDir>
<ObjDir>$(WebApp)\obj</ObjDir>
</PropertyGroup>
<RemoveDir Directories="$(binDir);$(ObjDir)"/>
<Exec Command="$(MSBuildExe) $(SolutionName).sln /t:clean"/>
</Target>

Run it with “msbuild WebDeploy.msbuild /t:CleanAll” and watch it clean.

If you want to just check your code builds as well, create yourself a “Build” target:

<Target Name="Build">
<Exec Command='$(MSBuildExe) $(SolutionName).sln /t:Build /p:Configuration=$(Configuration)'/>
</Target>

And a quick point about the MsBuild syntax: I’m calling MsBuild from within MsBuild, using the msbuild.exe like you would on the command line. MsBuild also has its own “MsBuild” task syntax, which looks like this:

<MSBuild Projects="$(SolutionName).sln" Targets="Build" Properties="Configuration=$(Configuration)"/>

Use either version, they both call MsBuild with the specified script (a “.sln”, “.csproj”, or your own custom MsBuild file), and run the specified target with the supplied properties.

Building and deploying in a single step

So a quick detour – our goal is to build a package and then deploy that package in a separate step. But first, let’s take the “build and deploy” example above and remove Visual Studio from the process.

We’re going to create a “BuildAndDeploy” target. And to make sure we do a “CleanAll” first, the “DependsOnTargets” lets you specify other targets that must be run first. My initial target looks like this:

<Target Name="BuildAndDeploy" DependsOnTargets="CleanAll">
<Exec Command='$(MSBuildExe) $(WebApp)\$(WebApp).csproj /t:Build /p:VisualStudioVersion=11.0 /p:DeployOnBuild=True /p:PublishProfile="$(PublishProfileDeploy)" /p:Configuration=$(Configuration) /p:Password="$(AzurePassword)"'/>
</Target>

Note: when setting up my publish profile, I checked “save password”. Inspecting the .pubxml shows a “_SavePWD” property but no “Password” property. Checking the generated msdeploy command shows no password, and deployment fails. So add your password to the publish profile (in “Password” property) and don’t override on your MsBuild call, or add a “/p:AzurePassword=letmein” argument to the msbuild call. And yes, having a property called “AzurePassword” that we then assign to “Password” looks strange. I prefixed all the Azure properties with “Azure” for readability. If we really did want to do a single-step “build and deploy”, using the expected “Password” property would make a lot more sense, but the whole point of this post is for those times when you don’t want to “build and deploy”.

You’ll notice that we’re not calling MsDeploy directly, just building the project, but that triggers calling MsDeploy. You’ll also notice from the msdeploy command that the “.SetParameters.xml” file is referenced to supply some of the deployment values. Also, setting the VisualStudioVersion is probably necessary (removing it can cause issues with directory paths).

Building a deployment package

Having to build the code at deployment time isn’t always very useful, because you end up rebuilding the same thing over and over as you promote the same code to different environments (also, it can screw up your versioning strategy). So we already saw how Visual Studio’s publish does a two-step “build/assemble the site, then deploy it”. We’d like to perform these as separate actions:

    1. Build and assemble into a package, environment-neutral (optionally stamped with a version number)
    2. Some time later deploy the package to test, applying “test” configuration
    3. Some time later deploy the package to production, applying “production” configuration

This is important: The package doesn’t know where it will eventually be deployed and configured.

So in Visual Studio, go to publish your site, but this time create a new publish profile of type “Web Deploy Package”. In settings, for the package location enter “ReleasePackage\SiteName.zip” and leave the site name blank. Publish and you get a zip of your site and a few XML files. You’ll also get a “.deploy.cmd” file, which contains everything you need for a one-click deploy. Except, it’s missing a few necessary parameters and values that you need to figure out. Thanks, Microsoft. Actually, all the deployment “.cmd” gives you is the MsDeploy command we’re about to build.

Time to recreate this in our script, so back to the MsBuild, and add some new targets:

<Target Name="BuildPackage" DependsOnTargets="CleanAll">
<Exec Command='$(MSBuildExe) $(WebApp)\$(WebApp).csproj /t:Build /p:VisualStudioVersion=11.0 /p:DeployOnBuild=True;PublishProfile="$(PublishProfilePackage)";ProfileTransformWebConfigEnabled=False;Configuration=$(Configuration)'/>
<CallTarget Targets="_CopyPackageToDeploymentDirectory"/>
</Target>

<Target Name="_CopyPackageToDeploymentDirectory">
<ItemGroup>
<PackageFiles Include="$(PackageAssemblyDir)\**"/>
</ItemGroup>
<Copy SourceFiles="@(PackageFiles)" DestinationFolder="$(PackageDeploymentDir)\%(RecursiveDir)"></Copy>
</Target>

So this should build a big ZIP archive of your site, with a few other generated “.cmd” and “.xml” files.

One of the things I’ve added here is the copying of the package from the original “assembly” directory to somewhere else. In real world scenarios, your build will end up generating a package that you then stash somewhere (eg. a “build artefacts” folder) then copy to whatever workspace directory your deployment process runs in for the publishing. So to avoid any reliance on anything other than what’s in the package, you can build the package, move it somewhere for deployment, then clean your workspace.

Note: The MsBuild call for building the package is a little strange – you “build” the project but specify that you want a package (deploy with the publish profile, which creates a package). There’s also a “Package” target, which skips building and compiles a .zip package, but may or may not contain everything. Publishing to a package from Visual Studio gives you a reference zip file.

Deploying the package

Right. Scary experiment time! Add the deployment target:

<Target Name="DeployPackage">
<Exec Command='$(MsDeployExe) -source:package="$(PackageDeploymentDir)\$(WebApp).zip" -dest:auto,ComputerName="https://$(AzureUrl)/msdeploy.axd?site=$(AzureSite)",UserName="$(AzureUsername)",Password="$(AzurePassword)",IncludeAcls="False",AuthType="Basic" -verb:sync -setParam:name="IIS Web Application Name",value="$(AzureSite)" -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose -userAgent="VS12.0:PublishDialog:WTE12.3.51016.0" -whatif'/>
</Target>

This is the MsDeploy call we stole from the Visual Studio publish output, tweaked with some readable/configurable parameter values. If msdeploy.exe isn’t already on your path, go to the PropertyGroup and set the value of $(MsDeployExe) to “C:\Program Files (x86)\IIS\Microsoft Web Deploy V3\msdeploy.exe” (including the quotes).

OK, try running this:

msbuild WebDeploy.msbuild /t:DeployPackage /p:AzureUsername="U";AzurePassword="P"

(If you haven’t already set the Azure username/password in your build file, make sure you pass the correct values in).

You should see a big long msbuild.exe call in the output, assuming there’s no missing quotes or other typos.

This is the point where it might all fail and spew red text and error messages, and you have to go check the MsDeploy syntax and figure out which parameters to set and what values to use. It’s essential to eliminate anything relating to the “build” environment where the package was created, so ensure no reference to hard-coded build location or deployment Urls in the “package.pubxml” and no use of a generated “parameters.xml” file (a source of many problems).

We haven’t actually published any changes, yet. See the final “-whatif” parameter? That runs MsDeploy in a demo mode where it doesn’t actually change anything (but will complain at the slightest syntax error or missing/incorrect parameter name/value). So remove the “-whatif” and try a real deployment.

Deploying Only What Has Changed

Usually, you only want to deploy the changes. After all, if your application consists of thousands of files, of which only 20 have actually changed in a given build, you want the process to be fast. MsDeploy will deploy only the changes, but by default this is based on file dates. If you do a clean checkout and full rebuild, your “unchanged” files may have a more recent date than their identical counterparts on the server. To prevent redeploying everything, add the “-useCheckSum” argument to your MsDeploy call:

msdeploy.exe -usechecksum

You’ll get a summary of the changes actually shipped when you run MsDeploy, so you can verify if it’s copying too many files.

Bonus: Deploying to IIS

So on my way to figuring out getting MsDeploy to work with Azure, I started with the most basic publish to a local IIS site. If you want to adapt this experiment for IIS instead of Azure, you’ll need the MsDeploy service running on your server.

My original command went something like this (site is named “DeployableWebApp” in IIS):

<Exec Command='$(MsDeployExe) -source:package="fullpathtopackage.zip" -dest:auto,includeAcls="False" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -setParamFile:"fullpathto.SetParameters.xml"' />
<Exec Command='$(MsDeployExe) -verb:sync -source:contentPath="fullpathtopackage\Web.$(Configuration).config" -dest:contentPath="DeployableWebApp\web.config",includeAcls="False" -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension' />

Note the use of the “SetParameters.xml” file. It didn’t “just work” when I added the step to move the package from an assembly to a deployment directory before deploying. It even complained about the app pool settings (which was probably actually more to do with not finding the dest site and assuming defaults). So let’s get rid of that hassle:

<Exec Command='$(MsDeployExe) -source:package="$(PackageDeploymentDir)\$(WebApp).zip" -dest:auto,includeAcls="False" -verb:sync -setParam:name="IIS Web Application Name",value="DeployableWebApp" -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose -userAgent="VS12.0:PublishDialog:WTE12.3.51016.0"' />
<Exec Command='$(MsDeployExe) -source:contentPath="$(WorkingDir)$(PackageDeploymentDir)\Web.$(Configuration).config" -dest:contentPath="DeployableWebApp\web.config",includeAcls="False" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose -userAgent="VS12.0:PublishDialog:WTE12.3.51016.0"' />

Again, the “package” and “selected file” deployments are different. “IIS Web Application Name” (your destination configuration) is not an option when the source is “contentPath”.

Build Once, Deploy Often

If this is somehow working, what you have is pretty awesome – you can call “BuildPackage” then “DeployPackage” for a two-step build-and-deploy process. But we’ve only deployed to a single location. And you only get the default configuration – add a “/p:Configuration=Test” onto the end of your deployment command and nothing happens.

Config transforms and parameters

Managing configuration options, especially when you don’t know at build time which environment(s) your build will get deployed to, is a topic for another post.

However, as a bonus step, it’s worth taking a quick detour into web.configs to illustrate the subtle ways in which MsDeploy can treat shipping deployment packages and individual files differently.

Bonus – Transforming and deploying a web.config

Web.configs aren’t really designed for use in a two-stage “build/package then deploy” process, but you can cheat for now. See this post on Pre-Transforming Web.Configs For Deployment.

Build once, deploy everywhere

What about taking our “dev” package and deploying it to “test”? We need to check that nothing in our generic package is tied to our current dev environment, because we want to deploy the package to “dev”, then “test”, then “production”. We can test this easily:

    1. Go back to your Azure dashboard and create another two web apps for “deployme-test” and “deployme-production”
    2. Grab the publish profile files
    3. Publish!

Now you could go through all the previous steps of creating a publishing profile, testing the build-and-deploy in Visual Studio then the command line, and then trying to publish the package. Or… you could just trust that your “DeployPackage” target actually uses the property values you supply. So as a quick test, you can edit the hard-coded values for your Azure site and credentials, or just skip the properties and create multiple “DeployPackageDev”, DeployPackageTest” etc. targets.

But really, you want to pass these values in on the command-line with the “/p:Param=Value” syntax, and specify your “AzureUrl”, “AzureSite”, “AzureUsername” and “AzurePassword”.

Remember, you have to run “BuildPackage” to get your changes into the package, but then you can deploy that to wherever you want. And after you’ve built the package, you can clean your solution, including the package assembly directory (the package has been copied elsewhere) and deploy that package again and again.

This is where having web.config transforms set up is awesome – you can transform all the configs, bundle them into the package, then have the deployment set whatever configuration works for your target environment.

So you could do

  • “BuildPackage”, “DeployPackage” (with dev settings and config), “DeployPackage” (with test settings and config)
  • “BuildPackage”, “CleanAll”, “DeployPackage” (to dev), “DeployPackage” (to test), etc

One final thing – you may still have your Azure credentials hidden in your build file, which is going to end up in source control with the rest of your project, and maybe even with the package. So you might want to remove these (and pass them in via parameter when you run the deployment script).

Shipping the build script

If you actually want to be deploying these packages you create (eg. from your build server), you’ll need access to the build script. Which probably means shipping the build script with the package (so the correct version of the script is used to create and then ship the package). You can do this with some simple file copying – just create yourself an MsBuild target to have the MsBuild script include itself alongside the package.

Summary

The things to look out for

Getting to this point took a lot of trial and error and searching and wondering if it was even possible to drive MsDeploy outside of Visual Studio. The main things that caused me problems:

  • Building the package – beware the project “Package” target, make sure you build and zip up everything (or your deployed app may complain about missing assemblies). Also, the permissions may differ between Visual Studio-generated packages and those your build script spits out (file sizes differ by a few bytes), shouldn’t actually cause any issues.
  • Setting the VisualStudioVersion on build package calls – fail to enforce this and strange things happen, like your published package not ending up in the expected folder, which can then kill subsequent actions (like those web.config transforms). The “DesktopBuildPackageLocation” path in your publish profile may be ignored in favour of just using the “app/obj/Release/Package” directory for assembling the release package.
  • The MsBuild.exe used on the local path may not be the same MsBuild used by your build server, build packaging behaviour differs between versions.
  • Reading the error messages – first make sure you know whether the error is just a typo with your MsBuild/MsDeploy syntax (a misplaced quote), a missing file, or an actual MsDeploy issue.
  • Parameters XML files – caused me endless pain, setting values based on local settings, refusing to let me override values for the remote server, erroneously complaining about ACL errors, AppPool settings (when deploying to IIS), these seem to be a real problem if you don’t publish from the directory you originally assembled the package in. I basically just tried to get rid of these and specify as arguments to my MsDeploy command instead. If you can see everything in one big long MsDeploy command, you might spot issues quicker.
  • Absolute paths and local workspace build paths getting passed to MsDeploy (check those parameters files, and your publish profile). When this ends up on a build server, not only will you be deploying from a different location to where you built your package, but you won’t even be building the package in the same location you did on your local workspace.
  • Relative paths and contentPath – when deploying files (source is “contentPath”) rather than a package, MsDeploy needs a full path to the file, relative paths give you “site does not exist” errors because your source files can’t be found.
  • Viewing a working MsDeploy command – capture the working Visual Studio version, use as a reference, dump the output of your command-line deployment run to a text file and inspect for differences.
  • The MsDeploy “source”/”dest” arguments – take care that you set source to “package” for your zipped package, “contentPath” for individual files, and match with a “dest” of “auto” or “contentPath” (especially if source and dest names change).
  • “IIS Web Application Name” is only applicable in certain deployment cases.
  • The package is not your average “zip” file – if you were to open up the generated .zip package, amend its contents, then zip it up again, you might find deploying it as a package won’t work (complaints about “IIS Web Application Name”, because MsDeploy doesn’t recognise it as a package). Just because a file is called “*.zip”, doesn’t mean it is one. I saw this once when cheating the scripted build process, and I think it’s a compression issue. In these examples, I’m deploying the generated package, then the configs. But you might want to be sneaky and hack the package you’re about to deploy so you can do something clever, like having automatic deployment to a “staging” slot that swaps that to “live” only if the single deployment step is successful.
  • Deploying all the files all the time – add the “useCheckSum” argument to prevent this.

What we have achieved

All this changing the site, doing a “BuildPackage” then a “DeployPackage” is fun, but what have we actually achieved? We should now have the following at our disposal:

  1. A strategy for mimicking Visual Studio’s “build and deploy” process (useful for getting a reference deploy command)
  2. A process for building our web app into an independent deployable package (without using Visual Studio)
  3. A process for deploying our generic web application package to a destination of our choice (without using Visual Studio), we don’t even have to choose between IIS and Azure until we deploy
  4. A process for applying a chosen web.config at deployment time (without using Visual Studio)

And once you replace Visual Studio with something that runs on the command line, suddenly all sorts of automated build and deployment scenarios open up. Imagine a scenario where your build and deployment pipeline is:

  1. Continuous integration builds a release candidate package on every build (and stamps a version number on it)
  2. Package is later deployed to test environment
  3. Later, the same package (that’s the tested code, not the current “head” of your development branch) is deployed to staging, production, etc.

There are a few things missing to get there, and I’ll cover these in other posts:

  1. Configuration – how web.config transforms, and an alternative “deployment parameters” approach, lets you tweak the configuration at deployment time (including shipping to environments that didn’t exist at build time)
  2. Managing the chaos of the MsBuild file when you want generic build and deploy targets to serve multiple apps and environments
  3. Adding these build steps to a build server for true automated deployment
  4. Adding database migrations to the process
  5. Adding other non-web-config-transform configurations (eg. theme assets) to your deployments