Including Additional / Generated Files When Publishing a .Net Web Application

When you publish a .Net web app, you typically set up a publish profile, either to do an immediate build-and-publish, or to publish to a package for later deployment (eg. single-trunk / build-once-deploy-many-times scenarios).

So for building a package, the MsBuild syntax would look like:

msbuild WebApp\WebApp.csproj /t:Build /p:VisualStudioVersion=11.0;DeployOnBuild=True;PublishProfile="DeploymentPackage.pubxml";ProfileTransformWebConfigEnabled=False;Configuration=Release;

You just use MsBuild and ask your project to build itself. The good bit is that your project knows what it contains, and therefore what needs deployed. The bad bit is that your project knows what it contains, and therefore what needs deployed… Any files generated by an external tool, or any other files you didn’t explicitly tell Visual Studio (listed as content in the .csproj file) are excluded. Thankfully, you can include extra stuff into your deployment package with a few lines of MsBuild commands. While you can probably hack the .csproj, the easiest way I’ve found is just to plug some MsBuild directly into the publish profile.

This location does make sense – you’re specifying what gets published after all. Note that if you have several publish profiles (maybe you have branch-based deployment and per-environment build-and-publish) then things get more complicated. I’m going to stick with the simple case – a single publish profile for creating a generic deployment package that we can then deploy (with appropriate configuration) to any target environment.

I recently had to solve this problem for handling some generated CSS, but you can use it for any non-solution files you want to include as part of the web application deployment package zip.

A quick warning – this does involve editing a generated file (the .pubxml file that you probably only create once).

Tweaking The Publish Profile

So assume I have some tools generating CSS (in “/webproj/content/styles”) and extra text files (in “/webproj/content/documentation”). I don’t even need to set up tools to generate these files – I can actually just simulate by manually adding some dummy files. I’m assuming that the files get generated within the project directory (because you want to deploy them to IIS or Azure when you deploy your site).

Anyway, create the extra files within the project directory, don’t tell Visual Studio or your project about them, and then build the deployment package. You’ll see they get omitted. We can fix that.

Go edit your publish profile, and add these lines inside the “PropertyGroup” element – we need to add a “CopyAllFilesToSingleFolderForMsdeployDependsOn” property.

 <CopyAllFilesToSingleFolderForMsdeployDependsOn>
 IncludeCustomFilesInPackage;
 $(CopyAllFilesToSingleFolderForMsdeployDependsOn);
 </CopyAllFilesToSingleFolderForMsdeployDependsOn>
 </PropertyGroup>

This goes off and calls a target called “IncludeCustomFilesInPackage” that we’ll create in a minute. The name of the “CopyAllFilesToSingleFolderForMsdeployDependsOn” property is important (MsBuild will look for it), the name of the custom “IncludeCustomFilesInPackage” target can be whatever we want (change the names and see what happens).

There is also a “CopyAllFilesToSingleFolderForPackageDependsOn” property, but this seems to get ignored even when building in package mode. For completeness, it would look like this:

<CopyAllFilesToSingleFolderForPackageDependsOn>
 IncludeCustomFilesInPackage;
 $(CopyAllFilesToSingleFolderForPackageDependsOn);
</CopyAllFilesToSingleFolderForPackageDependsOn>

But you don’t seem to need it – just add the “CopyAllFilesToSingleFolderForMsdeployDependsOn”.

Now we can go define our “IncludeCustomFilesInPackage” target, so add this to the .pubxml file, inside the “Project” element:

<Target Name="IncludeCustomFilesInPackage">
 <Message Text="Collecting custom files..."/>
 <ItemGroup>
  <CustomFiles Include="Content\styles\**\*" />
  <CustomFiles Include="Content\documentation\**\*" />
  <FilesForPackagingFromProject Include="%(CustomFiles.Identity)">
   <DestinationRelativePath>%(CustomFiles.Identity)</DestinationRelativePath>
  </FilesForPackagingFromProject>
 </ItemGroup>
 <Message Text="Add to package %(FilesForPackagingFromProject.Identity)"/>
</Target>

Add as many entries as you need into the ItemGroup. I added a couple of “message” calls to output progress.

So when you run your package build and inspect the final .zip package that gets created (check in the “obj” directory eg. “projectName/obj/projectName/projectName.zip”) then it should contain all those extra files that your .csproj didn’t know about.

Troubleshooting

When you’re setting this up, you might run into problems getting the paths correct. You can always add in a load of Message calls, and redirect the output of your MsBuild run to a text file.

You can add the following in your custom target for debugging purposes:

<ItemGroup>
 <GeneratedIncludeFiles Include="Content\documentation\**\*" />
</ItemGroup>
<Message Text="Generated files to include: %(GeneratedIncludeFiles.Identity)"/>

A note about “DestinationRelativePath” – I have seen this specified as “%(RecursiveDir)%(Filename)%(Extension)” instead of using the “CustomFiles” item group, but I had trouble getting this to actually include the custom files.

Advertisements

.Net Web App Configuration with Parameters – Moving beyond web.config files

For any web application, you’ll typically want to configure your application for the different local, Dev, Test and Production environments. And you want all your building and deployment to be handled by your build server, which means anything you do needs to run using MsBuild, Powershell, command-line .exes, or whatever nice wrappers your build server provides.

Web.Config files

When you first start building .Net web applications, you probably use web.configs. And then when you need more than the out-of-the-box “debug” and “release” configurations, you resort to adding more build configurations and web.config transforms. And these are fine when you’re selecting a configuration to build and run locally, or publishing from Visual Studio, but this doesn’t scale beyond a quick experimental project or hackathon to an actual production site.

So if you want “test” and “production” configs, the temptation is to use web.config files and transforms, where local dev uses the “debug” configuration, and everything else uses an optimised “release” configuration. Actually adding another configuration involves first adding a build configuration, then adding the new web.config transforms. If you’ve ever done web publishing through Visual Studio, you’ll probably have spent a lot of time setting up build configurations for each environment (that should be a warning), then creating the web.config transforms so you can use them when publishing.

However, web.configs have a couple of issues:

  1. They hold different configurations for different purposes – your application “build” configuration for debug/release configurations, and your environment “deployment” configuration (local, test, production).
  2. Web.configs are designed to be used in a “configure, build, deploy” process, and for a “build once, deploy often” continuous delivery pipeline, we want to “build, deploy, configure”.

Whether your branching-and-deployment strategy is “single trunk” (build package once, deploy to each environment) or “branch-based” (rebuild from a known code snapshot/revision and configure before each deployment), it pays to start thinking of “build” and “deployment” configurations as separate.

You can cheat (I did at first) and run all your web.config transforms at build-time, bundle them all up into your release package, and ship the correct config at deployment time (by configuring your deployment, or by deploying a generic package followed by a specific config file). My original web.config approach is covered in this epic post.

Web.config files and the Continuous Deployment Pipeline

It makes sense to have developers configure your core application “build” configuration at build time, but you don’t necessarily want to restrict the “business preference” or environment-specific configurations to requiring a complete application rebuild as you make a code change to inject those changes into the start of the continuous deployment pipeline. Having configuration settings that can be changed outside of the commit, build, deploy process is really useful – you should still manage these configuration values in source control or other change management system.

Using SetParameter.xml Files

So instead of putting our environment config values in web.config files, we’re going to put them in a separate file – the SetParameters.xml file. This makes things cleaner, and is also necessary when using certain deployment tools that only work on whole packages not individual files (eg. Azure Powershell cmdlets).

We need a web app that we can build into a deployment package, and somewhere we can deploy to. I’m using a new Azure web app, but parameters files also work with IIS.

Creating the Deployment Package

First we need an application that can be built into a deployable web package – skip this section if you already have a project you want to deploy.

As a demo, I set up a new basic MVC web app (just add a single HomeController and Index view). I also create appSettings entries for “EnvironmentName” and “EnvironmentDescription”, so I have some values to set. I can read these from the ConfigurationManager (eg. ConfigurationManager.AppSettings[“EnvironmentName”]) and display on my view. The plan is that I have different parameter files for my dev, test and live environments, and when I deploy with the appropriate configuration, an on-screen message shows that the appropriate configuration has been applied. Web.config transform files already exist for the debug and release build configurations (we could add more), but we’re not going to use these for configuring our environments.

Now I’m going to publish my site to a deployment package, so I can later have a two-stage build process:

  1. Build into a deployment (zip) package
  2. Deploy to dev/test/live environment and apply appropriate configuration

So in Visual Studio, go to build->publish, select profile type “custom” and name it “DeploymentPackage”. Connection method is “Web Deploy Package”, select a location (eg. “obj/appName”) and site name. . Go to solution explorer, look in yourAppName/Properties/PublishProfiles and see the newly-created publishing profile
(you can now edit the publish profile and change these values if you want).

Build the application with

msbuild appName.csproj /t:Build /p:DeployOnBuild=True;PublishProfile=DeploymentPackage;ProfileTransformWebConfigEnabled=False;Configuration=Release

Or just run the publish from Visual Studio.

This builds your site (in the “Release” configuration), creates your deployment package zip and a couple of extra files – a manifest, .cmd, readm, and a SetParameters.xml file. The parameters file is really basic:

<?xml version="1.0" encoding="utf-8"?>
<parameters>
 <setParameter name="IIS Web Application Name" value="MyWebAppName" />
</parameters>

But we can expand this with our own settings.

Deploying the Web App

OK, so I’m going to deploy this package to an Azure site.

Using Powershell, the syntax is:

Publish-AzureWebsiteProject -Name appname -Package myapp.zip -SetParametersFile appname.SetParameters.xml

For MsDeploy, the syntax looks like this:

msdeploy.exe -source:package="appname.zip"
-dest:auto,ComputerName="https://appname.scm.azurewebsites.net/msdeploy.axd?site=appname",UserName="$user",Password="extractFromPublishProfile",IncludeAcls="False",AuthType="Basic" 
-verb:sync -setParam:name="IIS Web Application Name",value="appname" -disableLink:AppPoolExtension 
-disableLink:ContentExtension -disableLink:CertificateExtension -retryAttempts=2 -verbose 
-userAgent="VS12.0:PublishDialog:WTE12.3.51016.0" -setParamFile:"appname.SetParameters.xml"

When I run this, I find that this hasn’t actually done anything to override the settings with the values in the SetParameters file, because I didn’t set the parameter value. Also, I have to tell our app which parameters actually exist.

So go back to Visual Studio and add a “parameters.xml” file to the root of your web project. It should look like this:

<?xml version="1.0" encoding="utf-8" ?>
<parameters>
 <parameter name="EnvironmentName"
 description="blah blah blah"
 defaultValue="notsure">
 <parameterEntry kind="XmlFile" scope="Web.config"
 match="/configuration/appSettings/add[@key='EnvironmentName']/@value" />
 </parameter>
</parameters>

This creates a named parameter with a default value, and specifies how this should affect the web.config at deployment time. Rebuild and re-package the application, and the generated “SetParameters.xml” file now has a “setParameter” entry for “EnvironmentName”. Awesome. You can go edit that file, change the value, and run
MsDeploy / Publish-AzureWebsiteProject again to deploy the changes (and only the changes get shipped). This is basically doing a web.config transform at deployment time. And of course, you can expand your parameters.xml file for more settings. Update parameters.xml, rebuild the package, and you’ll see new entries in SetParameters.xml.

If you only want to change a single setting, you can do that with parameters added to your MsDeploy call. In fact, you could have a load of settings in a parameters.xml file, and then apply those as a base while overriding selected settings at deploy time. This might prove useful if you have to deploy multiple copies of your app to production, yet tag each one differently (eg. for A/B testing, “developer” API deployments, etc). Anyway, just add this to the end of your MsDeploy call:

-setParam:SettingName=value

This is obviously a really simple example, just changing appSettings. You can also use parameterization to set connection strings and other configuration settings. This post details the parameters syntax: http://vishaljoshi.blogspot.co.uk/2010/07/web-deploy-parameterization-in-action.html

Configuring for Different Environments

So far, all we have is a single appname.SetParameters.xml file, and it gets generated at build time, so we don’t want to be editing that. What we actually need is a SetParameters.xml file for each environment, and we need to make sure their values survive rebuilding the project. So you can either create additional SetParameters.xml files within your project and include them at build time (basically the equivalent of web.config transforms), or hold your SetParameters.xml files somewhere else, and just consume them at deploy time.

Note that if you do just include the parameters.xml files in your code, it might seem like you’ve gained nothing over hacky web.config transforms – you’re still configuring at commit time and shipping all the possible configurations to use at deployment time. However, you have de-coupled the code configuation (debug/release build config) from the environment configuration, and removed the need to have all those near-duplicate build configurations cluttering up your solution just to enable web.config transforms.

Summary

So now we have a scripted process to

  1. Build our application into a deployment package, including a specification for how the application can be configured
  2. Deploy our application to our chosen environment, applying the necessary configuration at deployment time

This allows you to implement a continuous delivery pipeline with whatever build/deployment server you use, and have “build once, deploy repeatedly” behaviour at each dev/test/live stage of your pipeline.

Gotcha: Don’t Deploy All The Parameters!

You’ll want to have the various parameters available at deployment time – read on for notes on including them with your build artifact so you have a “deployable zip package plus a set of parameter config files” – but you don’t actually want them included inside the “site.zip” package, because then they’ll end up getting deployed, and you don’t want people being able to go to http://www.yourawesomesite.com/parameters.test.xml and seeing all your secrets.

So the answer is to omit them from the build, either using the file properties, to set “Build Action” to be “none” rather than “content”, and “Copy to output directory” to “do not copy” on individual files, or by using some scripting (eg. MsBuild within your publish profile) to decide what ends up in the deployment package.

It’s best not to rely on humans setting the build properties and ensure that your build process omits the parameters (or have a CI/post-build check inspecting the deployment package for contraband).

You can add the following entry to your “DeploymentPackage.pubxml” publish profile to omit them:

<ExcludeFilesFromDeployment>
 parameters*.xml
</ExcludeFilesFromDeployment>

This is used by MsBuild when generating the package:

msbuild WebApp.csproj /p:DeployOnBuild=True;PublishProfile=DeploymentPackage.pubxml

When the deployment package is built this should exclude the matching parameter files.

Summary – what you should have is:

  1. Access to the appropriate “parameters-environment.xml” file at deployment time (either a release artifact that contains your site.zip and parameter files, or a separate repository of configuration files)
  2. The “parameters.xml” template in your “site.zip” deployment package
  3. Absolutely no “parameters-env.xml” files in the actual “site.zip” deployment package (expand the zip archive and navigate down to the “PackageTmp” folder to verify)

Bonus: Shipping Environment Configurations with the Deployment Package

One approach to including SetParameters files within the codebase might look like this.

Add a set of “parameters.dev.xml”, “parameters.test.xml”, “parameters.live.xml” files to your solution, clone them from the generated SetParameters.xml file eg:

<?xml version="1.0" encoding="utf-8"?>
<parameters>
 <setParameter name="IIS Web Application Name" value="MyAppName" />
 <setParameter name="EnvironmentName" value="Test" />
 <setParameter name="EnvironmentDescription" value="Testing" />
</parameters>

Now I need to handle building the package and copying of all the necessary release files into a “BuildArtifacts” folder. This could be achieved in different ways:

  • Set the parameters.*.xml properties to “copy to output directory=copy always”
  • Use an MsBuild or other scripted step to copy them into your final “artifacts” folder

For the MsBuild approach, I have a “WebPackage.msbuild” file in my solution with the following fragment:

<Target Name="BuildPackage">
 <Exec Command='$(MSBuildExe) $(WebApp)\$(WebApp).csproj /t:Build /p:VisualStudioVersion=11.0 
/p:DeployOnBuild=True;PublishProfile="$(PublishProfilePackage)";ProfileTransformWebConfigEnabled=False;Configuration=$(Configuration)'/>
 <CallTarget Targets="_CopyPackageToDeploymentDirectory"/>
 <CallTarget Targets="_CopyParametersToDeploymentDirectory"/>
</Target>

<Target Name="_CopyPackageToDeploymentDirectory">
 <ItemGroup>
 <PackageFiles Include="$(PackageAssemblyDir)\**"/>
 </ItemGroup>
 <Copy SourceFiles="@(PackageFiles)" DestinationFolder="$(BuildArtifactsDir)\$(WebApp)\%(RecursiveDir)"></Copy>
</Target>

<Target Name="_CopyParametersToDeploymentDirectory">
 <ItemGroup>
 <ParameterFiles Include="$(WebApp)\parameters.*.xml"/>
 </ItemGroup>
 <Copy SourceFiles="@(ParameterFiles)" DestinationFolder="$(BuildArtifactsDir)\$(WebApp)"></Copy>
</Target>

This is part of a “WebPackage.msbuild” script within my solution, as long as I have all the properties defined (in a PropertyGroup within the MsBuild script) I can just call

msbuild WebPackage.msbuild /t:BuildPackage

to generate a nice “BuildArtifacts” folder with everything I need to deploy my app.