Overview
This documents the development and deployment processes to be used by developers, engineers and architects at Quandis.
Quandis uses the following tools for development and deployment:
Visual Studio
C# class libraries for application-tier projects
ASP.NET web projects for web-tier projects
SQL Projects for database components
SVN, Ankh and Tortoise for source control
MSBuild to create deployment files
Powershell to coordinate the deployment of build files across multiple machine
Source Control
Quandis uses a single SVN repository for all source-controlled projects and solutions. For QBO3, the structure is:
http://source.quandis.com
Branches
qbo.3
{YYYYMM}
Tags
Trunk
qbo.3
Publish: contains powershell scripts used for deployments
qbo.Core
Application Tier: core C# libraries
Plugins: C# libraries dependent on third-party products, like Amazon, Google, Crystal Reports, Aspose, etc.
Themes: client or product specific overrides to core QBO3 components
Web Tier: core ASP.NET web projects
qbo.DB
Standard: core QBO3 database components (tables, user-defined functions, etc.)
Visual Studio Solutions
Core Solution
For QBO3-based core projects, there is a qbo.Core.sln solution file found at:
Source > Trunk > qbo.3 > qbo.Core > qbo.Core.sln
All changes that apply to core QBO3 code should be made in this solution.
When you are stepping through QBO3 code to learn or debug it, you should do so from this solution, as it will contain ALL of the QBO projects that you might need to debug from a core QBO3 perspective.
Plugin Solutions
For QBO3-based plugin solutions, the source folder structure should be:
Source > Trunk > qbo.3 > qbo.Core > Plugins > {Third party product}
Dependencies: folder containing any DLLs that need to be referenced by projects in the solution
qbo.{Module}.{Third Party Product}: folder containing plugin project
qbo.{Third Party Product}.sln
The solution file should mirror the folder structure:
qbo.{Third Party Product}.sln
Dependencies: solution folder containing third-party DLLs that need to be referenced
qbo: solution folder containing core QBO3 application-tier projects that are referenced by the plugin projects
qbo.{Core Module}.{Third Party Product}: project file that is the plugin
all such projects should be based on the ASP.NET Web Project template, so that they can be deployed using MSBuild and WebDeploy
The images to the right are screen shots of the Amazon solution. Items to note:
qbo.Attachment.Amazon allows us to use S3 as a backing store for Attachments
qbo.Logging.Amazon allows us to use Dynamo as a logging sink for any logs, including error logs
qob.Message.Amazon allows us to use SMS as bulk SMTP mail delivery platform
all of these projects require use of AWSSDK.dll, so this DLL is included in the Dependencies folder
if AWSSDK.dll is updated, this solution will immediate tell you if the new DLL will break existing plugins (that's why all projects using AWSSDK.dll are in the same solution!)
the qbo solution folder contains the core QBO3 modules needed for the plugins
qbo.Attachment.Amazon needs qbo.Application, qbo.Attachment and qbo.Exception
qbo.Logging.Amazon needs qbo.Application, qbo.Logging, and qbo.Exception
qbo.Message.Amazon needs qbo.Application, qbo.Message, and qbo.Exception
it is critical that this solution reference the core existing projects, so that you are always building against the latest QBO3 core code when working with plugins
Theme Solutions
Themes are used to:
Override standard XSLTs found in qbo.Core.sln or plugins
Include client-specific plugins (generally non-QBO plugins)
Since themes are used to deploy XSLT overrides, a theme should always be the last project deployed when creating or updating a website.
Building
Overview
A successful build and deployment of QBO3 has a lot of moving parts. To deploy:
Open Powershell as an Administrator
Execute MSBuild against an appropriate project file
& 'C:\Program Files (x86)\MSBuild\14.0\Bin\amd64\MSBuild' .\qbo3.Sample.proj
& 'C:\Program Files (x86)\MSBuild\14.0\Bin\amd64\MSBuild' .\qbo3.Sample.proj /p:"server=1.2.3.4;User=me;Pwd=secret"
Execute MS Build against a specific branch:
& 'C:\Program Files (x86)\MSBuild\14.0\Bin\amd64\MSBuild' .\qbo3.Sample.proj /p:"BuildRoot=C:\Source\Branches\QMS\4.00"
There are several MSBuild scripts involved when invoking qbo.Sample.proj:
qbo.SvnTag.targets: includes a target for creating SVN tags
qbo.Core.targets: includes all standard qbo core web modules
qbo.Mortgage.targets: includes all standard qbo mortgage web modules
qbo.Amazon.targets: includes all Amazon-based plugins, including Attachments (S3), Logging (CloudWatch), Messaging (SES), and CloudSearch (document content searching)
qbo.Queuing.targets: includes standard queuing plugins
We use a mix of MSBuild scripts (.proj and .target files) and Powershell scripts (.ps1 files) to automate this deployment. Developers should create an MSBuild .proj file containing:
<Project ToolsVersion="12.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Full">
<Import Project="qbo3.Core.Targets"/>
<Import Project="qbo3.Mortgage.Targets"/>
<Import Project="qbo3.Amazon.Targets"/>
<Import Project="qbo3.Queuing.Targets"/>
<Import Project="qbo3.AttachmentPlugins.Targets"/>
<PropertyGroup>
<Nuget>"C:\Program Files (x86)\NuGet\nuget.exe"</Nuget>
<VS Condition=" '$(VS)'=='' ">14.0</VS>
<Configuration Condition=" '$(Configuration)'=='' ">Debug</Configuration>
<OutputRoot Condition=" '$(OutputRoot)'=='' ">$(MSBuildThisFileDirectory)..\BuildOutput\</OutputRoot>
<BuildRoot Condition=" '$(BuildRoot)'=='' ">$(MSBuildThisFileDirectory)..\qbo.Core</BuildRoot>
<SiteName Condition=" '$(SiteName)'==''">Default Web Site</SiteName>
<PublishFolder Condition=" '$(PublishFolder)'==''">c:\inetpub\wwwroot</PublishFolder>
<Server Condition=" '$(Server)'==''"></Server>
<User Condition=" '$(User)'==''"></User>
<Pwd Condition=" '$(Pwd)'==''"></Pwd>
<BuildTarget Condition="'$(BuildTarget)'==''">Rebuild</BuildTarget>
</PropertyGroup>
<!-- Define BuildProperties based on web deploy or file system deploy -->
<Choose>
<When Condition=" '$(Server)'!='' ">
<PropertyGroup>
<BuildProperties>VisualStudioVersion=$(VS);PublishProfile=$(MSBuildThisFileFullPath);DeployOnBuild=true;Configuration=Debug;OutputPath=$(OutputRoot);WebPublishMethod=MSDeploy;MSDeployServiceURL=$(Server);DeployIisAppPath=Default Web Site;SkipExtraFilesOnServer=True;MSDeployPublishMethod=WMSVC;EnableMSDeployBackup=True;AllowUntrustedCertificate=True;UserName=$(User);Password=$(Pwd)</BuildProperties>
</PropertyGroup>
</When>
<Otherwise>
<PropertyGroup>
<BuildProperties>VisualStudioVersion=$(VS);PublishProfile=$(MSBuildThisFileFullPath);DeployOnBuild=true;Configuration=Debug;OutputPath=$(OutputRoot);WebPublishMethod=FileSystem;PublishUrl=$(PublishFolder);SkipExtraFilesOnServer=True;</BuildProperties>
</PropertyGroup>
</Otherwise>
</Choose>
<!-- Define plugins specific to this client -->
<ItemGroup>
<Plugins Include="$(BuildRoot)\Plugins\Service\qbo.Service.HTTP\qbo.Service.Http.csproj"/>
<!-- Other client-specific plugins -->
</ItemGroup>
<!-- Define theme for this client -->
<ItemGroup>
<Client Include="$(BuildRoot)\sls.quandis.com\sls.MortgageWeb\theme.SLS.csproj"/>
</ItemGroup>
<!-- Make sure developer passes required parameters -->
<Target Name="Verify">
<Error Condition="$(Server) != '' and $(User) == ''" Text="User and Pwd are required parameters for web deployments."/>
<Error Condition="$(Server) != '' and $(Pwd) == ''" Text="User and Pwd are required parameters for web deployments."/>
</Target>
<!-- Deploy everything -->
<Target Name="Full">
<CallTarget Targets="Verify"/>
<CallTarget Targets="Core"/>
<CallTarget Targets="Mortgage"/>
<CallTarget Targets="Amazon"/>
<CallTarget Targets="Queuing"/>
<CallTarget Targets="AttachmentPlugins"/>
<MSBuild Projects="@(Plugins)" Properties="$(BuildProperties)" Targets="Build;Publish"/>
<CallTarget Targets="Theme"/>
</Target>
<!-- Deploy just the theme -->
<Target Name="Theme">
<CallTarget Targets="Verify"/>
<MSBuild Projects="@(Client)" Properties="$(BuildProperties)" Targets="Build;Publish"/>
</Target>
</Project>
Items to note:
A 'target' is a group of 1 or more MSBuild commands to execute; think of it as a function
Line 1 contains a DefaultTargets, which is a semi-colon delimited list of targets to be called if the user does not specify a target on the command line
This deploys everything: & 'C:\Program Files (x86)\MSBuild\14.0\Bin\amd64\MSBuild' .\qbo3.Client.proj
This deploys only the theme: & 'C:\Program Files (x86)\MSBuild\14.0\Bin\amd64\MSBuild' .\qbo3.Client.proj /t:Theme
The <Import> node references other MSBuild files. By convention, MSBuild files that are commonly reused by other MSBuild files end in '.targets'
We have 'common' targets for Core, Mortgage, etc. This is convenience only.
The <PropertyGroup> node contains variables used by the script. These variable can be overridden from the command line.
Default values can be set by using the Condition attribute
The <BuildRoot> variable stores the base location of where the projects are located. It can be overriden to point to other project locations such as a specific branch location.
The <BuildTarget> variable stored the targets to be called for each project; the default is Rebuild
The <BuildProperties> variable is critical; it contains all the parameters passed when actually building a project!
The parameters required for WebDeploy vs FileSystem deployments are quite different; a <Choose> task is used to determine which to use
If you pass a Server parameter, web deployment is assumed.
<ItemGroup> nodes enabling executing MSBuild commands against an array of objects
The nodes under <ItemGroup> can be named whatever you want; that's the name of the array you reference later
e.g. <MSBuild Projects="@(Plugins)" Properties="$(BuildProperties)" Targets="Build;Publish"/>
e.g. the @Plugins in the following means 'foreach ItemGroup child named Plugins'
The <Target> node contains 'commands' that do the actual work; everything else is just setup
The <CallTarget> node enables one target to execute another target
The <MSBuild> element actually executes MSBuild.exe against a .csproj (or .sqlproj, .vbproj, or other 'buildable' file)
To successfully build and deploy QBO3 from a powershell command line:
Ensure you have Visual Studio 2015 or later installed
Ensure you have nuget.exe installed (typically to C:\Program Files (x86)\NuGet\nuget.exe)
Using MSDeploy.exe requires:
Web Deployment Agent Service - Installed by using Web Platform Installer (Web Deploy 2.1)
Web Deploy must be installed as it installs msdeploy.axd
Web Management Service - Installed by using IIS roles and Features installation
From IIS Manager > {Machine} > Management Service, ensure remote connections are allowed (error: 403 forbidden)
From IIS Manager > {Website} > Bindings, ensure HTTPS is supported (a local certificate is okay) (error: no response from machine)
From an elevated command prompt, grant Web Management Service elevated privileges (error: 'A required privilege is not held by client')
sc.exe privs wmsvc SeChangeNotifyPrivilege/SeImpersonatePrivilege/SeAssignPrimaryTokenPrivilege/SeIncreaseQuotaPrivilege
the Web Management Service must be restarted after this command is issued
see qbo3.Powershell.targets for more information
Port 8172 needs to allow inbound traffic since this is what MSDeploy uses to connect
References
Nuget package restore with MSBuild
nuget.exe restore {path to solution file}
Removing Xamarin target messages
Remove Xamarin targets from the appropriate folder; keep it around if you ever need to build a Xamarin project
use the Deploy target instead of the Publish target
Passing semicolons to MSBuild parameters via Powershell
MSBuild standard.sqlproj /t:Deploy '/p:TargetConnectionString="Data Source=localhost;Integrated Security=True;Pooling=False"'
Troubleshooting Common Problems With Web Deploy
Step by step instructions to ensure Web Deploy is functional on target machine
Deployments
Client deployments shall be done as follows:
A developer shall create a MSBuild project file containing all the modules comprising a client machine.
See Trunk > qbo.3 > Publish > qbo3.Sample.proj for an example
Deployments shall be to a DEV environment, and each deployment to DEV should include an SVN tag.
Running Powershell ISE as an administrator, deploy your projects with:
& 'C:\Program Files (x86)\MSBuild\14.0\Bin\amd64\MSBuild' .\qbo3.Sample.proj /p:"tag=default"
Jasmine test specs shall be run in the DEV environments. Iterate through testing and deployment until all test specs pass.
Image the DEV environment, which will be used as the image for creating UAT, STAGE and PROD machines.