QBO Training‎ > ‎

Development and Deployment

Overview

This documents the development and deployment processes to be used by developers, engineers and architects at Quandis.

Quandis uses the following tools for development and deployment:
  • Visual Studio
    • C# class libraries for application-tier projects
    • ASP.NET web projects for web-tier projects
    • SQL Projects for database components
  • SVN, Ankh and Tortoise for source control
  • MSBuild to create deployment files
  • Powershell to coordinate the deployment of build files across multiple machine

Source Control

Quandis uses a single SVN repository for all source-controlled projects and solutions. For QBO3, the structure is:
  • http://source.quandis.com
    • Branches
      • qbo.3
        • {YYYYMM}
    • Tags
    • Trunk
      • qbo.3
        • Publish: contains powershell scripts used for deployments
        • qbo.Core
          • Application Tier: core C# libraries
          • Plugins: C# libraries dependent on third-party products, like Amazon, Google, Crystal Reports, Aspose, etc.
          • Themes: client or product specific overrides to core QBO3 components
          • Web Tier: core ASP.NET web projects 
        • qbo.DB
          • Standard: core QBO3 database components (tables, user-defined functions, etc.)

Visual Studio Solutions

Core Solution

For QBO3-based core projects, there is a qbo.Core.sln solution file found at:

Source > Trunk > qbo.3 > qbo.Core > qbo.Core.sln

All changes that apply to core QBO3 code should be made in this solution. 

When you are stepping through QBO3 code to learn or debug it, you should do so from this solution, as it will contain ALL of the QBO projects that you might need to debug from a core QBO3 perspective.

Plugin Solutions

For QBO3-based plugin solutions, the source folder structure should be:
  • Source > Trunk > qbo.3 > qbo.Core > Plugins > {Third party product}
    • Dependencies: folder containing any DLLs that need to be referenced by projects in the solution
    • qbo.{Module}.{Third Party Product}: folder containing plugin project
    • qbo.{Third Party Product}.sln
Amazon plugin solution structure
The solution file should mirror the folder structure:
  • qbo.{Third Party Product}.sln
    • Dependencies: solution folder containing third-party DLLs that need to be referenced
    • qbo: solution folder containing core QBO3 application-tier projects that are referenced by the plugin projects
    • qbo.{Core Module}.{Third Party Product}: project file that is the plugin
      • all such projects should be based on the ASP.NET Web Project template, so that they can be deployed using MSBuild and WebDeploy
The images to the right are screen shots of the Amazon solution. Items to note:
  • qbo.Attachment.Amazon allows us to use S3 as a backing store for Attachments
  • qbo.Logging.Amazon allows us to use Dynamo as a logging sink for any logs, including error logs
  • qob.Message.Amazon allows us to use SMS as bulk SMTP mail delivery platform
  • all of these projects require use of AWSSDK.dll, so this DLL is included in the Dependencies folder
  • if AWSSDK.dll is updated, this solution will immediate tell you if the new DLL will break existing plugins (that's why all projects using AWSSDK.dll are in the same solution!)
  • the qbo solution folder contains the core QBO3 modules needed for the plugins
    • qbo.Attachment.Amazon needs qbo.Application, qbo.Attachment and qbo.Exception
    • qbo.Logging.Amazon needs qbo.Application, qbo.Logging, and qbo.Exception
    • qbo.Message.Amazon needs qbo.Application, qbo.Message, and qbo.Exception
    • it is critical that this solution reference the core existing projects, so that you are always building against the latest QBO3 core code when working with plugins
Theme Solutions

Themes are used to:
  • Override standard XSLTs found in qbo.Core.sln or plugins
  • Include client-specific plugins (generally non-QBO plugins)
Since themes are used to deploy XSLT overrides, a theme should always be the last project deployed when creating or updating a website.

Building

Overview

A successful build and deployment of QBO3 has a lot of moving parts. To deploy:
  • Open Powershell as an Administrator
  • Execute MSBuild against an appropriate project file
    • & 'C:\Program Files (x86)\MSBuild\14.0\Bin\amd64\MSBuild' .\qbo3.Sample.proj 
    • & 'C:\Program Files (x86)\MSBuild\14.0\Bin\amd64\MSBuild' .\qbo3.Sample.proj /p:"server=1.2.3.4;User=me;Pwd=secret"
  • Execute MS Build against a specific branch:
    • & 'C:\Program Files (x86)\MSBuild\14.0\Bin\amd64\MSBuild' .\qbo3.Sample.proj /p:"BuildRoot=C:\Source\Branches\QMS\4.00"
There are several MSBuild scripts involved when invoking qbo.Sample.proj:
  • qbo.SvnTag.targets: includes a target for creating SVN tags
  • qbo.Core.targets: includes all standard qbo core web modules
  • qbo.Mortgage.targets: includes all standard qbo mortgage web modules
  • qbo.Amazon.targets: includes all Amazon-based plugins, including Attachments (S3), Logging (CloudWatch), Messaging (SES), and CloudSearch (document content searching)
  • qbo.Queuing.targets: includes standard queuing plugins
We use a mix of MSBuild scripts (.proj and .target files) and Powershell scripts (.ps1 files) to automate this deployment. Developers should create an MSBuild .proj file containing:

<Project ToolsVersion="12.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Full">
  <Import Project="qbo3.Core.Targets"/>
  <Import Project="qbo3.Mortgage.Targets"/>
  <Import Project="qbo3.Amazon.Targets"/>
  <Import Project="qbo3.Queuing.Targets"/>
  <Import Project="qbo3.AttachmentPlugins.Targets"/>
  <PropertyGroup>
    <Nuget>"C:\Program Files (x86)\NuGet\nuget.exe"</Nuget>
    <VS Condition=" '$(VS)'=='' ">14.0</VS>
    <Configuration Condition=" '$(Configuration)'=='' ">Debug</Configuration>
    <OutputRoot Condition=" '$(OutputRoot)'=='' ">$(MSBuildThisFileDirectory)..\BuildOutput\</OutputRoot>
    <BuildRoot Condition=" '$(BuildRoot)'=='' ">$(MSBuildThisFileDirectory)..\qbo.Core</BuildRoot>
    <SiteName Condition=" '$(SiteName)'==''">Default Web Site</SiteName>
    <PublishFolder Condition=" '$(PublishFolder)'==''">c:\inetpub\wwwroot</PublishFolder>
    <Server Condition=" '$(Server)'==''"></Server>
    <User Condition=" '$(User)'==''"></User>
    <Pwd Condition=" '$(Pwd)'==''"></Pwd>
    <BuildTarget Condition="'$(BuildTarget)'==''">Rebuild</BuildTarget>
  </PropertyGroup>
  
  <!-- Define BuildProperties based on web deploy or file system deploy -->
  <Choose>
    <When Condition=" '$(Server)'!='' ">
      <PropertyGroup>
        <BuildProperties>VisualStudioVersion=$(VS);PublishProfile=$(MSBuildThisFileFullPath);DeployOnBuild=true;Configuration=Debug;OutputPath=$(OutputRoot);WebPublishMethod=MSDeploy;MSDeployServiceURL=$(Server);DeployIisAppPath=Default Web Site;SkipExtraFilesOnServer=True;MSDeployPublishMethod=WMSVC;EnableMSDeployBackup=True;AllowUntrustedCertificate=True;UserName=$(User);Password=$(Pwd)</BuildProperties>
      </PropertyGroup>
    </When>
    <Otherwise>
      <PropertyGroup>
        <BuildProperties>VisualStudioVersion=$(VS);PublishProfile=$(MSBuildThisFileFullPath);DeployOnBuild=true;Configuration=Debug;OutputPath=$(OutputRoot);WebPublishMethod=FileSystem;PublishUrl=$(PublishFolder);SkipExtraFilesOnServer=True;</BuildProperties>
      </PropertyGroup>
    </Otherwise>
  </Choose>

  <!-- Define plugins specific to this client -->
  <ItemGroup>
    <Plugins Include="$(BuildRoot)\Plugins\Service\qbo.Service.HTTP\qbo.Service.Http.csproj"/>
    <!-- Other client-specific plugins -->
  </ItemGroup>
  
  <!-- Define theme for this client -->
  <ItemGroup>
    <Client Include="$(BuildRoot)\sls.quandis.com\sls.MortgageWeb\theme.SLS.csproj"/>
  </ItemGroup>

  <!-- Make sure developer passes required parameters -->
  <Target Name="Verify">
    <Error Condition="$(Server) != '' and $(User) == ''" Text="User and Pwd are required parameters for web deployments."/>
    <Error Condition="$(Server) != '' and $(Pwd) == ''" Text="User and Pwd are required parameters for web deployments."/>
  </Target>
  
  <!-- Deploy everything -->
  <Target Name="Full">
    <CallTarget Targets="Verify"/>
    <CallTarget Targets="Core"/>
    <CallTarget Targets="Mortgage"/>
    <CallTarget Targets="Amazon"/>
    <CallTarget Targets="Queuing"/>
    <CallTarget Targets="AttachmentPlugins"/>
    <MSBuild Projects="@(Plugins)" Properties="$(BuildProperties)" Targets="Build;Publish"/>
    <CallTarget Targets="Theme"/>
  </Target>
  
  <!-- Deploy just the theme -->
  <Target Name="Theme">
    <CallTarget Targets="Verify"/>
    <MSBuild Projects="@(Client)" Properties="$(BuildProperties)" Targets="Build;Publish"/>
  </Target>
</Project>

Items to note:
  • A 'target' is a group of 1 or more MSBuild commands to execute; think of it as a function
  • Line 1 contains a DefaultTargets, which is a semi-colon delimited list of targets to be called if the user does not specify a target on the command line
    • This deploys everything: & 'C:\Program Files (x86)\MSBuild\14.0\Bin\amd64\MSBuild' .\qbo3.Client.proj 
    • This deploys only the theme: & 'C:\Program Files (x86)\MSBuild\14.0\Bin\amd64\MSBuild' .\qbo3.Client.proj /t:Theme
  • The <Import> node references other MSBuild files. By convention, MSBuild files that are commonly reused by other MSBuild files end in '.targets'
    • We have 'common' targets for Core, Mortgage, etc. This is convenience only.
  • The <PropertyGroup> node contains variables used by the script. These variable can be overridden from the command line.
    • Default values can be set by using the Condition attribute
    • The <BuildRoot> variable stores the base location of where the projects are located. It can be overriden to point to other project locations such as a specific branch location.
    • The <BuildTarget> variable stored the targets to be called for each project; the default is Rebuild
  • The <BuildProperties> variable is critical; it contains all the parameters passed when actually building a project!
    • The parameters required for WebDeploy vs FileSystem deployments are quite different; a <Choose> task is used to determine which to use
    • If you pass a Server parameter, web deployment is assumed.
  • <ItemGroup> nodes enabling executing MSBuild commands against an array of objects
    • The nodes under <ItemGroup> can be named whatever you want; that's the name of the array you reference later
    • e.g. <MSBuild Projects="@(Plugins)" Properties="$(BuildProperties)" Targets="Build;Publish"/>
    • e.g. the @Plugins in the following means 'foreach ItemGroup child named Plugins'
  • The <Target> node contains 'commands' that do the actual work; everything else is just setup
  • The <CallTarget> node enables one target to execute another target
  • The <MSBuild> element actually executes MSBuild.exe against a .csproj (or .sqlproj, .vbproj, or other 'buildable' file)

To successfully build and deploy QBO3 from a powershell command line:
  • Ensure you have Visual Studio 2015 or later installed
  • Ensure you have nuget.exe installed (typically to C:\Program Files (x86)\NuGet\nuget.exe)
  • Using MSDeploy.exe requires:
    • Web Deployment Agent Service - Installed by using Web Platform Installer (Web Deploy 2.1)
      • Web Deploy must be installed as it installs msdeploy.axd
    • Web Management Service - Installed by using IIS roles and Features installation
      • From IIS Manager > {Machine} > Management Service, ensure remote connections are allowed (error: 403 forbidden)
      • From IIS Manager > {Website} > Bindings, ensure HTTPS is supported (a local certificate is okay) (error: no response from machine)
      • From an elevated command prompt, grant Web Management Service elevated privileges (error: 'A required privilege is not held by client')

        sc.exe privs wmsvc SeChangeNotifyPrivilege/SeImpersonatePrivilege/SeAssignPrimaryTokenPrivilege/SeIncreaseQuotaPrivilege

        • the Web Management Service must be restarted after this command is issued
        • see qbo3.Powershell.targets for more information
    • Port 8172 needs to allow inbound traffic since this is what MSDeploy uses to connect
References

Deployments

Client deployments shall be done as follows:
  • A developer shall create a MSBuild project file containing all the modules comprising a client machine.
  • Deployments shall be to a DEV environment, and each deployment to DEV should include an SVN tag.
    • Running Powershell ISE as an administrator, deploy your projects with:
      & 'C:\Program Files (x86)\MSBuild\14.0\Bin\amd64\MSBuild' .\qbo3.Sample.proj /p:"tag=default"
  • Jasmine test specs shall be run in the DEV environments. Iterate through testing and deployment until all test specs pass.
  • Image the DEV environment, which will be used as the image for creating UAT, STAGE and PROD machines.


Comments