One of the most challenging things when dealing with an ongoing Sitecore project is how to ensure the correct Sitecore item changes are deployed along with the right code.  Ensuring the correct items are being promoted through environments along with the associated code can be both time consuming, and fraught with disaster if something goes wrong.

As someone who is continually trying to drive out inefficiencies and create repeatable processes, I really struggled with the manual way my company tracked and pushed items between environments!  The process was as follows;

• Developer makes changes locally and notes all the item changes in the jira issue
• Deployment engineer takes that list and packages all the items from the integration environment
• Deployment engineer installs package to upper environment

If all goes well, the same package can be used again for further environments.

NARRATOR: Things rarely went well

Between the human error of not noting all items required, and then not packaging everything in the list, we found we were spending hours a week troubleshooting issues relating to Sitecore items missing from environments.  Worse still was when the same item was modified for multiple unrelated issues, but only one was to be promoted to the next environment.

After some investigation we calculated actual numbers for how long we were spending on these manual processes, and some pretty good estimates on how long we spent trying to fix errors.  The numbers were terrifying – tens of hours a week during busy periods (I work for an agency with some very active clients)!

We use TDS to manage Sitecore item changes, and there is an option within TDS to generate packages based on what has been synced.  However we found that the time to install that package with every deploy became an inconvenience – we wanted fast builds to integration environments to get the shortest feedback loop, and these packages of 1000+ items were not cutting it.

The solution

I will prefix this by saying this solution is not for everyone.  In many cases using the built in functionality of TDS will be sufficient, however we had some specific requirements that lead us down this path.  As with most software development, if we could go back and do it again, perhaps we would do it differently! Maybe we’d even use Unicorn instead, who knows…

In the future I will try to expand on each of these steps with specific blog posts to detail some of the specifics we implemented.

Step 1 – Source Control is the Source Of Truth

In a tightly controlled deployment pipeline, if something is not in Source Control or your Application Lifecycle Management (ALM) tool, it shouldn’t be going anywhere near your production server.  We pulled that same logic back all the way to our integration environment.  This meant that for a Sitecore item change to be made anywhere other than a developer’s local instance, it had to be checked into git (our source control provider of choice).  This wasn’t anything new, but for that very reason, we had an accurate ledger of Sitecore changes we could peek into at any given moment.  This will come in very useful down the road.

Step 2 – Automate everything

It used to be that to deploy code to the integration environment a developer had to merge their code into the integration branch, then RDP into the server, pull the integration branch, open it in Visual Studio then run the appropriate build THEN manually sync TDS items.

Between the time wasted merging, the money wasted on additional licenses and the general opportunity for mistakes (or nefarious “let me just fix this bug directly on the server.. oops I forgot to commit it” moments), it was not a pretty process.

Every single part of this could be automated, it just needed a change in approach.  Enter; TeamCity.  TeamCity is a fantastic tool that is essentially a Build Runner that knows a bunch of stuff, but nothing specific.  By this I mean that it comes with out of the box support for all the major version control systems, it knows about a lot of build runners (think MSBuild, make, NAnt, powershell etc) and it can trigger builds on demand or by triggers.

What we did was configure builds that would, when a developer pushes a change in a feature branch, automatically merge that branch into the integration branch.  That will then fire another build that runs all required compilation (.NET via MSBuild and front end assets via grunt), builds a Sitecore Update package of changed items and combines everything together in one NuGet compatible artifact to send to Octopus deploy.

All this took about a week of two people’s time to build out, test and finalize, but in that time we had eliminated approximately 1.5 hours per day of developer wasted time!

Step 3 – Wait… Building a Sitecore update package?

Yes!  By utilizing some code from the fantastic Unicorn and Sitecore Courier projects, it is possible to read TDS serialized items and generate a Sitecore Update package from them.

Earlier I noted that Source Control becomes the source of truth for Sitecore items, and this is where that comes into play.  We can use the TeamCity REST API to get a list of all the changes made between two builds (which directly correlates to two commits in source control).  That way we can get a definitive list of Sitecore items that have changed, run some logic on those items (to exclude content items that may have been committed, for example) then build an update package.

Step 4 – So you have your artifact, now what?

For the longest time we utilized TeamCity as our deployment tool as well – it does the job, but it’s not really what it is designed for.  Over time we migrated to using a tool called Octopus Deploy to actually push our code into the various environments.  This way we could have a tool with a true audit trail, that was designed to deploy web applications, and did so over secure HTTP connects to remote agents, rather than via UNC path that we had to use for TeamCity.

Our deploys via Octopus as a sequence of powershell scripts that we have custom written to handle some of our specific environment setups.  One of the key steps in this process is to push the Update package to the CM server, install it and publish those items.

To do this, we utilize a tool by an old colleague, Kevin Obee, called Sitecore.Ship.  This acts as a CI helper within Sitecore to do everything we need it to.  We customized the standard version of Ship (thank you, Open Source) to remove some third party dependencies and to be dropped into a running application with as little fuss as possible.

After pushing our update package into Sitecore, we use a JSON manifest that is generated as part of the Update package generation process to inform Ship of what items to publish.  This means we don’t have to do a manual publish, and also that we don’t have to do a full site publish to ensure we get every item that was installed.

 

Summary

This was a high level overview of how we can push changes to our clients environments much quicker, much cheaper, and with lower rates of errors (those errors still happen, but they are generally caused by items not being checked in, and therefore they are found on the integration environment very quickly).  If people are using other solutions to get their item changes up through environments, I’d love to hear about them!