As part of the recent work I’ve done on migrating from TDS to SCS I had some tidying up to do. I needed to remove the TDS projects from the solution and all the files and sub-folders, including the .item files.
We had around 66 TDS projects in the Solution so this wasn’t something I wanted to do manually as It would take a while and I’d likely missing something. I figured someone must have written a script to do this but I couldn’t find anything. So I created a PowerShell Script myself to do this.
Note 1:There is a dependency here on the dotnet cli. This was the most efficient and supported way of cleaning up the solution file and most developers will likely have this installed already.
Note 2:I setup SCS to create my serialized items in an folder called scs. This was so all the legacy TDS items could stay in the serialization folder instead to separate them. If you’ve not done this you may need to modify the script below for your needs to re-configure the SCS folder before using my script.
If you save the script to your machine from the Gist below it will look something like this:
Show me the Script Already
When you open the script you need to set the variables for your Solution path, Solution name and if projects, files, folders and parent folders should be removed. Most of these should be set to $true but depending on your folder structure you might need to set $tidyUpParentFolders to $false:
After the script has run you will see an output showing what the script has removed, including project files, item files and folders:
One you’ve run it check the removed files look correct before checking in your changes to source control. Also ensure your solution still builds and there are no issues.
Hopefully this is useful for anyone else who makes the move to TDS from SCS.
A client I’m currently working with recently upgrade to Sitecore 10.3 and I was tasked with migrating our TDS projects to the much newer Sitecore Content Serialization (SCS). I imagine a lot of people need to do this so thought I’d share the approach I took and steps I followed to do this.
Why migrate to SCS?
SCS is now the de facto standard for Content Serialization for Sitecore – not only for Sitecore XP but also for XM Cloud too. It has now matured to a point where it has all the features we should need as Sitecore developers and is well established. It’s also built in an modern way so is easy to use and very fast.
If you are currently using TDS then this is now end of life/legacy and will likely soon not be supported. In addition to this; managing items via TDS is tedious and slow. With SCS we can just run the watch command and pickup changes from Sitecore without having to manually add items, which will be a lot quicker. You also need a license for TDS and don’t for SCS (via the CLI at least).
I‘ve always preferred Unicorn to TDS, but on the current project I’m working on TDS was already implemented. If you are using unicorn though there are similar approaches to migrating from Unicorn to SCS too.
Differences between TDS project files and SCS Modules
TDS Project files are a proprietary XML based and contains list of all the items to Serialize, they are designed to be used via the TDS Visual Studio plugin to manage items.
SCS Module files on the other hand are JSON based files which include a list of the files to sync. The items are then serialized to Disk as .YAML files. The format is a little different for these but essentially the definition is quite similar. Here is a nice comparison from Jason St-Cyr’s blog post:
Approaches to Migration
There are two different approaches to migration to consider here.
Manual Migration
You could manually migrate each TDS project item by item but unless your working on a very small Sitecore project or you’ve not got many items in source control this will likely take quite some time so I wouldn’t recommend it.
Tools to Automate Migration
I spent quite a while looking around the Sitecore community to see what was already out there and there were 3 pretty good options that I found:
Sitecore Serialisation Converterby Joseph Long – .Net console app. supports item and role serialization, full featured: save path/relative save path, ignore list etc.
A PowerShell Script by Aaron Bikle – impressive power shell script, less feature rich than Sitecore Content Serializer but does support an ignore list and consolidate paths which is nice.
TDSC by Matthew Ellins – a .Net Core console app. Supports item serialization but not roles. Not as full featured as Sitecore Serialisation Converter.
I looked at all of these and tested them but ultimately landed on Sitecore Serialisation Converter as it was the most full-featured, robust and flexible option.
I did however make quite a few updates to it and raised an PR – which has been merged today by Joseph so you can take advantage of them too:
Improved the error handing by adding more checks around various aspects and logging out all errors/warnings and progress
Added logging via log4net (ssc-skipped.log and ssc-all.log)
Added tracking of errors/stats with summary at end of success, failed, skipped etc
Automation of the Json module listing to add to your Sitecore.json file
Added support for removing ‘TDS’ from module/namespaces (StripTDSFromName option) – for if you have TDS included in the name of your TDS projects and want to remove it.
Added support for Filtering on just specific project name (ProjectNameToMatch option) – to allow running on a single project/testing. Leave blank to include all Projects.
Added a new SkipCreateIfExists option to not create the Json file if it already exists.
I also stole some of Aaron’s exclude list and added that :-).
The following steps give you an overview of the process I followed to migrate to SCS. It’s probably not the only way to do this but it worked well for me and is likely pretty close to the approach you will need to take to do this:
Edit the appsettings.config file as follows: ProjectDescription – set this to your project name, it will be added to your Module files. SolutionFolder – set this to the root of your Solution folder. TDS projects will be found from here. SavePath – you probably only want to set this for testing so all module files are created in the same place (use relative path instead to create them in each feature/foundation/project folder). UseRelativeSavePath – set to false when testing using the SavePath above, but true when your ready to create the module files in a relative location. RelativeSavePath – set this if UseRelativeSavePath is set to true above. For me this path was ../. StripTDSFromName – set to true if you have the text ‘TDS’ in your project names and it will be removed when creating the .Json files. ProjectNameToMatch – set this to filter on one or more projects. Useful for testing or for just targeting specific projects. Leave blank to find all TDS projects. SkipCreateIfExists – set to true if you want .Json files to be skipped instead recreated if they already exist. Useful if you’ve run SSC a few times.
Either Run the project in debug mode in Visual Studio OR Build it and go to the .exe in the output folder via command line and run the .Exe like so: SitecoreSerialisationConverter.exe
Check the console/logs for any issues or errors, you should find these in the debug folder
Check the summary looks correct in the Console and the Logs and no projects or important items are missing
Copy your Module list from the output in the Console to use in your Sitecore.json file, e.g:
Verbose – Run all the commands with -v after them so that you get full debugging of any issues. e.g: dotnet sitecore ser validate -v -i MyProject.Feature*
Filters – Use the -i param to filter by a specific project name pattern, e.g: dotnet sitecore ser pull -v -i MyProject.Feature* – will match all projects starting with ‘MyProject.Feature’
Validate – Use the validate command to check the your Module Json file paths are valid. e.g: dotnet sitecore ser validate -v. You can combine this with -i to validate just one project at a time.
SCS Issues and Errors
I hit a bunch of errors when trying to run the pull and/or validate command. I think I had perhaps 100 errors or more to resolve across around 60 converted TDS projects so it took me quite some time to resolve them all. It’s definitely easier if you target on Project at a a time, fix the issues, commit to source control and move to the next though. Here are the issues I tracked and what I had to do to fix them.
Duplicate Paths cannot be serialized
If this happens it means you have multiple module files which have references to the same item/path. You need to find the duplicate references and remove the path from all but one Module file and try and run the command again.
Non Unique Paths cannot be serialized
If this happens it means an item with the same name exists more than once in Sitecore and it’s trying to serialize it to disc as an YAML file but can’t create the item more than once with the same path name.
To fix it remove one or more copies of the item from Sitecore and try and run the command again. You will need to work out which is the correct one to remove. I used SPE to remove items by ID to speed this up.
Serialized item contained a blank value
You will see an error like so if this happens:
Check your serialised item, you will see there is no language set. Then check the language versions in Sitecore and look for an Invariant Language. I found I had to run an SQL command on my master and core database to remove version languages:
DELETE FROM [sitecore_master].[dbo].[VersionedFields] WHERE [Language] = ''
DELETE FROM [sitecore_core].[dbo].[VersionedFields] WHERE [Language] = ''
Blank Database Values
There were some situations where the database in my Json files ended up empty for some reason. I did make some updates to the Sitecore Serialization Converter to fix this by defaulting the value to Master if the Project name contains ‘Master’ and Core if it contains ‘Core’. But if this happens to you then you will need to fix this manually in the Json files else SCS will throw errors:
Cache already contains GUID
I also had some errors related to items already existing in the cache, something like:
“/sitecore/content/* (GUID) attempted to be cached, but the cache already contained /sitecore/content/*. Non-unique paths cannot be serialized. Please choose a different item name.“
If you get this error then you should be able to fix it with the following command or perhaps a combination of this fix and the non unique items fix:
dotnet sitecore ser validate --fix
Source Control
One other thing to consider is how you approach adding your changes to Source Control to allow for easy tracking of changes and roll-back etc. I did this my adding all my Json module files first and then pulling each Helix layer (feature/foundation/project) one at a time and pushing the Yaml files in batches.
Next Steps
The next steps for me are adding the integration with the CLI to Github Actions, pushing the items from Source control to other environments and removing the TDS projects and serialized files. I will write about this in a future blog post.
Links
As usual there were a lot of links that helped with this, here are some of them:
I don’t often write Sitecore blog posts from a business perspective but I today I thought I would as I think some Sitecore customers are still on 8.x or 9.0 and are considering an upgrade to Sitecore 9.2.
In this post I’m going to outline what the benefits of moving to 9.2 are and also the considerations you should make before embarking on an upgrade.
What are Benefits / Features of Sitecore 9.2?
This is usually one of the first questions that will be asked. I’m not going to keep this fairly high-level as there are many in-depth blog posts out there on Sitecore 9.1 & 9.2 features but the following should provide a succinct summary.
Sitecore 9.1 & 9.2 bring a lot of new features and there has been a big focus on splitting features and functionality out from the previous large monolithic architecture to a smaller modern micro-service based architecture:
Sitecore Cortex – Sitecore’s new Machine Learning Brain which can Suggested Personalisation and process and leverage customer data
Sitecore Identity – Single Sign On and Federated Authentication (using IdentityServer)
Sitecore Universal Tracker – Provides an central Analytics API to track interactions from any device (e.g mobile apps, IoT, AR and VR)
Sitecore JavaScript Services (JSS) – allows development of apps using front-end frameworks such as Angular, React and Vue, using Sitecore as the source for the data (including SXA & Sitecore Forms support in 9.2).
Sitecore Accelerator Framework (SXA) Accessibility Improvements – SXA has been around for a couple of years and is a package or pre-built components for Sitecore allowing you to get sites up and running quicker.
Sitecore Host – A lean common runtime for .NET Core applications (Powers: Horizon, Universal Tracker, Sitecore Identity)
Helix Configuration – 9.1 shipped with some Helix configuration already setup. Helix is Sitecore’s recommended approach to building Sitecore Sites it sets out an number of overall design principles and conventions to follow.
Sitecore Installer Assistant (SIA) – 9.2 provides a new GUI for easily installing Sitecore in a few clicks
SSL offloading – 9.2 provides an SSL offloading config to Improve security and performance by shifting SSL Processing onto separate processors
Active Personalization Dashboard – 9.2 has a new new dashboard which provides an overview of all personalization actively occurring, providing visibility of poorly/well performing tactics to act upon.
Many other improvements including: Search, Content Delivery, xDB and Analytics, Sitecore Forms, EXM, SIF(2), YAML Item Serialization, Bug Fixes and performance Improvements
But We’re on Sitecore 8.2 or below, what else do I get?
Mainstream Support
If you are yet to upgrade to Sitecore 9 then you need to consider this soon as Mainstream support for 8.2 and below expires in December 2019. https://kb.sitecore.net/articles/641167
This means that whilst Sitecore will still provide security updates and fixes and endeavour to assist with product incidents they may not provide support for the following:
Assistance with errors or unexpected behaviour during installation or development
Addressing product defects as hot-fixes or patches
Compatibility fixes for supported technology platforms.
Therefore if you are on 8.2 or below after December this year then you may need to upgrade to resolve certain issues (depending on what the issues are).
Sitecore 9.0 Features
If you are on 8.2 or below you will also benefit from all the features released in 9.0, here are the key ones:
xConnect – Provides an unified API that centralises data access
Headless CMS support and JSS – Allows web apps to be built with React, Angular, Vue using Sitecore data
Rules based configuration – Configuration is now much simpler as it can be set based on the server role (e.g Content Delivery, Content Authoring)
Sitecore Install Framework (SIF) – SIF is a framework for installing Sitecore using automated scripts. This is how Sitecore 9 and above is now installed and it means deployments can be automated easier.
Sitecore Accelerator Framework (SXA) – SXA is a pre-built set of components to allow pages to be developed quicker with less development effort
Sitecore Forms – Sitecore 9.0 has newly designed forms to replace WFFM. These are built in a modern way and are easier to configure and customise.
Upgrade Considerations
So there are clearly some significant benefits to carrying out an upgrade to Sitecore 9.2, especially if you are on 8.2 still, as some customers are.
There are some additional requirements and considerations when moving to Sitecore 9.2 and standard upgrade considerations which need to be planned for:
Additional Servers / Roles – Due to changes to the architecture to split out elements of Sitecore into smaller services Sitecore 9, 9.1 and 9.2 require some additional Servers and roles. These are for xConnect, Identity Server, Cortex and processing. When you are planning resources for your upgrade this should be considered, particularly on Azure as there are quite a few additional services required.
Module Upgrades – Most Sitecore sites will make use of a number of Modules to extend the out of the box functionality. Some popular modules are: Web Forms for Marketers (WFFM), Url Rewrite and Sitecore Powershell Extensions (SPE). It is important to ensure that each module you are using is supported in Sitecore 9.2 and to consider time for upgrading each module as part of the upgrade process.
Updated Licence – Most customers will require a new licence generating when upgrading to Sitecore 9 and above. Therefore ensure you request this from your account manager in advance and that you have all features/modules you require included on your licence.
Active Directory module – Sitecore 9.1 & 9.2 or later does not support the Active Directory module. Sitecore uses Identity server to handle logins instead of legacy methods such as AD. Therefore if you are upgrading to 9.1 or above and use the AD module you will need to implement an integration with Active Directory from Identity Server.
Search – Sitecore 9 and above no longer supports Lucene so you must use SOLR or Azure Search. Lucene did not work well in distributed search scenarios so moving to SOLR or Azure Search has been recommended for a while. SSL is also required for SOLR in 9.0.
Code & Configuration Updates – Sitecore try and reduce breaking changes where possible but sometimes they are unavoidable, so as with all Sitecore upgrades some custom code and configuration will need to be updated to be compatible with Sitecore 9.2.
To assist with this process Sitecore does provide an Express Migration Tool (to get to 9.0) and an Update Installation Wizard to update to 9.2. However these tools will likely only get you so far for highly customised Sitecore implementations and will likely need some manual upgrading too. Any code that needs upgrading will not be auto-upgraded by the tool. If you are on 8.2 then there are quite a few changes for analytics with xConnect. All references in Visual Studio will need updating to 9.2 also.
Support Patches & Hotfixes – You will likely have some hotfixes or support patches that have been applied to your solution over time. These will need analysing as part of the upgrade to understand if they are still required or not in 9.2. Many of these may have been rolled into 9.2 already.
Implementation Quality – Best practice Sitecore implementations do not customise any out of the box Sitecore files and instead use patch files and extensions to customise configuration and other Sitecore functionality. Ideally the base platform should be installed ‘as is’ and the customisations layered on top to assist with upgradability. Depending on how well your implementation has been carried out will impact how easy it is to upgrade. If your implementation is not best practice then your team should take the time during the upgrade to correct this and take advantage of some of the newer and better ways of doing this.
A Note on Upgrading WFFM
Often one of the key stumbling blocks for upgrading Sitecore to 9.1 or 9.2 is Web Forms For Marketers, this is because it was deprecated in Sitecore 9.1 and is also not supported in Sitecore 9.2. Sitecore 9.0.2 is the last release where WFFM can be used. Sitecore Forms has replaced it. If customers have many forms this can be a bit daunting.
So you could just upgrade to Sitecore 9.0.2 and stop there, but you would be missing out on a lot of features and you will need to upgrade again in the near future. Thankfully there is another option, there is now a community built tool to help automate this for you and convert WFFM forms and data to Sitecore Experience Forms: https://github.com/afaniuolo/WFFM-Conversion-Tool. I have yet to use this but have heard good things about it and it is regularly updated.
I’ve tried to cover the key features of 9.2 and considerations for upgrading here. As you can probably see the longer you leave an upgrade the more complex it becomes as there are more changes to consider. Frequent upgrades of Sitecore should be the aim as this will reduce the time and investment needed to carry them out. Hopefully you’ll find this post useful for planning an upgrade to Sitecore 9.2 and leverage the investment you have made in the platform.
