Tag: YAML

A Guide to Migrating from Sitecore TDS to SCS

A client I’m currently working with recently upgrade to Sitecore 10.3 and I was tasked with migrating our TDS projects to the much newer Sitecore Content Serialization (SCS). I imagine a lot of people need to do this so thought I’d share the approach I took and steps I followed to do this. 

Why migrate to SCS?

SCS is now the de facto standard for Content Serialization for Sitecore – not only for Sitecore XP but also for XM Cloud too. It has now matured to a point where it has all the features we should need as Sitecore developers and is well established. It’s also built in an modern way so is easy to use and very fast.  

If you are currently using TDS then this is now end of life/legacy and will likely soon not be supported. In addition to this; managing items via TDS is tedious and slow. With SCS we can just run the watch command and pickup changes from Sitecore without having to manually add items, which will be a lot quicker. You also need a license for TDS and don’t for SCS (via the CLI at least). 

I‘ve always preferred Unicorn to TDS, but on the current project I’m working on TDS was already implemented. If you are using unicorn though there are similar approaches to migrating from Unicorn to SCS too.

Differences between TDS project files and SCS Modules

TDS Project files are a proprietary XML based and contains list of all the items to Serialize, they are designed to be used via the TDS Visual Studio plugin to manage items.

SCS Module files on the other hand are JSON based files which include a list of the files to sync. The items are then serialized to Disk as .YAML files. The format is a little different for these but essentially the definition is quite similar. Here is a nice comparison from Jason St-Cyr’s blog post:

Approaches to Migration

There are two different approaches to migration to consider here.

Manual Migration

You could manually migrate each TDS project item by item but unless your working on a very small Sitecore project or you’ve not got many items in source control this will likely take quite some time so I wouldn’t recommend it.

Tools to Automate Migration

I spent quite a while looking around the Sitecore community to see what was already out there and there were 3 pretty good options that I found:

  1. Sitecore Serialisation Converter by Joseph Long – .Net console app. supports item and role serialization, full featured: save path/relative save path, ignore list etc.
  2. A PowerShell Script by Aaron Bikle – impressive power shell script, less feature rich than Sitecore Content Serializer but does support an ignore list and consolidate paths which is nice.
  3. TDSC by Matthew Ellins – a .Net Core console app. Supports item serialization but not roles. Not as full featured as Sitecore Serialisation Converter.

I looked at all of these and tested them but ultimately landed on Sitecore Serialisation Converter as it was the most full-featured, robust and flexible option.

I did however make quite a few updates to it and raised an PR – which has been merged today by Joseph so you can take advantage of them too:

  • Improved the error handing by adding more checks around various aspects and logging out all errors/warnings and progress
  • Added logging via log4net (ssc-skipped.log and ssc-all.log)
  • Added tracking of errors/stats with summary at end of success, failed, skipped etc
  • Automation of the Json module listing to add to your Sitecore.json file
  • Added support for removing ‘TDS’ from module/namespaces (StripTDSFromName option) – for if you have TDS included in the name of your TDS projects and want to remove it.
  • Added support for Filtering on just  specific project name (ProjectNameToMatch option) – to allow running on a single project/testing. Leave blank to include all Projects.
  • Added a new SkipCreateIfExists option to not create the Json file if it already exists.
  • I also stole some of Aaron’s exclude list and added that :-).

More info below about how to use this.

Steps to migrate to SCS from TDS

The following steps give you an overview of the process I followed to migrate to SCS. It’s probably not the only way to do this but it worked well for me and is likely pretty close to the approach you will need to take to do this:

  1. Use Sitecore Serialization Converter to create new SCS Json module files from your existing TDS projects
  2. Update the Sitecore.Json file to include the new modules
  3. Install the Sitecore CLI and run the dotnet sitecore init command to create the Sitecore.json file.
  4. Install the SCS Plugin using the following command:
    dotnet sitecore plugin add -n Sitecore.DevEx.Extensibility.Publishing
  5. Login the CLI to Sitecore
  6. Validate the module files with the command dotnet sitecore ser validate -v and modify them as required
  7. Run dotnet sitecore ser pull -v to sync your items to disk
  8. Resolve any issues with running the pull commands
  9. Push the new modules and .YAML files to source control
  10. Remove TDS projects and TDS item files from the solution
  11. Test the sync to another environment using the -whatif param
  12. Update your release pipeline to remove the TDS sync and add the SCS sync (a blog post will be coming soon on this)

Note: if your struggling with setting the CLI then this is a really useful guide.

I’ve included some tips below on using the SCS commands.

Using Sitecore Serialization Converter

  1. Pull the project from GitHub
  2. Open the project in Visual Studio
  3. Edit the appsettings.config file as follows:
    ProjectDescription – set this to your project name, it will be added to your Module files.
    SolutionFolder – set this to the root of your Solution folder. TDS projects will be found from here.
    SavePath – you probably only want to set this for testing so all module files are created in the same place (use relative path instead to create them in each feature/foundation/project folder).
    UseRelativeSavePath – set to false when testing using the SavePath above, but true when your ready to create the module files in a relative location.
    RelativeSavePath – set this if UseRelativeSavePath is set to true above. For me this path was ../.
    StripTDSFromName – set to true if you have the text ‘TDS’ in your project names and it will be removed when creating the .Json files.
    ProjectNameToMatch – set this to filter on one or more projects. Useful for testing or for just targeting specific projects. Leave blank to find all TDS projects.
    SkipCreateIfExists – set to true if you want .Json files to be skipped instead recreated if they already exist. Useful if you’ve run SSC a few times.
  4. Either Run the project in debug mode in Visual Studio OR Build it and go to the .exe in the output folder via command line and run the .Exe like so:
    SitecoreSerialisationConverter.exe
  5. Check the console/logs for any issues or errors, you should find these in the debug folder
  6. Check the summary looks correct in the Console and the Logs and no projects or important items are missing
  7. Copy your Module list from the output in the Console to use in your Sitecore.json file, e.g:

    "modules": [ c:/temp/MyProject.Project.Master.module.json",
    c:/temp/MyProject.Feature.Banner.Master.module.json",
    c:/temp/MyProject.Foundation.Caching.Master.module.json
     ],


Tips on Serialize Commands

  • Verbose – Run all the commands with -v after them so that you get full debugging of any issues. e.g: dotnet sitecore ser validate -v -i MyProject.Feature*

  • Filters – Use the -i param to filter by a specific project name pattern, e.g: dotnet sitecore ser pull -v -i MyProject.Feature* – will match all projects starting with ‘MyProject.Feature’

  • Validate – Use the validate command to check the your Module Json file paths are valid. e.g: dotnet sitecore ser validate -v. You can combine this with -i to validate just one project at a time.

SCS Issues and Errors

I hit a bunch of errors when trying to run the pull and/or validate command. I think I had perhaps 100 errors or more to resolve across around 60 converted TDS projects so it took me quite some time to resolve them all. It’s definitely easier if you target on Project at a a time, fix the issues, commit to source control and move to the next though. Here are the issues I tracked and what I had to do to fix them.

Duplicate Paths cannot be serialized

If this happens it means you have multiple module files which have references to the same item/path. You need to find the duplicate references and remove the path from all but one Module file and try and run the command again.

Non Unique Paths cannot be serialized

If this happens it means an item with the same name exists more than once in Sitecore and it’s trying to serialize it to disc as an YAML file but can’t create the item more than once with the same path name.

To fix it remove one or more copies of the item from Sitecore and try and run the command again. You will need to work out which is the correct one to remove. I used SPE to remove items by ID to speed this up.

Serialized item contained a blank value

You will see an error like so if this happens:

Check your serialised item, you will see there is no language set. Then check the language versions in Sitecore and look for an Invariant Language. I found I had to run an SQL command on my master and core database to remove version languages:

DELETE FROM [sitecore_master].[dbo].[VersionedFields] WHERE [Language] = ''

DELETE FROM [sitecore_core].[dbo].[VersionedFields] WHERE [Language] = ''


Blank Database Values

There were some situations where the database in my Json files ended up empty for some reason. I did make some updates to the Sitecore Serialization Converter to fix this by defaulting the value to Master if the Project name contains ‘Master’ and Core if it contains ‘Core’. But if this happens to you then you will need to fix this manually in the Json files else SCS will throw errors:

Cache already contains GUID

I also had some errors related to items already existing in the cache, something like:

“/sitecore/content/* (GUID) attempted to be cached, but the cache already contained /sitecore/content/*. Non-unique paths cannot be serialized. Please choose a different item name.“

If you get this error then you should be able to fix it with the following command or perhaps a combination of this fix and the non unique items fix:

dotnet sitecore ser validate --fix

Source Control

One other thing to consider is how you approach adding your changes to Source Control to allow for easy tracking of changes and roll-back etc. I did this my adding all my Json module files first and then pulling each Helix layer (feature/foundation/project) one at a time and pushing the Yaml files in batches.

Next Steps

The next steps for me are adding the integration with the CLI to Github Actions, pushing the items from Source control to other environments and removing the TDS projects and serialized files. I will write about this in a future blog post.

Links

As usual there were a lot of links that helped with this, here are some of them:

https://sitecore.stackexchange.com/questions/36312/how-to-migrate-from-tds-to-cli

https://amitkumarmca04.blogspot.com/2021/07/Sitecore-Content-Migration-Using-Sitecore-Content-Serialization.html

https://spyn.me/sitecore-migrating-from-tds-to-scs/

https://sitecore.stackexchange.com/questions/33034/guid-attempted-to-be-cached-but-the-cache-already-contained-sitecore-c

https://jasonstcyr.com/2023/11/28/converting-a-tds-project-to-sitecore-content-serialization-scs/#:~:text=Migrating%20from%20TDS%20to%20Sitecore,start%20working%20with%20XM%20Cloud!


Hopefully this is useful to others who need to do this.

Excluding Specific Fields from Unicorn Serialisation (Field Filter)

unicorn

This week I had a situation where I needed to include some Sitecore Items in Unicorns Sync but exclude certain fields from those items.  In my case these were navigation items which had a link on them. I wanted to maintain the ids of the navigation item across environments but not the link – as it changes per environment.

I’ve used Unicorn a lot in quite a few Sitecore projects but I’ve never had to do this before so I wasn’t sure if it was possible or not.

It turns out it is, Unicorn has an feature called a ‘Field Filter’.

I can’t work out exactly when this became available but it looks like since version 4.0 Unicorn supports config based filtering.

Luckily we are on Unicorn 4.0.8 so we didn’t need to upgrade.

How do you use it?

Field filters are actually used in the <defaults> node in the Unicorn.config that is provided by default by Unicorn, here is an example of the Last run field being excluded:

<fieldFilter type=”Rainbow.Filtering.ConfigurationFieldFilter, Rainbow” singleInstance=”true”>
<exclude fieldID=”{B1E16562-F3F9-4DDD-84CA-6E099950ECC0}” note=”‘Last run’ field on Schedule template (used to register tasks)” />
</fieldFilter>

Adding the fieldFilter for a specific Configuration

Anything in Unicorn defaults can be overridden/defined at a configuration level  in my case this is what I wanted to do, here is how my custom configuration looks after adding in the exclusion rule for the Home Link field on my Navigation items:

After deploying this update and running Reserialize on this config I could see that the Home Link field had been excluded in the YAML file written to disk :-).

I couldn’t find much on this so thought this might help others who need to do this in future.

Further Options in Unicorn 4.1 – Field Transforms

After posting this Mark Cassidy (who works on Unicorn) tweeted me to remind me about a feature that has just been released and he blogged about it last week:

marks-tweet  

Essentially allowing you to keep certain fields under Source Control, but giving you much more control over what happens to the field values in specific environments using filters.

I haven’t tested this but it looks like this could be used like so to exclude my Home Link field:

In this case I’ve chosen to put the fieldTransforms on the <include> node but if you wanted to apply the filter to the whole config by adding it to the <predicate> instead.

When we’ve upgraded Unicorn to 4.1 I’ll test this and see how it works out.