Convert Publish files to Sitecore CLI JSON format

I’m currently working on an Sitecore upgrade for a client and this week I needed to upgrade the Scheduled Publishing Module to be compatible with Sitecore 10.3. And whilst the code had been upgraded recently by Nehemiah Jeyakumar, there was no package for it still.

I was really keen to use an Item as Resource File (IAR) version, but to do so I’d need an Sitecore CLI JSON file which I didn’t have. There was however a Package.xml file which was used to create previous Sitecore Packages for the module.

I wondered if I’d be able to use this to create an Sitecore CLI JSON file but couldn’t find anything online that anyone had done. So I decided I’d write a PowerShell Script to do this for me. You can find this below:

The script essentially loads each x-item entry in an Package.xml file, calculates the item path and name and then generates and JSON file in the Sitecore CLI Serialization format and saves it to disk for you.

How to use it

Open the script in PowerShell ISE on your PC and update the 4 variables below.
The $inputPackageFilePath should be your existing package file and $outputJsonFilePath where you would like to save the JSON file. The namespace and description should reflect your module.

variables - UPDATE THESE 4 AS REQUIRED
$inputPackageFilePath = "C:\Projects\SCScheduledPublishing\Packages\Sitecore Scheduled Publish.xml"
$outputJsonFilePath = "C:\Projects\SCScheduledPublishing\Sitecore.Schedule.Publish-module.json"
$namespace = "Sitecore.Schedule.Publish"
$description = "Sitecore.Schedule.Publish Module Items"

Once you have update these variables you can run the script and all being well you will get a JSON file saved out as specified. You should see an output similar to below with a summary of the number of items created and the location of the JSON file:

You should see that the script will automatically work out which database the item should be put into (from the package path) e.g Master or Core.
Note: The script currently sets all items as ‘SingleItem‘ scope and ‘CreateUpdateAndDelete‘ for allowedpushoperations, so you may want to adjust some of these manually.

After that you can just run the serialize command from your Sitecore Solution folder like so:
dotnet sitecore ser pull

And then the run the Item as Resource command to create one or more .dat files with the items included:
dotnet sitecore itemres create -i Sitecore.Schedule.Publish -o SCScheduledPublishing\src –overwrite

I have a blog post here with some more info on these steps actually (it’s for an older version of the CLI but will work fine): https://www.flux-digital.com/blog/creating-custom-items-resources-sitecore-cli-4-0/

Hopefully this is useful for others who need to do this for a module.

Indecently if you need an Sitecore 10.3 version of the SCScheduledPublishing module you can find an normal and IAR package for 10.3 here: https://github.com/nehemiahj/SCScheduledPublishing/tree/main/Packages

My Sitecore SUGCON 2023 Takeaways – Day 1

I’ve just got back from and awesome weekend in Malaga at Sitecore SUGCON 2023. If you are not aware of what it is, it’s the developer conference organised by the Community each year to bring Sitecore Developers together in Europe. I’ve been to quite a few SUGCONs over the years but I think this has been one of the best. The talks were really interesting and it was great to catch-up with everyone in the Sitecore Community.

I arrived late on Wednesday at the rather impressive Melia near Malaga – right next to the beach. I’ve certainly been to worse locations for conferences. More venue’s like this for future SUGCONs please Sitecore :-).

The roof top bar was particularly special – but there was some learning to do before I would be able to enjoy the pool.

> DAY TWO - you can read about Day Two here.

Day 1

I had a fairly clear idea of the talks I wanted to see and I managed to stick to it pretty well.

Opening Keynote – Steve Tzikakis and Dave O’Flanagan

The following day after a late breakfast I went to register and then to see Dave O’Flanagan kick off the conference and introduce Steve.

Steve couldn’t attend so was on Video link and he explained that given the Economy downturn innovations such as ChatGPT would be key for diversifying product offerings, Steve then hinted that something was in the works with an integration of ChatGPT with Sitecore.

Many companies such as SAP and Oracle have slowed down to adapt to SaaS and Headless; while Sitecore have grown a lot over the past few years. He said that Sitecore has outgrown Adobe for the past 6 quarters with around 20% growth, Sitecore are 2nd place in industry rankings and aim to catch Adobe.

There is an healthy 16% R&D investment and Steve set out his ambitious target of going from 40,000 Sitecore developers to 100,000 in the market. He said that he felt the tough part (moving to SaaS and creating/integrating new SaaS platforms) was now over and Sitecore are ready to move forwards. Sitecore intend to lead by innovation and partners and developers need to be onboard to continue investment and growth.

Dave then took back the stage and stated that Sitecore want to lead with Composable DXP and be the best in the market. He also confirmed that DXP is not going anywhere and 10.4 is currently in the works. There are clients who are restricted geographically with what they can do with SaaS or the want full control of their data and platform so there is still a place for self-hosting with DXP.

Then Dave his us with a bit of very welcome honesty. He said that the SaaS products that Sitecore have procured/integrated and built over the past couple of years are now in a good place but he acknowledged that the documentation, marketing, information on migration and features is not great and Sitecore are going to work on this ASAP. This was great to hear as I think that there is some confusion right now for current customers and potential new customers with all of the different SaaS offerings and XP/XM. I feel It’s quite tricky to understand and the messaging from Sitecore needs improving, especially around the migration path to SaaS.

Dave went on to say that there are no new product announcements this SUGCON, instead Sitecore will slow down and help customers understand the new products better and explain how to migrate to them. It was good to hear that Sitecore understand there are many customers heavily invested in XP and that it’s not that easy for them to just jump ship to SaaS and that they want to know more about how they can help customers with this journey.

The discussion then turned to the Content, Engagement and Commerce clouds and the work Sitecore have done here with huge investment, especially in Content Cloud – of which XM Cloud is key part and is getting better by the day.

Content Hub 1 is Sitecore’s answer to a full headless CMS and they will work on integrating it into XM Cloud to allow you to pull content into XM Cloud seamlessly. Content Hub 4.3 is the last version of Content Hub and all customers are now updated.

Search is a new SaaS offering from Sitecore, it is developed on top of Discover (a SaaS platform Sitecore purchased, originally called Reflektion). It has now been developed further and is able to search all content and is powered by AI. I would learn more about this at sessions later in SUGCON.

When looking at the cost of XM Cloud customers need to consider the TCO of their existing XP/XM (or other DXP platform) and not just the licence, hosting and development costs. I think it’s true that many clients probably don’t consider the ongoing maintenance costs of Azure Infrastructure / AKS / Solr / Redis etc and the cost and complexity of Upgrades, Security patches and so forth. Not to mention the cost of DevOps/Build Pipelines and other services. When you add this all up the cost may be around the same or a bit cheaper. One of the challenges here I think is different budgets in organisations that traditionally just pay for the hosting or licence etc so this may be something to navigate when it comes to the new world of SaaS.

Dave then went on to talk about XM Cloud a bit further and that they do know there are challenges without having an CD instance anymore in XM Cloud (these are instead replaced by Edge Servers – experience edge which just returns the items via the layout service or GraphQL). He said that it is something they are trying to resolve and will look to the Community so potential solutions to this. This sounds interesting so I’ll be keen to follow this idea further and see where it goes.

He mentioned the impressive looking Component Builder in Pages (which I also attended a session on later at SUCON). Dave also discussed that Forms in XM Cloud is something they are working on currently; they have recognised it’s very important and are listening. This is a bit of a gap in the XM Cloud offering at the moment and some clients won’t be keen to use a 3rd-party option for this, so it’s good to hear this is in progress.

Another interesting statement Dave made was that the feedback from front-end developers was that JSS is Complex and not something they really want to learn, it has a lot of complexity baked in which some feel is not needed. Therefore, Sitecore are looking at how they can simplify this. I wasn’t exactly sure what was meant by this (I’ll try and find out more) but reducing complexity and barriers to entry is generally a good thing I feel.

Finally Dave discussed Sitecore Connect which is one of Sitecore’s most recent SaaS offerings which allows SaaS products to be integrated via a low code / no code approach.

Dave mentioned connect recipes will be provided to help customers to move to SaaS and these could be taken and customised to fit your requirements. I’d seen a bit on this and how it works from other Sitecore Developers such as this great post by Jeremy Davis. However, I was looking forward to learning more about how it all works later at SUGCON.

Life at the Edge with Vercel and Next.js – Javi Velasco

Next up was a partner talk from Vercel about their platform and different offerings. Javi explained how customers expect a lot more now in 2023 (faster, more dynamic & personalised) and the pandemic effectively jump technology advancements and expectations forward by two years to 2025.

He talked about how computing and innovations in compilers has improved vastly in recent years and how Edge workers can now execute code extremely fast providing similar performance to Static Page Generation as well as Edge Middleware.

I’ve not used Vercel yet but I’ve heard good things about it and Next.js (which they also created) and I know that Sitecore partner with them for XM Cloud so it was interesting to learn more about it all.

Accelerate website builds with Headless SXA and XM Cloud – Martin Miles

We then needed to select our talks for the afternoon. I decided to pick Martin Miles’s talk about XM Cloud and while I’ve learned a fair bit about XM Cloud in the past 4 or 5 months and spoken on it at both the Manchester and Columbus SUGs I still learned a lot from Martin’s talk.

I know Martin plans to share his slides on his blog and the slides are very detailed. So I’ve tried to pick out some key slides which I thought were really useful or had important information that I hadn’t really seen detailed elsewhere.

Auto update and upgrade information & Licensing model:

Some additional limitations I wasn’t aware of:

Further architecture details:

More details about Webhook event handlers:

Some GraphQL details and limitations:

More authentication and authorisation details:

Some more details on Embedded Personalisation & Analytics from CDP:

Docker development details:

Sitecore CLI details:

XM Cloud folder structure, files and folders overview:

I hadn’t really though about how XM Cloud deploys items but it makes sense it uses Items as Resource Files:

SPE usage in XM Cloud:

Some useful developer tips and tricks:

Some really useful migration details for headless solutions:

Two different MVC migration routes:

Lastly some really useful Headless SXA details:

Once Martin share his talk I’ll link it here as there was lots more information that was useful.

SXA MVC & Headless SXA – a MOVING tale… – Jason Wilkerson

I’ve known Jason for quite a while and he’s always an entertaining speaker so I was looking forward to this one. It didn’t disappoint and Jason started with a story about Hipster developers…

Being an predominantly back-end/c#/.Net developer and coming from the Microsoft stack I can really relate to this too. All this new-fangled hipster FE/Headless development is kind of ‘mind boggling’ and a big shift in thinking for those of us who’ve been around since the WebForms (or in my case classic ASP days).

I’ve done some React development with Sitecore and the JSS Training course but I’m still not 100% comfortable with the shift yet.

Jason’s talk was great as he showed us how you would go about building an example ‘Spotlight’ component in Class SXA/MVC and how that differs when building an JSS Headless component in React.

Here are a couple of slides from building the MVC Component, I think this is pretty well understood by most Sitecore Devs:

There were some learnings that Json Shared with us around the differences with doing this for JSS instead:

The rendering variants setup is quite different for MVC:

Then the JSS variant, this looks quite a nice approach:

There are limitations of JSS Renderings which are that the search component of SXA are not available, also if your using SSG then you can’t use forms. There are also personalisation restrictions too:

Lastly Jason also had some training links he shared for those new to this and need to know a good place to start.

XM Cloud and Content Hub ONE Battle Royale – Rick Bauer & Richard Seal

The final session of the day for me was Rick and Richards talk which was positioned as a battle between the two platforms. It made for a pretty fast-paced and entertaining talk.

Pretty much all the info was on the slides so I’m going to just drop them all below:

At the end there was an final summary that confirmed that XM Cloud and Content Hub one are different products and are positioned separately in the market to meet different requirements:

End of Day 1

It was then time for Dinner, catching up with friends from the Sitecore Community and the MVP awards ceremony. Unfortunately the MPV awards had got stuck in Customs again (despite Tamas’s best efforts) but we got a few photos and there was an entertaining performance from Rodrigo and Sebastian and also the quiz; followed by a few well earned beers.

You can read about what I learned on day two here.

Creating Custom Item as Resources Files with Sitecore CLI 4.0

Back when Sitecore 10.1 was released in February Sitecore include a significant change to how the out of the box items were shipped with the release. They were no longer stored in the databases and instead are shipped as ‘items.x.dat’ files in the /App_Data/items folder. These files are in a new Protocol Buffers (ProtoBuf) format, referred to as ‘Items as Resources’ and make upgrades simpler. Jeremy has a really good over-view of this if you want to learn more about it.

However until recently it was not possible to create your own custom ProtoBuf files to deploy your items as it wasn’t supported in Sitecore 10.1 or using Sitecore CLI 3.0.

Sitecore CLI 4.0 was released at the end of September though and now includes a ‘Sitecore ResourcePackage plugin’ for doing this.
Note: it is only possible to use Items as Resources with 10.1 and 10.2.

TLDR

Use Sitecore CLI 4.0 to Serialize your items to disk. Then run the sitecore itemres create command to create a .dat file and deploy it to another environment or Solution.

Ok but why would we want to do this?

I recently released a Module called Sitecore Sloth and for an update to it (coming soon) I wanted to add a settings item in Sitecore for it. I figured this would be a good opportunity to see if I can use the new Items as Resources functionality in my Module package. I haven’t seen anyone blog about doing this in the supported way yet so here goes.
It could be really useful for Deploying developer owned resources to other Sitecore instances and Containers and keeping the database clean. No deployment scripts are needed for the .dat file install as they are auto-installed on Sitecore start-up.

Sounds good how do I do it?

So the first thing you need to do is install the new Sitecore CLI 4.0, if you have any trouble with Sitecore’s guide then this is an really easy blog post to follow and includes useful info on setting up the serialization. If you have CLI 3.0 installed already you can upgrade it.

Prerequisites – Serializing your Items to disc

Assuming you now have the CLI setup and authenticated you need to do the following to Serialize your items. If you’ve done this already then skip this section:

1. Ensure you have an Sitecore.json file at the root of your solution with the new Sitecore.DevEx.Extensibility.ResourcePackage included as well as the Serialization one:

{
 "$schema": "./.sitecore/schemas/RootConfigurationFile.schema.json",
 "modules": [
 "src/*/*/*/*.module.json"
 ],
 "plugins": [
 "Sitecore.DevEx.Extensibility.Serialization@4.0.0",
 "Sitecore.DevEx.Extensibility.Publishing@4.0.0",
 "Sitecore.DevEx.Extensibility.ResourcePackage@4.0.0"
 ],
 "serialization": {
 "defaultMaxRelativeItemPathLength": 100,
 "defaultModuleRelativeSerializationPath": "items",
 "removeOrphansForRoles": true,
 "excludedFields": []
 }
}

Mine is here:

2. Add a module JSON file for your Feature or Module like so:

3. Configure the JSON as required. I’ve included my Sloth data template and root item in mine:

{
 "namespace": "FluxDigital.Foundation.Sloth",
 "items": {
 "includes": [
 {
 "name": "templates",
 "path": "/sitecore/templates/Foundation/Sloth",
 "allowedPushOperations": "createUpdateAndDelete",
 "scope": "itemAndDescendants"
 },
 {
 "name": "sitecore sloth",
 "path": "/sitecore/system/Modules/Sitecore Sloth",
 "allowedPushOperations": "createUpdateAndDelete",
 "scope": "SingleItem"
 }
 ]
 }
}

4. CD into your module or feature directory and Run the ser pull command to serialize your items to disc:

dotnet sitecore ser pull

You should see the .yml files on disc in an items folder within your module / feature:

Creating the Item as Resource file

Ok so now were ready to create our Item as Resource (.dat) file.

1. Ensure your command prompt is still within the Solution route and run the itemres create command like so:

dotnet sitecore itemres create -i FluxDigital.Foundation.Sloth -o src\Foundation\Sloth\code\App_Data\items\master\sloth –overwrite

Note: the -o param to choose the output folder for your .dat file.

You should then see you have a file saved to your output folder like so:

Deploying your items as resource file

So the final step is to deploy the .dat file to another environment (e.g UAT or Staging). In the real world this could be via NuGet or inside a Sitecore package or a deployment pipeline.

I’m going to include this file in my the Sloth 1.1 release via NuGet and Sitecore packages for Sitecore 10.1 and 10.2. You can see the contents of it here (my items.master.sloth.dat file is inside the /App_Data/items/master/ folder):

To test this I installed this NuGet package to a new Sitecore 10.2 instance I setup locally and auto-magically the items appeared:

pretty cool huh?!

Hopefully this is useful for others interested in using this new feature.

Sloth 1.1 will be released later this week hopefully using this new feature to ship the items for Sitecore 10.1 and 10.2.

Sitecore Symposium 2020 Highlights

On Tuesday and Wednesday last week it was Sitecore Symposium 2020. This year it was all on-line so was a bit different and that was great in terms of catching up on all the content but not so great from a networking and social perspective.

I normally write a fairly detailed over-view of what I learned in the sessions over the 2-3 days; partly to share what I learned with others but also so I don’t forget what I learned either by the time I get back to the UK.
However this time round as most of the sessions are available on https://sym.sitecore.com/ you can watch a lot of it yourself if you missed it. I had an ‘All Access Pass’ so you will need to upgrade to view some of the content I saw.

So instead I’m going to summarise the 6 key themes over the 2 days instead with some links to key sessions I watched. I’ve kept the slides below to a minimum, but please check out my Twitter feed for more screen shots.

1. Pandemic Impact & Agility

The Pandemic has changed how customers expect to interact with brands and companies have had to respond very quickly to rapidly changing customer needs. In the opening Keynote and theme throughout Symposium was that a “Moment to Moment” Mindset is what is required to meet customers needs in these moments and how Sitecore’s innovations can help with achieving this.

Brands need to move a lot faster in the current climate and solve their internal operational challenges responds quickly to customers.

2. Auto Personalisation

The new Sitecore CEO Steve Tzikakis announced that all Sitecore 10 customers would receive access to Auto Personalisation Standard at no additional cost. This is clearly a key innovation for Sitecore going forwards.

There were some demos of this in action across the various sessions and It looks pretty cool. It could certainly take away a lot of the heavy lifting needed for personalisation and allow content editors to focus on other tasks configured and trained correctly.

We saw an example of how this works, how you can identify (and fix) customer ‘clusters’ which have gaps in personalisation and an example of the dashboard you would see showing performance and other metrics.

3. Content as a Service (CaaS)

Steve Tzikakis also announced that CaaS would be available in both managed cloud and Public Cloud. CaaS will deliver Sitecore content to any digital channel required, providing direct access to content in a decoupled way, allowing for faster development and delivery.

In a later CaaS preview session by Alistair Deneys & Andy Cohen we were given a lot more insight into how this will work and also told that it will not require Content Hub to work either as CaaS can be used directly against Sitecore too (via the JSS Layout Service). It’s great to see a full Headless offering from Sitecore using GraphQL to access the data and support for Next.Js too.

4. Content Hub

There were a lot of mention of Content Hub at Symposium this year, Jake the compère and comedian (who was pretty funny still despite having no live audience) joked that we should drink each time we here it. I guess it makes sense we’d hear it a lot as Content Hub is a the Centre of Sitecore’s SaaS approach.
For those that don’t know Sitecore Aquired Content Hub (then Stylelabs) in 2018 and it both an DAM, PCM and CMs with lots of impressive functionality for managing and organising content for all channels in a single location.

It was really interesting is to see how Sitecore are using Content Hub for Sitecore.com and the challenges they faced.

One of the other sessions I watched on Content Hub was from Fellow Sitecore MVP Akshay Sura from Konabos and Sumith Damodaran from Sitecore. This talk was regarding how the new Content Hub Connector allows Sitecore Customers to integrate content from Sitecore XP into Content Hub where they can organise, collaborate and personalise content before distribution.

It was interesting to see how Sitecore fields are mapped on the Sitecore Template to the fields in Content Hub in the CMP connector and then how this displays in Content Hub.

CMP is available for Sitecore 9.2 and 9.3 currently and it may be merged with the DAM Connector in an upcoming release.

5. Containers are the Future

Container support was released as part of the new DevEx with Sitecore 10 and it will soon become the defacto way to develop locally for Sitecore. Whilst Containers do bring some complexity they also provide a lot of benefits such as portability, consistancy, being able to quickly destroy and re-create Sitecore instances and having multiple client instances or versions on a single machine.

However another one of the drivers for using containers is that they can be used in Production and you will soon be able to deploy to AWS, Google Cloud, your own Cluster or AKS! This means you will no be tied to Azure App Services.

As I understand it App Service Support will be marked as obsolete in Sitecore 10.1 and removed in Sitecore 10.2. This isn’t that Sitecore won’t work in App Services anymore but more that ARM templates will not be provided anymore and Support for Sitecore in App Services may be limited.

The session by Bart Plasmeijer and Rob Earlam on how to create an AKS Cluster, Windows Node Pool & deploy an Sitecore instance into it was really insightful. They managed to do all this within 20 minutes.

It was also really interesting to see some of the tooling in action like
@k8slens and K9s showing how they work and the features they provide.

Screenshot 2020-10-28 115726

There is another on-demand session on this too which was really informative around what the benefits are for Kubernetes for various roles in an organisation.

6. New Sitecore DevEx

A significant part of the Sitecore 10 release was the new Sitecore Development Experience (DevEx).
As well as containers for development one of the other new features is the availability of an new serialization option, which comes as standard with Sitecore. There are two flavours of this: the Sitecore CLI (using JSON) Or the GUI (Sitecore for VS). The CLI is free where as Sitecore for VS requires an (TDS) licence.

I saw an demo of this and it looked really interesting. I’ve tried it out myself too and it works really well. I’m interested to see if there are any features missing that unicorn or TDS have. It does look like Sitecore have most things covered though with the rules support.

Another part of the new DexEx is the new ASP.Net Core Rendering SDK which I saw an session on. This is a new way of building Sitecore sites in Sitecore 10.x, allowing you to use Layout Service to return content from Sitecore to display using .Net Core Components using Tag Helpers.

This approach has full support for Sitecore features such as editing in Experience Editor and also Analytics and Testing.

Nick also explained the .Net Core Request Pipeline works with the SDK, this is shown in the diagram below. It looks a little complicated but makes sense once you break it down.

There is a lot of new stuff to learn here but it’s great to see .Net Core support and another option available for Sitecore development. I’m learning more about this currently and plan to share what I learn in the near future.

It’s a wrap

The closing keynote with Leslie Odom Jr was entertaining and there were a lot of other on-demand sessions I watched too. It was really cool to hear that the next Symposium is going to be in Vegas next year :-), Hopefully I’ll see some of you there.

Thanks to Sitecore and the speakers for putting on another great event and for making it so accessible. Lets hope the Sitecore Community can meet up in Person in 2021!.

Tag: Sitecore-CLI