Doug on SharePoint

Stuff on SharePoint and other things.

Weird XML problem adding AAMs or taking the CA role on a new server

The other day I was upgrading the WFE servers to 64Bit and were rotating some WFEs in and out of the farm. While doing so I was moving the Central Admin around and got a failure running the command psconfig –cmd adminvs –provision.

The exception in the PSCDiagnostics log was this:

10/28/2009 10:48:59 7 ERR An exception of type System.Xml.XmlException was thrown. Additional exception information: Unexpected end of file while parsing Name has occurred. Line 27, position 14.

System.Xml.XmlException: Unexpected end of file while parsing Name has occurred. Line 27, position 14.

at System.Xml.XmlTextReaderImpl.Throw(Exception e)

at System.Xml.XmlTextReaderImpl.ParseQName(Boolean isQName, Int32 startOffset, Int32& colonPos)

at System.Xml.XmlTextReaderImpl.ThrowTagMismatch(NodeData startTag)

at System.Xml.XmlTextReaderImpl.ParseEndElement()

at System.Xml.XmlTextReaderImpl.ParseElementContent()

at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)

at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)

at System.Xml.XmlDocument.Load(XmlReader reader)

at System.Xml.XmlDocument.LoadXml(String xml)

at Microsoft.SharePoint.Administration.SPAlternateUrlCollection.HasMissingUrl(String xml)

at Microsoft.SharePoint.Administration.SPContentDatabase.UpdateAlternateAccessMapping(SPAlternateUrlCollection collection)

at Microsoft.SharePoint.Administration.SPAlternateUrlCollection.UpdateAlternateAccessMappingInContent()

at Microsoft.SharePoint.Administration.SPAlternateUrlCollection.Update()

at Microsoft.SharePoint.Administration.SPAlternateUrlCollection.Add(SPAlternateUrl alternateUrl, Boolean fUpdate, Boolean throwIfExists)

at Microsoft.SharePoint.Administration.SPAdministrationWebApplication.Provision()

at Microsoft.SharePoint.Administration.SPWebServiceInstance.Provision()

at Microsoft.SharePoint.PostSetupConfiguration.CentralAdministrationSiteTask.ProvisionAdminVs()

at Microsoft.SharePoint.PostSetupConfiguration.CentralAdministrationSiteTask.Run()

at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

You will also be able to see these errors in the Application Event log with event IDs 100 and 104 and they essentially say the same thing.

The app pool and the web site provisioned but the Central Admin doesn’t show that server as having that role, the CA even works from that server. While doing some mucking around (that’s a technical term if you don’t know it) I tried to change the AAMs to include the new servers name in the AAM list. Well basically I got the same error just in the “friendly” SharePoint error as seen below

Finally broke down and gave Microsoft a call and after some looking around I have found out that there was an issue that was fixed in the re-release of the SharePoint’s August CU. Well that was fine but what exactly is it that was going on?

You can see in the KB what was going on: http://support.microsoft.com/kb/2000628

In a nutshell, in the central admin content db a field for the AAMs is too short — nvarchar(1023). The AAM XML in the db is truncated anything after 1023 characters and you have a malformed XML and thus the end of file exception.

This was causing a problem adding the CA because it needs to read that field and update it…This will also be a problem with any web apps which the AAM XML exceeds the 1023 limit – you would reach this pretty quickly if you added 4-5 AAMs for your sites. Here is an example of the XML, I formatted it to make it easier to read:

<AlternateDomains Count=”4″ Name=”Some Web App”>
<AlternateDomain>
<IncomingUrl>http://SomeIncomingURL#1.com</IncomingUrl&gt;
<UrlZone>Default</UrlZone>
<MappedUrl> http://SomePublicURL.com </MappedUrl>
</AlternateDomain>
<AlternateDomain>
<IncomingUrl> http:// SomeIncomingURL#2.com </IncomingUrl>
<UrlZone>Default</UrlZone>
<MappedUrl>http://SomePublicURL.com</MappedUrl&gt;
</AlternateDomain>
<AlternateDomain>
<IncomingUrl>http:// SomeOtherIncomingURL#3.com </IncomingUrl>
<UrlZone>Default</UrlZone>
<MappedUrl> http://SomePublicURL.com </MappedUrl>
</AlternateDomain>
<AlternateDomain>
<IncomingUrl> http://SomeOtherIncomingURL#4.com </IncomingUrl>
<UrlZone>Default</UrlZone>
<MappedUrl> http://SomePublicURL.com </MappedUrl>
</AlternateDomain>
</AlternateDomains>

That was about 870 characters, give-or-take. You can see how we can reach the 1023 limit pretty easily.

Disclaimer: DO NOT DO THIS ON YOUR PRODUCTION FARM WITHOUT GETTING APPROVAL FROM MICROSOFT SUPPORT OR YOU MIGHT BECOME UNSUPPORTED!

Now the workaround (approval was given) was to run the following on the Central Admin Content Database (needed to do this on the CA content DB because adding the new Central Admin was failing) Delete from DatabaseInformation where Name = ‘AlternateAccessMappingXml’

That was it…deleting from the DB allowed the process to read the XML from the field (which wasn’t there) and continue without error. Now if you still have AAMs that will cause the XML to exceed 1023 then it will still be truncated but the write operation will still succeed and you can go about your business. Next time there will still be an problem

Something to note: If you are consolidating a lot of portals into one central SharePoint portal and you plan to point your old URLs to the new portal, you should consider updating the system to at least August CU.

Advertisements

10/30/2009 Posted by | Administration, SharePoint, Upgrade | , , | Leave a comment

Moss Object Caching

First I wanted to thank Sean McDonough for his blog post MOSS Object Cache Memory Tuning is not an Intuitive Process it assisted a great deal with a stability problem we were having with a publishing portal.

So here was the problem: The company has an internal portal (yes it is on a 32 Bit machine) that is getting around 65-70K unique visitors per day. It wasn’t until the last couple of weeks that the traffic was that high, but due to a DNS change the old portal is now pointing to the new SharePoint Site. Soon after the change, the site just was not stable at all. We were seeing a lot of app pool resets, slow response, and a lot of out of memory errors.

After doing some searching we found that the Object cache settings was set to 1GB (the default is 100MB)! You can read about SharePoint caching here on TechNet. The thing is, the object cache sits in the worker process and on a 32 Bit system there is only the 2GB application memory space. With the added pressure on the application memory it just could not handle it.

Using Sean’s blog and his recommendation of looking at the SharePoint Publishing Cache/Total number of cache compactions, we reduced the cache to 300MB (it was set at 500MB as an arbitrary number that the application team would agree to) watched it for a day and we saw an increase in the compactions to 2-3, at 500 the counter was at 0. We then increased the memory to 350MB and watched it for a day and continued until the optimal setting was found for the Object Cache.

Imagine what would happen if you has a single web application for publishing sites? Every site collection admin could change this setting. Now just because the setting is at something like 500MB doesn’t mean that they will automatically reserve that amount of memory, it just means that if needed the cache will grow to that amount but on a 32 bit machine it would have a great impact on all of the sites in the web app. This got me thinking, how can I control this? I could not find a global setting in the central admin nor could I find an STSADM command (out of the box) to run. PowerShell!

I created a PowerShell script that I could use to loop thru the site collections on a web app and set the cache back to the default (or whatever you want it to be). Here is what I did.

[System.Reflection.Assembly]::LoadWithPartialName(“Microsoft.SharePoint”)

[System.Reflection.Assembly]::LoadWithPartialName(“Microsoft.SharePoint.Publishing”)

$site = new-object Microsoft.sharepoint.spsite(“http://mosssite&#8221;)

$webapp = $site.Webapplication

foreach ($sites in $webapp.sites){

$cacheSettings = new-object Microsoft.SharePoint.Publishing.SiteCacheSettingsWriter($sites.url);

if ($cacheSettings.objectcachesize -gt 100){

$cacheSettings.ObjectCacheSize = 100;

$cacheSettings.Update();

$sites.url;

}

}

Note: dont forget to dispose!

09/18/2009 Posted by | Administration, Powershell, SharePoint | , | 1 Comment