Thursday, January 31, 2019

Migrating Search cluster in Team Foundation Server

Original Post: 6/25/2018 1:20:27 PM

Migrating the Elastic search (ES) cluster to a new machine can come as a requirement for different reasons -

  • AT scale out: The TFS Admin could have initially setup a single AT  with a local ES instance in the same machine and now wants to scale out TFS by adding new load balancer ATs.
  • ES scale up: ES would have been setup in a separate remote machine and the TFS Admin now wants to provision a larger machine for ES for better performance.

Note: If you are configuring Search in  a multi AT scenario it is strongly recommended to setup ES in a separate remote machine.

Currently, data migration is not seamlessly supported while moving data. (A few manual hacks are required to ensure the the configurations are in place post the migration). Meanwhile, the general recommended procedure is to re-index the collections. The following content explains the step-by-step approach to ensure this migration happens as per expectation.

Configuration Steps

Scenario 1: Single AT + Local ES setup. ES is initially setup locally in the AT machine. Now, ES needs to be setup in a remote machine.
  1. Unconfigure Search from the AT instance.
  2. Setup ES in a new remote machine using the Configure-TFSSearch.ps1 script from package as described here.
  3. Configure Search. Update the Search URL in the configuration wizard page to point to the new URL.
  4. Complete the Search configuration. This will auto-trigger a re-indexing of all the collections where Search extension (Code, WorkItem) is installed.
  5. Refer the blog post here to monitor the progress of the indexing.
  6. If you are plan to add more ATs to TFS, no additional Search configuration is required in those ATs. I have explained the same in the blog post here
Scenario 2: Multi-AT + Remote ES setup. Now, ES needs to be setup in a different machine. (AT topology remains the same)
  1. Unconfigure Search in the primary AT where it was initially configured.
  2. Setup ES in a new remote machine using the Configure-TFSSearch.ps1 script from package as described here.
  3. Configure Search in the primary AT instance. Update the Search URL in the Admin page to point to the new URL.
  4. Complete the Search configuration. This will auto-trigger a re-indexing of all the collections where Search extension (Code, WorkItem) is installed.
  5. Refer the blog post here to monitor the progress of the indexing.
No additional Search configuration changes are required in any of the load balancer ATs.

Both of the above scenarios have essentially the same steps. For clarity I have separated them out and used terms to distinguish primary and load balancer instances.
Validation and Troubleshooting
Until TFS 2018 Update2, there was a separate TFS job that runs and makes necessary updates in the collection databases to ensure this new Search URL is updated correctly. You can check the status of the job through the following query -
SELECT * FROM [Tfs_Configuration].[dbo].[tbl_JobHistory]
where JobId = 'D34ABD0B-E54B-4FA2-A484-926C2F64BC7D'
You need to also check if the properties in the collection databases are updated successfully -
SELECT * FROM [Tfs_Configuration].[dbo].[tbl_JobHistory]
where ResultMessage like '%Updated SearchUrl%'
There should be one job run completion entry for each collection. And it should show up ResultMessages like -
"Updated SearchUrl from {OldESUrl} to {NewESUrl} in {IndexingUnit data}")
To verify that the URLs have been updated correctly in both Configuration DB and all the Collection DBs, check for these entries.
In Configuration DB -
SELECT [RegValue]
FROM [Tfs_Configuration].[dbo].[tbl_RegistryItems]
where ChildItem like '%SearchPlatformConnectionString%'
The output values should be pointing to new ES URL.
In each of the Collection DB (where at least one Search entity extension is installed) -
SELECT EntityType, Properties
FROM [<CollectionDB>].[Search].[tbl_IndexingUnit]
where PartitionId > 0
and IndexingUnitType = 'Collection'
The Properties should have 2 attributes pointing to new ES URL
<IndexESConnectionString>{New ES URL}</IndexESConnectionString>
<QueryESConnectionString>{New ES URL}</QueryESConnectionString>
If any of them is incorrect, the re-indexing has not happened correctly. To mitigate,
  • If the URL in configuration DB is still stale (i.e. pointing to old ES URL), you could reattempt reconfiguration following the same sequence of steps mentioned above for the 2 migration scenarios.
  • If the URLs in the collection DBs are stale, you could reattempt manual trigger of indexing as mentioned in the blog post here. There are 2 possibilities -
    • If the URL is incorrect for all EntityType's properties in the collectionDB, it is suggested to follow the "Clean-up Index Data and Re-index" steps.
    • If the URL is incorrect for specific EntityTypes (say, WorkItem was re-indexed successfully in the new ES instance, but Code wasn't yet), follow the "Collection re-indexing through script" steps. There is one extra step I have added here compared to the steps mentioned in the blog post, hence listing down here once again -
-  Run the SQL query:
    DELETE FROM [<CollectionDB>].[Search].[tbl_IndexingUnit]
    where PartitionId > 0
    and EntityType = '{EntityTypeToBeFixed}' // Code, WorkItem
    and IndexingUnitType = 'Collection'
-  Run this script (pick from the correct TFS release folder): https://github.com/Microsoft/Code-Search/blob/master/TFS_2017Update2/TriggerCollectionIndexing.ps1
-  To monitor the indexing progress, check the blog post here.

Post these troubleshooting measures, if you are still unable to view the Search results post re-indexing, do contact the CSS for assistance.

3 comments:

Installing Azure DevOps Server Search extension from Local Gallery

One of the mistakes that Azure DevOps Server Admins make is installing the Search extension from the Visual Studio Market Place . The Sear...