Installing Azure DevOps Server Search extension from Local Gallery

One of the mistakes that Azure DevOps Server Admins make is installing the Search extension from the Visual Studio Market Place. The Search extensions in Market Place are compatible only with Azure DevOps Service account. For Azure DevOps Server (i.e. OnPremises), the extension need to be installed from the Local Gallery page. This has been well-documented here.

In case you have already installed the incorrect extension, refer to the following sequence of steps to fix it:

NOTE: These steps are applicable for all TFS versions where Search is supported. Hence, replace \Azure DevOps Server 2019\ with the appropriate TFS version folder based on your current installation.

• Uninstall the incorrect extensions for each of the collection where it's already installed. For that, follow the guidelines mentioned in this blog. It is important to get the database into a clean state before we can proceed with the extension install.
• Go to the local gallery management page: http://<serverurl>/tfs/_gallery/manage and upload new extension.
• Choose the VSIX package from \Program Files\Azure DevOps Server 2019\Tools\Deploy\TfsServicingFiles\Extensions for the correct entity (Code, WorkItem, WIKI) - 
• "\Program Files\Azure DevOps Server 2019\Tools\Deploy\TfsServicingFiles\Extensions\Microsoft.TeamFoundation.Extension.Search.Web.Code.vsix"
• "\Program Files\Azure DevOps Server 2019\Tools\Deploy\TfsServicingFiles\Extensions\Microsoft.TeamFoundation.Extension.Search.Web.WikiOnPremises.vsix"
• "\Program Files\Azure DevOps Server 2019\Tools\Deploy\TfsServicingFiles\Extensions\Microsoft.TeamFoundation.Extension.Search.Web.WorkItemOnPremises.vsix"
• Once the vsix package is uploaded, verify that the extension shows up in the local gallery page i.e. http://<serverurl>/tfs/_gallery
• Install the specific extension for a collection from the local gallery.

Extension install triggers a re-indexing of that entity (Code, WorkItem, WIKI) for that collection. You may now monitor the indexing progress for that collection as mentioned in the blog here.

If the Search still does not work as expected post that, try the Repair-Search script for further troubleshooting and mitigation.

Upgrading Search in TFS 2018 Update 3

Original Post: 10/6/2018 9:40:45 AM

In TFS 2018 Update 2 and 3, Elasticsearch support was updated to ES 5.4.1. If you are upgrading from any earlier version of TFS, a Search re-indexing will be required. TFS 2018 Update 3 also introduced basic auth support for connecting to Elasticsearch. To sum up the matrix –

Pre-TFS 2018 Update 2  >> TFS 2018 Update 2 : Re-indexing required

Pre-TFS 2018 Update 2  >> TFS 2018 Update 3 : Re-indexing required + Basic auth support for ES

TFS 2018 Update 2  >> TFS 2018 Update 3 : Basic auth support for ES

Performing an upgrade should be straight forward. I’ll however cover a bit of internals here and some guidance to steps in case the upgrade does not go as expected. The flow I’ve explained below is one of a remote Elasticsearch scenario (i.e. ES and TFS are installed in separated machines).

General Upgrade flow –

• Unconfigure Search and Application Tier
• Install new version of TFS
• Configure Application Tier (on all AT machines in case it’s a multi-AT setup)
• Configure Search on the primary AT machine (refer this blog post on configuring Search in multi-AT scenario)

While configuring, you need to opt-in to install and configure search.

The existing Search Service URL (i.e. Elasticsearch URL) will be pre-populated. Follow the instruction as mentioned to copy the package to the Elasticsearch machine and run the Configure-TFSSearch.ps1 script:

.\Configure-TFSSearch.ps1 -Operation update -User {username} -Password {password}

The script will stop the existing ES service, rename the index data folder to something like {IndexDataFolder}.{Timestamp}.old, install the ES 5.4.1 version and start the service.

Input the same username/password in the configuration wizard and proceed with the configuration on the primary AT box.

When the configuration is complete, re-indexing would have been triggered for the entities. You can verify the same quickly in the configuration logs (C:\ProgramData\Microsoft\Team Foundation\Server Configuration\Logs)

[Info   @12:55:16.098] +-+-+-+-+-| Running CodeReindexingJob: Triggering Code Search indexing on existing collections ... |+-+-+-+-+-
[Info   @12:55:16.099]
[Info   @12:55:16.099] +-+-+-+-+-| Triggering Code Search indexing on existing collections ... |+-+-+-+-+-
[Info   @12:55:16.099] Starting Node: CTRIGGERCODESEARCHINDEXINGONEXISTINGCOLLECTION
[Info   @12:55:16.099] NodePath : Container/Conditional/BestEffort/CTRIGGERCODESEARCHINDEXINGONEXISTINGCOLLECTION
[Info   @12:55:16.983] Node returned: Success

(Similar logs will show up Work Item and WIKI entities)

Note, no manual uninstall and re-install of extensions was required as part of the upgrade process. Complete re-indexing duration will depend on the volume of entity data. You can check this blog post on how to monitor the indexing progress.

In the scenario where search is not working post upgrade for any of the users, you can try uninstalling and re-installing the search entity extension. Go through the detailed reset instructions mentioned in the “Re-index at Collection level: Extension Uninstall and Install” section in the this blog post.

FAQs on TFS-Search Admin Operations

Original Post: 11/13/2018 4:57:22 AM

For various TFS Admin operations at Server, Collection or specific Project level, at times there are ambiguities around what additional Search related configurations, settings  update or other manual actions are required for the Search to continue functioning. This post covers a few of those scenarios and the corresponding Search admin related actions (if any) that needs to be performed.

• Which all TFS release version upgrades require re-indexing?
  • TFS 2018 Update 2  is the release when migration to ES 5.4.1 was done. So for any upgrade that spans over this release, a re-indexing is required. For e.g.
    • TFS 2018 RTM >> TFS 2018 Update 3 : Re-indexing required
    • TFS 2018 Update 2 >> TFS 2018 Update 3: No re-indexing required.
  • Also, note that the re-indexing is auto-triggered during the upgrade and Search configuration and no additional manual actions are required.

• How does renaming a Team Collection impact Search?
  • Re-indexing is triggered automatically for the entire Collection. No additional manual action required from Search perspective.
  • Query for ‘BeginEntityRename-CollectionRenameRepairJob’ in [Tfs_Configuration].[dbo].[tbl_JobHistory]’s ResultMessage column to validate that a re-indexing has been kicked off.

SELECT * FROM [Tfs_Configuration].[dbo].[tbl_JobHistory]
WHERE ResultMessage like '%BeginEntityRename-CollectionRenameRepairJob%'

• How does Attach/Detach of a Team Collection impact Search?
  • Refer the blog post here for details.

• How does renaming a Team Project impact Search?
  • Re-indexing is triggered automatically for the entire Project. No additional manual action required from Search perspective.
  • Query for 'BeginEntityRename-ProjectRenameNotification’ in [Tfs_Configuration].[dbo].[tbl_JobHistory]’s ResultMessage column to validate that a re-indexing has been kicked off.

• How does deleting a Team Project impact Search?
  • Documents indexed for all entities (such as Code, Work Item) gets deleted automatically.
  • Query for 'Delete-ProjectRenameNotification' in [Tfs_Configuration].[dbo].[tbl_JobHistory]’s ResultMessage column to validate that a delete operation sequence has been kicked off.

• Does modifying the TFS Service account impact Search?
  • No

• What actions needs to be taken while modifying the Elasticsearch (Search Service) service account?
  • The action itself is possible through a Search re-configure (since there is no alternate setting options exposed currently from the TFS Admin Console). The reconfigure though will be lightweight in this scenario. No re-indexing or any other additional actions are required. Search will continue to work as expected post the account change.

• Does modifying the TFS notification URL impact Search?
  • No

• How does changing GUIDs of Collection/Server impact Search?
  • If you refer to the documentation TFS here, GUID changes are done for specific purpose. For instance, creating a Clone of a server instance and doing hardware migration is one such predominant scenario. All such scenarios require a Search re-configure and re-indexing. Note that Admin operations such as Clone implicitly cleans up the Search related tables in the Collection database and cleans up the Search extensions for that collection too. Hence, consider a scenario where the GUIDs are changed as part of Cloning operation. Post that, the Search configuration on the new server instance with the cloned databases will install the Search extensions and trigger the required indexing. Overall the guidance is, whenever there are GUID changes, Search re-indexing is required.

I will keep updating this post with more scenarios. If you have any scenarios that need clarification, do leave a comment below.

Search administration in TFS for Collection Attach/Detach operations

Original Post: 10/19/2018 6:37:48 AM

Collection detach / attach operations will not require any additional management operation from Search configuration side if it's done on the same TFS release instance. There are however few management operations that you need to aware of for more complex scenarios such as collection migration to a new TFS instance, collection re-attach to an upgraded TFS instance, etc. This post lists out these scenarios and the corresponding actions are expected to be taken by the TFS Administrator.

Collection Detach

Prior to collection detach, you need to uninstall the Search extension(s) from the collection if the detach is done for any of the following purpose –

• It is part of a planned activity to migrate this collection to another TFS instance.
• It is part of a planned activity to re-attach this collection post the TFS instance upgrade, and the TFS upgrade release requires a Search re-indexing. [Refer section below for “TFS Releases which require Search re-indexing”]
• The collection is no longer required and you wish to clean up all collection data from the TFS instance (including Search index data associated with that Collection). This would help reclaim the disk space occupied by the Search index.

Refer the post here on how to clean uninstall the Search extension(s).

You need *not* uninstall the Search extension(s) from the collection if it’s done for any of the following purpose –

• The collection is temporarily detached and will be attached back to the same TFS instance. Avoiding Uninstall and Reinstall of the Search extension(s) in such scenario will save on the re-indexing cost.
• The collection is temporarily detached and will be attached back to the same TFS instance post upgrade, and the TFS upgrade release does *not* require a Search re-indexing. [Refer section below for “TFS Releases which require Search re-indexing”]

Collection Attach

Collection attach does not auto-trigger any re-indexing. However, re-indexing would be required in following scenario(s):

• Collection was migrated to another TFS instance.
• Collection was earlier detached from a TFS instance. The TFS instance is now upgraded to a release which requires Search re-indexing. [Refer section below for “TFS Releases which require Search re-indexing”]
• Search extension(s) were not installed earlier for this collection. Now you wish to enable Search for this collection post attach.

In all of the above 3 scenarios, you need to manually install search extensions (Code, Work Item, WIKI) from the Local Gallery (http://{Server}/tfs/_gallery).

Collection attach does not require re-indexing in the following scenario(s):

• The collection was just temporarily detached and has now been attached back to the same TFS instance.
• The collection was just temporarily detached and has now been attached back to the same TFS instance post upgrade, and the TFS upgrade release does *not* require a Search re-indexing. [Refer section below for “TFS Releases which require Search re-indexing”]

TFS Releases which require Search re-indexing

Following are the TFS release(s) which requires Search re-indexing when upgraded to from an earlier version:

• TFS 2018 Update 2 (requires ES upgrade to version 5.4.1)

[Note: Search feature in TFS was introduced first in TFS 2017 RTM]

Performing a clean uninstall of Search extensions in Team Foundation Server

Original Post: 10/6/2018 8:38:10 AM

TFS currently supports search for Code, Work Item and WIKI. Each of these entities have their own extensions that need to be installed for the feature to be enabled for that collection. Extension install is done automatically during Search configuration (note: Work Item and WIKI search extensions are installed by default, whereas Code search is an opt-in feature). Extension can also be installed later from the Local Gallery page (http://{Server}/tfs/_gallery). Extension uninstall can be done from the “Manage extensions” page (http://{Server}/tfs/{Collection}/_admin/_extensions?tab=manage&status=active).

Extension uninstall could be required for multiple reasons –

• You might not require the feature any longer. Hence, cleaning it up would reclaim disk space taken by that entity index in the Elasticsearch machine.
• As part of troubleshooting, you might need to reset the index. Uninstalling and re-installing the extension is one of the approach (more details in the blog post here)

Extension uninstall does the following internally –

• Removes the search feature (for that entity) from the portal
• Stops all future indexing of the entity data.
• Deletes the index data for that entity in the Elasticsearch (if the ES is still accessible)
• Cleans up all tables entries associated with that entity in the Collection DB.

Getting all the above 4 (especially the last 2) is essential to ensure that future extension installation works as expected. Following are the guidance to ensure a clean uninstall –

• Disable the FaultManagement feature temporarily. That will help speed up the uninstall process (specifically in the scenario where the Elasticsearch cluster is inaccessible; such offline scenarios can potentially occur when the collection properties are pointing to some old cluster). Run the following script on Configuration DB:

declare @features dbo.typ_KeyValuePairStringTableNullable
insert into @features values('#\FeatureAvailability\Entries\Search.Server.FaultManagement\', '0')
exec prc_UpdateRegistry @partitionId=1, @identityName = '00000000-0000-0000-0000-000000000000', @registryUpdates = @features

• Go to “Manage extensions” page (http://{Server}/tfs/{Collection}/_admin/_extensions?tab=manage&status=active) and uninstall the entity extension for that collection

The extension uninstall triggers a sequence of clean up jobs per repository under that collection to delete the indices. In the [Tfs_Configuration].[dbo].[tbl_JobHistory] table, you can see delete jobs cleaning up the ES documents. There will be one job result entry for each repository in that collection.

SELECT [JobId], [StartTime], [Result], [ResultMessage]
FROM [Tfs_Configuration].[dbo].[tbl_JobHistory] as History
INNER JOIN
[Tfs_Configuration].[dbo].[tbl_ServiceHost] as ServiceHost
ON History.JobSource = ServiceHost.HostId
WHERE ResultMessage like '%Delete-SearchExtensionEventNotification%'
ORDER BY StartTime desc

• [This step applies ONLY to TFS 2017 Update 2 and earlier. The uninstall-install synchronization is built into the indexer pipeline in later releases]

Wait for 10-15 min for documents in the index to be cleaned up.

• Ensure all IndexingUnits and ChangeEvents for that entity in the Collection DB are cleaned up (run the scripts in the following sequence only)

DELETE FROM [Search].[tbl_IndexingUnitChangeEvent]
WHERE IndexingUnitId in
(
    SELECT IndexingUnitId FROM [Search].[tbl_IndexingUnit] WHERE EntityType = '%EntityType%' 
)
DELETE FROM [Search].[tbl_IndexingUnit] where EntityType = '%EntityType%'

(where, %EntityType% would be 'Code',  'WorkItem' or 'WIKI' depending on the extension being uninstalled)

• Enable back the FaultManagement feature by running the following script on Configuration DB:

declare @features dbo.typ_KeyValuePairStringTableNullable
insert into @features values('#\FeatureAvailability\Entries\Search.Server.FaultManagement\', '1')
exec prc_UpdateRegistry @partitionId=1, @identityName = '00000000-0000-0000-0000-000000000000', @registryUpdates = @features

• [This step applies ONLY to TFS 2017 Update 3 and later]

Verify that the #\Service\ALMSearch\Settings\IsExtensionOperationInProgress\%EntityType%\Uninstalled either does not exist, or is reset correctly to false.

SELECT *
FROM [<CollectionDB>].[dbo].[tbl_RegistryItems]
WHERE ParentPath = '#\Service\ALMSearch\Settings\IsExtensionOperationInProgress\%EntityType%\' and ChildItem = 'Uninstalled\'

If it is set to 'True', execute the following command to reset it:

declare @registryValue dbo.typ_KeyValuePairStringTableNullable
insert into @registryValue values('#\Service\ALMSearch\Settings\IsExtensionOperationInProgress\%EntityType%\Uninstalled\', 'False')
exec prc_UpdateRegistry @partitionId=1, @identityName = '00000000-0000-0000-0000-000000000000', @registryUpdates = @registryValue

You have now cleanly removed all footprints of the search entity extension from TFS.

Migrating Search cluster in Team Foundation Server

Original Post: 6/25/2018 1:20:27 PM

Migrating the Elastic search (ES) cluster to a new machine can come as a requirement for different reasons -

• AT scale out: The TFS Admin could have initially setup a single AT  with a local ES instance in the same machine and now wants to scale out TFS by adding new load balancer ATs.

• ES scale up: ES would have been setup in a separate remote machine and the TFS Admin now wants to provision a larger machine for ES for better performance.

Note: If you are configuring Search in  a multi AT scenario it is strongly recommended to setup ES in a separate remote machine.

Currently, data migration is not seamlessly supported while moving data. (A few manual hacks are required to ensure the the configurations are in place post the migration). Meanwhile, the general recommended procedure is to re-index the collections. The following content explains the step-by-step approach to ensure this migration happens as per expectation.

Configuration Steps

Scenario 1: Single AT + Local ES setup. ES is initially setup locally in the AT machine. Now, ES needs to be setup in a remote machine.

1. Unconfigure Search from the AT instance.
2. Setup ES in a new remote machine using the Configure-TFSSearch.ps1 script from package as described here.
3. Configure Search. Update the Search URL in the configuration wizard page to point to the new URL.
   
4. Complete the Search configuration. This will auto-trigger a re-indexing of all the collections where Search extension (Code, WorkItem) is installed.
5. Refer the blog post here to monitor the progress of the indexing.
6. If you are plan to add more ATs to TFS, no additional Search configuration is required in those ATs. I have explained the same in the blog post here

Scenario 2: Multi-AT + Remote ES setup. Now, ES needs to be setup in a different machine. (AT topology remains the same)

1. Unconfigure Search in the primary AT where it was initially configured.
2. Setup ES in a new remote machine using the Configure-TFSSearch.ps1 script from package as described here.
3. Configure Search in the primary AT instance. Update the Search URL in the Admin page to point to the new URL.
   
4. Complete the Search configuration. This will auto-trigger a re-indexing of all the collections where Search extension (Code, WorkItem) is installed.
5. Refer the blog post here to monitor the progress of the indexing.

No additional Search configuration changes are required in any of the load balancer ATs.

Both of the above scenarios have essentially the same steps. For clarity I have separated them out and used terms to distinguish primary and load balancer instances.

Validation and Troubleshooting

Until TFS 2018 Update2, there was a separate TFS job that runs and makes necessary updates in the collection databases to ensure this new Search URL is updated correctly. You can check the status of the job through the following query -

SELECT * FROM [Tfs_Configuration].[dbo].[tbl_JobHistory]
where JobId = 'D34ABD0B-E54B-4FA2-A484-926C2F64BC7D'

You need to also check if the properties in the collection databases are updated successfully -

SELECT * FROM [Tfs_Configuration].[dbo].[tbl_JobHistory]
where ResultMessage like '%Updated SearchUrl%'

There should be one job run completion entry for each collection. And it should show up ResultMessages like -

"Updated SearchUrl from {OldESUrl} to {NewESUrl} in {IndexingUnit data}")

To verify that the URLs have been updated correctly in both Configuration DB and all the Collection DBs, check for these entries.

In Configuration DB -

SELECT [RegValue]
FROM [Tfs_Configuration].[dbo].[tbl_RegistryItems]
where ChildItem like '%SearchPlatformConnectionString%'

The output values should be pointing to new ES URL.

In each of the Collection DB (where at least one Search entity extension is installed) -

SELECT EntityType, Properties
FROM [<CollectionDB>].[Search].[tbl_IndexingUnit]
where PartitionId > 0
and IndexingUnitType = 'Collection'

The Properties should have 2 attributes pointing to new ES URL

<IndexESConnectionString>{New ES URL}</IndexESConnectionString>
<QueryESConnectionString>{New ES URL}</QueryESConnectionString>

If any of them is incorrect, the re-indexing has not happened correctly. To mitigate,

• If the URL in configuration DB is still stale (i.e. pointing to old ES URL), you could reattempt reconfiguration following the same sequence of steps mentioned above for the 2 migration scenarios.
• If the URLs in the collection DBs are stale, you could reattempt manual trigger of indexing as mentioned in the blog post here. There are 2 possibilities -
  • If the URL is incorrect for all EntityType's properties in the collectionDB, it is suggested to follow the "Clean-up Index Data and Re-index" steps.
  • If the URL is incorrect for specific EntityTypes (say, WorkItem was re-indexed successfully in the new ES instance, but Code wasn't yet), follow the "Collection re-indexing through script" steps. There is one extra step I have added here compared to the steps mentioned in the blog post, hence listing down here once again -

-  Run the SQL query:
    DELETE FROM [<CollectionDB>].[Search].[tbl_IndexingUnit]
    where PartitionId > 0
    and EntityType = '{EntityTypeToBeFixed}' // Code, WorkItem
    and IndexingUnitType = 'Collection'

-  Run this script (pick from the correct TFS release folder): https://github.com/Microsoft/Code-Search/blob/master/TFS_2017Update2/TriggerCollectionIndexing.ps1

-  To monitor the indexing progress, check the blog post here.

Post these troubleshooting measures, if you are still unable to view the Search results post re-indexing, do contact the CSS for assistance.