We were tasked with splitting one large project collection into two. We split our large (1.3TB) project collection which had about 66 projects including one large (260GB) project into two collections. Everything went fine deleting the smaller (up to 30GB) projects until we got to the largest one.
The deletion process would fail via the GUI and tfsdelete command even after increasing the timeout to 10 hours! When we increased the timeout (we did so gradually – initially to 30 min then 4 hours and ultimately 10) it failed with the following message on version control:
[00:40:17.757] ++ Executing - Operation: ProjectDelete, Group: ProjectDelete.TfsSourceControl [00:40:17.757] +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ [00:40:17.757] Executing step: Delete the team project data from Version Control [00:40:17.757] Executing step: 'Delete the team project data from Version Control' VersionControl.DeleteTeamProject (2 of 11) [01:40:21.473] [Error] TF246018: The database operation exceeded the timeout limit and has been cancelled. Verify that the parameters of the operation are correct. [01:40:21.647] Microsoft.TeamFoundation.Framework.Server.DatabaseOperationTimeoutException: TF246018: The database operation exceeded the timeout limit and has been cancelled. Verify that the parameters of the operation are correct. ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception: The wait operation timed out
Asking the MSDN community generated some suggestions such as:
- Set the ethernet card to Full Duplex
- TCP connections to be allowed on the database server
- Clean the Cache folder on client computer. The folder path is: C:\Users\username(Team Explorer user name)\AppData\Local\Microsoft\Team Foundation\5.0\Cache. (os: Windows Server 2008 R2)
- Clean the Cache folder on Server machine. The folder path is: C:\ProgramData\Microsoft\Team Foundation\Web Access\Cache_v11.0. (os: Windows Server 2008 R2)
- After cleaned, on Server machine, click Start and select Run… to open the dialog box, then input iisreset.exe and click OK, wait it run completely.
While they were good suggestions, how much of a difference would the above advice make when the deletion task takes places on the SQL server is uncertain.
We execute the command on one of the TFS App tiers which then sends the instructions to the SQL server. So in my mind we would have to look at ways of what’s causing SQL to “choke” or basically time out. From what I can see in the log files is that before deletion occurs it takes a snapshot so that if it fails it reverts back. Therefore if you re-run the deletion task it will “resume” but actually start from the beginning if I am not mistaking.
So we had to reach out to Microsoft and have them take a look at the issue. Surely we were not the only ones out there who had to go through this process of deleting a large project, right? 😉
So looking back at the log files we could see that the TFSDeleteProject command had failed in deleting version control. This is because of the fact it is taking such a long time to delete the version control artefacts from the databases and has been cancelled as it is exceeding the time limit.
Deleting from Version Control ... TF246018: The database operation exceeded the timeout limit and has been cancell ed. Verify that the parameters of the operation are correct.
Microsoft suggested as a workaround, to use the tf destroy command to permanently delete (which we were using anyway), version-controlled files from Team Foundation version control. Try to delete maximum number of files as much as possible, (only the unwanted files) and then run the TFSDeleteProject command to delete the team project.
Microsoft notes on the use of the tf_destroy command:
The destroy action cannot be reversed, you must not destroy files that are still needed. Please take a backup of the database before you run this command. Before you run tf destroy without the /keephistory option, we recommend that you first delete the files you want to destroy. Refer: Delete Files and Folders from Version Control. After you delete the files you can synchronize the Team Foundation warehouse; otherwise, the warehouse will not be synchronized with the destroyed items
Other recommendations were to increase the SQL resources – which we did – but it didn’t help. While we thought that the deletion process would revert itself after a failed delete, it apparently doesn’t and remains in a stuck state meaning that you can potentially re-run it, hacking at it until it is all deleted…
Still our issues were that the deletion of the project would take an estimate of 50 days which was not really acceptable. Meanwhile we found out that even though we thought we increased the time-out time for the deletion process, it would still fail after 1 hour.
Two methods for increasing the time-out value for TFSDeleteProject command from within TFS:
In TFS web.config file (C:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\Web Services), you need to restart the IIS after that:
<httpRuntime targetFramework="4.5" <strong>executionTimeout="3600"</strong> maxRequestLength="2000000" requestValidationMode="2.0" maxUrlLength="2048" relaxedUrlToFileSystemMapping="true" encoderType="Microsoft.TeamFoundation.Server.WebAccess.TfsAntiXssEncoder, Microsoft.TeamFoundation.Server.WebAccess.Platform, Version=126.96.36.199, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
- Log on to the application-tier server.
- Choose Start, Run, type regedit, and then choose OK.
- In the browser pane, expand HKEY_ LOCAL_MACHINE:
- If the server runs a 32-bit operating system, expand: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\11.0\TeamFoundation\RequestSettings.
- If the server runs a 64-bit operating system, expand: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432\Microsoft\VisualStudio\11.0\TeamFoundation\RequestSettings.
- If the TeamFoundation\RequestSettings key does not exist, follow these steps to create it:
- Open the context menu for TeamFoundation, point to New, and choose Key.
- Name the key RequestSettings.
- Open the context menu for RequestSettings, point to New, and choose DWORD Value.
- Name the new value DefaultTimeout.
Open the context menu for DefaultTimeout and choose Modify. In Value Data, type the time-out period in milliseconds, and then choose Decimal.
For example, to increase the time-out period to 30 minutes, type 1800000. To change the time-out period back to 10 minutes, type 600000.
- Choose OK.
- On the File menu, choose Exit.
Additionally it also important to note that SQL has its own time-out setting as well which is 60 minutes, so check with your DBA on this!
Well those two methods didn’t work for us, we set the time-out to 100 hours and still fail after 1 hour, so still puzzled about the cause. After some backwards and forwards the final fix suggested did it for us:
DECLARE @registryUpdates typ_KeyValuePairStringTableNullable INSERT @registryUpdates ([Key], [Value]) VALUES ('#\Configuration\Database\VersionControl\Timeout\', 7200) EXEC prc_UpdateRegistry <strong>1</strong>, 'Test', @registryUpdates, 0
Note that the above fix will cause your LOG file to explode so make sure you have the additional space for it!
For us the third suggestion fixed our timing out issue. We are now able to delete the 14,000 branches located in the largest project. We created a small program (batch file) and scheduled it to delete out of hours so that it will not affect performance for the TFS environment.
Some rough estimates are that we are able to delete on average around 615 branches in 12 hours, therefore it should take us no more than 3 weeks to delete all of the 14,000 branches.
In case you are wondering how we were able to have two project collections side by side, we setup another TFS environment so that we can take our time in cleaning up (as we knew the troubleshooting and executing process would take longer than a weekend).