Verify TFS Backup Status

It seems as though when initiating a full backup in TFS (and of course depending on the size of your environment), TFS will report that it has successfully completed the backup even though it is not possible to complete a full backup of a 1TB database in 5 minutes:

414037While in the backup path the .bak files appear it still does not indicate (or possible for) that a backup has completed!

To truly see if a backup is taking place and what its current status is one can execute the following T-SQL query in SQL Server Management Studio:

SELECT command,
s.text,
start_time,
percent_complete,
estimated_completion_time/1000 as "seconds to go",
dateadd(second,estimated_completion_time/1000, getdate()) as "estimated completion time"
FROM sys.dm_exec_requests r
CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) s
WHERE r.command in ('RESTORE DATABASE', 'BACKUP DATABASE', 'RESTORE LOG', 'BACKUP LOG')

Microsoft seem to be aware of this and have suggested that I submit feedback on this issue.

MSDN Forum Source
SQL Query Source

Deletion of Large TFS Project Collection

We were tasked with splitting one large project collection into two. We split our large (1.3TB) project collection which had about 66 projects including one large (260GB) project into two collections. Everything went fine deleting the smaller (up to 30GB) projects until we got to the largest one.

The deletion process would fail via the GUI and tfsdelete command even after increasing the timeout to 10 hours! When we increased the timeout (we did so gradually – initially to 30 min then 4 hours and ultimately 10) it failed with the following message on version control:

[00:40:17.757] ++ Executing - Operation: ProjectDelete, Group: ProjectDelete.TfsSourceControl
[00:40:17.757] +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[00:40:17.757] Executing step: Delete the team project data from Version Control
[00:40:17.757] Executing step: 'Delete the team project data from Version Control' VersionControl.DeleteTeamProject (2 of 11)
[01:40:21.473] [Error] TF246018: The database operation exceeded the timeout limit and has been cancelled. Verify that the parameters of the operation are correct.
[01:40:21.647] Microsoft.TeamFoundation.Framework.Server.DatabaseOperationTimeoutException: TF246018: The database operation exceeded the timeout limit and has been cancelled. Verify that the parameters of the operation are correct. ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception: The wait operation timed out

MSDN
Asking the MSDN community generated some suggestions such as:

  • Set the ethernet card to Full Duplex
  • TCP connections to be allowed on the database server
  • Clean the Cache folder on client computer. The folder path is: C:\Users\username(Team Explorer user name)\AppData\Local\Microsoft\Team Foundation\5.0\Cache. (os: Windows Server 2008 R2)
  • Clean the Cache folder on Server machine. The folder path is: C:\ProgramData\Microsoft\Team Foundation\Web Access\Cache_v11.0. (os: Windows Server 2008 R2)
  • After cleaned, on Server machine, click Start and select Run… to open the dialog box, then input iisreset.exe and click OK, wait it run completely.

While they were good suggestions, how much of a difference would the above advice make when the deletion task takes places on the SQL server is uncertain.

We execute the command on one of the TFS App tiers which then sends the instructions to the SQL server. So in my mind we would have to look at ways of what’s causing SQL to “choke” or basically time out. From what I can see in the log files is that before deletion occurs it takes a snapshot so that if it fails it reverts back. Therefore if you re-run the deletion task it will “resume” but actually start from the beginning if I am not mistaking.

Microsoft
So we had to reach out to Microsoft and have them take a look at the issue. Surely we were not the only ones out there who had to go through this process of deleting a large project, right? ;)

So looking back at the log files we could see that the TFSDeleteProject command had failed in deleting version control. This is because of the fact it is taking such a long time to delete the version control artefacts from the databases and has been cancelled as it is exceeding the time limit.

Deleting from Version Control ... TF246018: The database operation exceeded the timeout limit and has been cancell ed. Verify that the parameters of the operation are correct.

Microsoft suggested as a workaround, to use the tf destroy command to permanently delete (which we were using anyway), version-controlled files from Team Foundation version control. Try to delete maximum number of files as much as possible, (only the unwanted files) and then run the TFSDeleteProject command to delete the team project.

Microsoft notes on the use of the tf_destroy command:

The destroy action cannot be reversed, you must not destroy files that are still needed. Please take a backup of the database before you run this command. Before you run tf destroy without the /keephistory option, we recommend that you first delete the files you want to destroy. Refer: Delete Files and Folders from Version Control. After you delete the files you can synchronize the Team Foundation warehouse; otherwise, the warehouse will not be synchronized with the destroyed items

Other recommendations were to increase the SQL resources – which we did – but it didn’t help. While we thought that the deletion process would revert itself after a failed delete, it apparently doesn’t and remains in a stuck state meaning that you can potentially re-run it, hacking at it until it is all deleted…

Still our issues were that the deletion of the project would take an estimate of 50 days which was not really acceptable. Meanwhile we found out that even though we thought we increased the time-out time for the deletion process, it would still fail after 1 hour.

Two methods for increasing the time-out value for TFSDeleteProject command from within TFS:

Method 1
In TFS web.config file (C:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\Web Services), you need to restart the IIS after that:

<httpRuntime targetFramework="4.5" <strong>executionTimeout="3600"</strong> maxRequestLength="2000000" requestValidationMode="2.0" maxUrlLength="2048" relaxedUrlToFileSystemMapping="true" encoderType="Microsoft.TeamFoundation.Server.WebAccess.TfsAntiXssEncoder, Microsoft.TeamFoundation.Server.WebAccess.Platform, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />

 Method 2

  1. Log on to the application-tier server.
  2. Choose Start, Run, type regedit, and then choose OK.
  3. In the browser pane, expand HKEY_ LOCAL_MACHINE:
    • If the server runs a 32-bit operating system, expand: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\11.0\TeamFoundation\RequestSettings.
    • If the server runs a 64-bit operating system, expand: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432\Microsoft\VisualStudio\11.0\TeamFoundation\RequestSettings.
  4. If the TeamFoundation\RequestSettings key does not exist, follow these steps to create it:
    • Open the context menu for TeamFoundation, point to New, and choose Key.
    • Name the key RequestSettings.
    • Open the context menu for RequestSettings, point to New, and choose DWORD Value.
    • Name the new value DefaultTimeout.
  1. Open the context menu for DefaultTimeout and choose Modify.
  2. In Value Data, type the time-out period in milliseconds, and then choose Decimal.

For example, to increase the time-out period to 30 minutes, type 1800000. To change the time-out period back to 10 minutes, type 600000.

  1. Choose OK.
  2. On the File menu, choose Exit.

Reference: http://msdn.microsoft.com/en-us/library/ms181482.aspx

Additionally it also important to note that SQL has its own time-out setting as well which is 60 minutes, so check with your DBA on this!

Well those two methods didn’t work for us, we set the time-out to 100 hours and still fail after 1 hour, so still puzzled about the cause. After some backwards and forwards the final fix suggested did it for us:

Method 3


DECLARE @registryUpdates typ_KeyValuePairStringTableNullable
INSERT @registryUpdates ([Key], [Value])
VALUES ('#\Configuration\Database\VersionControl\Timeout\', 7200)
EXEC prc_UpdateRegistry <strong>1</strong>, 'Test', @registryUpdates, 0

Note that the above fix will cause your LOG file to explode so make sure you have the additional space for it!

For us the third suggestion fixed our timing out issue. We are now able to delete the 14,000 branches located in the largest project. We created a small program (batch file) and scheduled it to delete out of hours so that it will not affect performance for the TFS environment.

Some rough estimates are that we are able to delete on average around 615 branches in 12 hours, therefore it should take us no more than 3 weeks to delete all of the 14,000 branches.

In case you are wondering how we were able to have two project collections side by side, we setup another TFS environment so that we can take our time in cleaning up (as we knew the troubleshooting and executing process would take longer than a weekend).

Team Foundation Server – Anti-Virus Exclusions

If your company policy is that a anti-virus has to be present on all environments and you happen to be running TFS then here is my list of exclusions which I have gathered from various places across the internet and compiled it together for my own future reference as well being of help to others.

Not only will the exclusions help with troubleshooting but also improve the performance of the servers.

Exclusions

  • C:\Windows\System32\inetsrv\w3wp.exe –  IIS Worker Process or w3wp.exe
  • C:\Users\TFSBuild\AppData\Local\Temp – The temporary work area used by the build server identity, if your build server runs as TFSBuild
  • C:\Program Files\Microsoft Visual Studio 11.0
  • C:\Program Files\Microsoft Team Foundation Server 2010
  • C:\Program Files\Microsoft Team Foundation Server 11.0
  • C:\Program Files (x86)\Microsoft Visual Studio 11.0
  • C:\Windows\Microsoft.NET\Framework\ –  SharePoint Portal & Web Access
  • C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Microsoft\Team Foundation – TFS local cache folder
  • D:\Builds – Build Agent(s) working directory. Look at your Build Agent Properties: Builds -> Manage Build Controllers -> Select Build agent(s) and click on Properties. Whatever the Working Directory is should be addded to your antivirus exclusion list. You should also include your “Drop” directory if you are using that.

Of course locations may vary depending on your setup and environment, but this should give you a good starting point.

Windows 10 Announced and Download

Windows 10, not Windows 9 has just been announced with a release date of 2015! Let’s see some of the features which will be coming with Windows 10.

Features

  • Start menu: The familiar Start menu is back, but it brings with it a new customizable space for your favorite apps and Live Tiles.
  • Everything runs in a window: Apps from the Windows Store now open in the same format that desktop apps do and can be resized and moved around, and have title bars at the top allowing for maximize, minimize, and close with a click.
  • Multiple desktops: Create desktops for different purposes and projects and switch between these desktops easily and pick up where you left off on each desktop.
  • Find files faster: File Explorer now displays your recent files and frequently visited folders making for finding files you’ve worked on is easier.
  • New task view button: There’s a new task-view button on the taskbar for quick switching between open files and quick access to any desktops you create.
  • Snap enhancements: You can now have four apps snapped on the same screen with a new quadrant layout. Windows will also show other apps and programs running for additional snapping and even make smart suggestions on filling available screen space with other open apps

You will be able to download Windows 10 shortly from Microsoft and it would be a good idea to try it out in a VM first even though continuous updates will be coming to make it better, you just know it will not be ready for full-time use.

Firefox Insecure Page Rendering

Since Firefox version 23 websites such as MSDN and Technet have not been displaying correctly like the image below:render Initially I thought Firefox was not rendering the CSS or something was up but it finally got to me that it must be a security setting somewhere which is affecting it. I thought it might have been a certificate relate issue but this advanced command in about:config resolved it.

1. Go to about:config in the address bar
aboutconfig2. Search for security.mixed_content.block_active_content and set its value to false aboutconfigcommand3. You do not have to restart Firefox, you can just refresh the page which was causing the issuetechnetMore information can be read on the Mozilla page.

Romanian Sopcast TV Channels – September

Updated list of Romanian & International TV channels:

1TV Neam? : http://1tv.ambra.ro
Acas? : sop://broker.sopcast.com:3912/80555
Acas? Gold : sop://broker.sopcast.com:3912/74099
Animal Planet : sop://broker.sopcast.com:3912/122855
Antena 1 : sop://broker.sopcast.com:3912/80625
Antena 1 2 : sop://broker.sopcast.com:3912/122832
Antena 1 3 : http://86.105.253.79:9033/antena1.asf
Antena 2 : sop://broker.sopcast.com:3912/120705
Antena 3 : sop://broker.sopcast.com:3912/140274
AXN : sop://broker.sopcast.com:3912/60709
AXN Crime : sop://broker.sopcast.com:3912/60711
AXN Sci-Fi : sop://broker.sopcast.com:3912/60710
AXN Spin : sop://broker.sopcast.com:3912/60705
B1 TV : sop://broker.sopcast.com:3912/74841
BBC Knowledge : sop://broker.sopcast.com:3912/122847
Bollywood TV : sop://broker.sopcast.com:3912/120603
Boomerang : sop://broker.sopcast.com:3912/60712
Canal 7 Baia Mare : http://tele-plus.canal7.ro:8080
Cartoon Network : sop://broker.sopcast.com:3912/80593
Cartoon Network 1 : http://fms1.mediadirect.ro/live3/_definst_/cartoon/playlist.m3u8?publisher=1
Cinemax : sop://broker.sopcast.com:3912/140566
Cinemax 2 : sop://broker.sopcast.com:3912/120609
Comedy Central : sop://broker.sopcast.com:3912/140610
Digi 24 HD : http://82.76.249.77:80/digi24edge/digi24hdhqhls/index.m3u8
Digi Life : sop://broker.sopcast.com:3912/74632
Digi World : sop://broker.sopcast.com:3912/74633
DigiFilm : sop://broker.sopcast.com:3912/140601
DigiSport 1 : sop://broker.sopcast.com:3912/146135
DigiSport 2 : sop://broker.sopcast.com:3912/146137
DigiSport 3 : sop://broker.sopcast.com:3912/144752
Discovery Channel : sop://broker.sopcast.com:3912/122848
Discovery ID : sop://broker.sopcast.com:3912/122851
Discovery Science : sop://broker.sopcast.com:3912/122850
Disney Channel : sop://broker.sopcast.com:3912/139573
Disney Junior : sop://broker.sopcast.com:3912/122838
DIVA Universal : sop://broker.sopcast.com:3912/120700
Euforia TV : sop://broker.sopcast.com:3912/74634
Eurosport : sop://broker.sopcast.com:3912/145596
Eurosport 2 : sop://broker.sopcast.com:3912/124991
Favorit TV : sop://broker.sopcast.com:3912/74635
Film Cafe : sop://broker.sopcast.com:3912/120610
GSP TV : sop://broker.sopcast.com:3912/122846
HBO : sop://broker.sopcast.com:3912/120702
HBO Comedy : sop://broker.sopcast.com:3912/74845
HBO HD : sop://broker.sopcast.com:3912/75086
History Channel : sop://broker.sopcast.com:3912/80623
Jurnal TV : http://ch0.jurnaltv.md/channel0.flv
Kanal D : sop://broker.sopcast.com:3912/140600
Kanal D : http://fms1.mediadirect.ro/live3/_definst_/kanald/playlist.m3u8?publisher=1
MGM : sop://broker.sopcast.com:3912/120607
Na?ional TV : sop://broker.sopcast.com:3912/60704
NatGeo Wild : sop://broker.sopcast.com:3912/80626
National Geographic : sop://broker.sopcast.com:3912/80397
Nickelodeon : sop://broker.sopcast.com:3912/144654
PartyTV : http://77.36.61.158:7421/PartyTV.sdp
Plus TV Suceava : http://85.186.146.34:8080
Prima TV : sop://broker.sopcast.com:3912/140599
Pro TV : sop://broker.sopcast.com:3912/142364
Pro TV 2 : sop://broker.sopcast.com:3912/122831
Pro TV HD : sop://broker.sopcast.com:3912/144604
Pro TV Interna?ional : sop://broker.sopcast.com:3912/90686
ProCinema : sop://broker.sopcast.com:3912/80622
Realitatea TV : sop://broker.sopcast.com:3912/74843
România TV : http://rtvflash.smcmobile.ro:8010/rtv.flv
România TV 2 : sop://broker.sopcast.com:3912/111690
Sport.ro : sop://broker.sopcast.com:3912/144038
Taraf TV : sop://broker.sopcast.com:3912/74846
Taraf TV 1 : http://fms1.mediadirect.ro/live3/_definst_/taraf/playlist.m3u8?publisher=1
TCM : sop://broker.sopcast.com:3912/120605
TLC : sop://broker.sopcast.com:3912/122854
TV 1000 : sop://broker.sopcast.com:3912/140285
TV Paprika : sop://broker.sopcast.com:3912/60708
TVM Piatra Neam? : http://tvm.ambra.ro/
TVR 1 : sop://broker.sopcast.com:3912/146134
TVR 2 : sop://broker.sopcast.com:3912/137222
Universal : sop://broker.sopcast.com:3912/140611
Viasat Explorer : sop://broker.sopcast.com:3912/60703
Viasat History : sop://broker.sopcast.com:3912/60702
Viasat Nature : sop://broker.sopcast.com:3912/60586

SharePoint 2013 – MSSQL$SharePoint Error Fix

Recently I have been wanting to install Windows Server 2012, SQL 2012, SharePoint 2013 & TFS 2012 all on one VM for testing, but after installing SQL and SP – running the SharePoint Products Configuration Wizard for the first time, produced the following error:

Failed to start the database service MSSQL$SharePoint.
Repair this product by selecting it from the Add/Remove Programs menu

Read more »

Windows Server 2012: Server Core Remote Administration

If you have just installed Server 2012 Core, configured it to your requirements and ready to manage remotely but you get the following error message: “Computer \\computername.domain.com cannot be managed. The network path was not found.” 

But you are 100% sure you have:

  • Configure Remote Management set to Enabled
  • Remote Desktop set to Enabled (all clients)

Then the following fix should work:

Read more »

Windows Server 2012: Server Core SCONFIG

Previously we executed some basic commands to get our environment up and running on the network so that it will be available for remote administration. As we saw, it can be fairly de-motivating having to execute long commands just to simply change the name of a server!

This is where SCONFIG comes in. It has been available since Server 2008 R2 to combat the problems we faced previously. Some of the features of SCONFIG are (other than being launched easily from the CMD window by simply executing sconfig):

  • Configure domain or workgroup membership
  • Change a server’s name
  • Add a local Administrator account
  • Configure remote management features
  • Configure Windows Update settings
  • Download and install Windows updates
  • Enable or disable Remote Desktop
  • Configure network settings for TCP/IP
  • Configure the date and time
  • Log off, restart, or shut down

SCONFIG in action