Exchange 2007 unable to mount database

Recently we had a power failure which caused an improper shutdown to our exchange servers. After recovered from power failure we restarted all servers and fixed some cluster issues but there is one database which is showing failed in cluster manager. I had tried to mount the database but it was not successful and showing that there are some log files missing due to which database cannot mount.

For checking what log files are missing I ran following commands against the database which was not mounting. For running this command we have to go to the log drive of the database first and then run this command.

eseutil /mh

The output of this command is showing that the database is in dirty shutdown state and also showing that what log files are missing. I have checked these log files and luckily I have all log files present in the log drives folder.

Now recovery is easy, I have just renamed the check point file to any other name and try to mount it again and this time it mounts successfully without any issues and new checkpoint file is created.

MOSS "trial period for this product has expired"

Recently I faced an strange issue related to MOSS. One of my content admin is complaining that he is unable to add any link or page in his site anymore. whenever he is trying to create any page he is getting this error

"trial period for this product has expired"

Our MOSS farm has correct license and we haven't upgraded MOSS from trial version. After thorough analysis and research, I found that there is a bug in MOSS SP2 which reinitializes the license file and turns MOSS normal license to 180 days trial  license.

For fixing this issue, you can go to MOSS central admin->operations->Convert license type->Enter product key and add the key again or apply the hotfix mentioned in the below KB article.

http://support.microsoft.com/kb/971620

Renewing Edge Synchronization

If you have added any domain on HUB server and you are not able to see this domain on Edge server then might be your synchronization between your hub and edge servers are not working. To test this you can run following command;

Test-EdgeSynchronization

If the connection result is showing failed then you need to do edge subscription again for this edge server. Following are the steps for this

1. On the edge server which is not synchronizing run following command;

New-EdgeSubscription -filename "C:\temp\edgesub.xml"

now copy this .xml file to hub server and run following command

New-EdgeSubscription -filename "C:\temp\edgesub.xml" –site London

Please note the site parameter is required to bind this edge server to a specific AD site.

Now run following command

Start-EdgeSynchronization

You will notice that now the edge server is synchronizing data with HUB server.

For verification you can run test command again and this time it should show you the success message

Test-EdgeSynchronization

Enabling SCR on WAN

SCR is an Exchange 2007 DR solution available from Exchange 2007 SP1 onwards.  You can enable SCR on LAN and also on high latency networks. SCR also gives you benefit to recover from a Datacenter failover scenario. Since last month i was working to enable SCR between our DCs connected via MPLS. After a thorough analysis and lots of try and error i have compiled following results;

SCR Enabling Steps:

1. Seeding of .Edb File.

2. Enable Replication.

3. Monitor Replication

 

Step 1: Seeding:

Seeding is a prerequisite for enabling SCR, we can enable seeding by automatic way and manual way, because i have to seed .edb file on MPLS which is a high latency network i have to go for manual procedure. Following steps i have followed for manual seeding.

1. Take a snapshot of storage group on which i want to enable SCR (we are using snapshot based backup solution) you can also take the normal backup of the storage group. This will also truncate all the committed logs and we have less log files to copy from source to destination after enabling replication.

2. Run  below Command

Enable-StorageGroupCopy -Identity StorageGroup1 -StandbyMachine London1 -ReplayLagTime 0.1:0:0 –SeedingPostponed

The above command will enable SCR on storage group “StorageGroup1”. Standby machine (destination server) used is London and the replay lag time i have mentioned is 1 minute. seeding postponed is used because i don't want to enable seeding automatically i want to copy .edb file manually to destination server.

After running the above command, log files for this storage group will not truncate unless until we enable replication. This means that the drive space of log files folder should also be carefully planned because if the latency is high and the .edb file is taking days to copy then the log file drive space might going to be fill which will affect the production environment.

3. Take a snapshot on the storage group again or take a normal storage group backup again.

4. Ask the backup team to provide the .edb file from the last snapshot.

5. Use TeraCopy to copy .edb file from source to destination server. make sure the drive letters for log and data drive are same on standby server because it is one of the prerequisites of SCR.

2. Enabling Replication:

Replication will copy all logs files from source to destination. Replication is an ongoing process and whenever a log file is created on source it has to be shipped to destination for making both source and destination mailbox databases in same state.

Following is the command for enabling replication

Resume-StorageGroupCopy -Identity StorageGroup1 -StandbyMachine London1

after running the above command you will see that the log files started to copy from source to destination server. Standby server will replay these log files after 50 logs files has been copied. The first log file after you enable SCR should be present on destination otherwise replication will fail.

3. Monitoring Replication:

It is very important to check the status of storage group copy. The below command will show us the health of SCR and how many files in copy queue and replay queue.

get-storagegroupcopystatus StorageGroup1 -standbymachine London1

If you are seeing failed  in the summarycopystatus field then there is a possibility that the database you have seeded and the logs files copied are not at the same state (LSN number mismatch) or might be you don't have the first log file available to replay on the destination.

If you are seeing large number of files in copy queue length then bandwidth might be the possible cause.

You can also run following command on standby server for checking replication status;

Test-ReplicationHealth

Moving LCS databases to Different SQL server

LCS use 2 databases for its configuration and user data

1. RTC
2. RTCConfig

If you want to move these databases to different SQL server then you need to follow below sequence

1. Stop Live communication Server Service.
2. Take backup of the above mentioned databases.
3. Restore to them to new SQL server.
4. Make sure you have all 4 LCS domain groups (mentioned below) added to new SQL server and have all the required permissions given (Compare permission with the existing SQL server).
   LCS Domain Groups:
   RTCABSDomainServices
   RTCDomainServerAdmins
   RTCDomainUserAdmins
   RTCHSDomainServices
5. Run Following Command for setting new SQL server information;  
   LcsCmd.exe /forest /action:UpdatePoolBackend /poolname:[PoolName] /poolbe:[SQLSeverName]
6. Start LCS service again.
7. Open LCS MMC.
8. Make sure all services are running fine.

Exchange 2010 Unified Communication webcast

During some searching i found an excellent webcast on Exchange 2010 unified communication features. Following is the link;

https://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032442908&CountryCode=US

SharePoint website not opening on same server

I faced a strange issue recently, i have deployed SharePoint 2003 servers for one project and one of the requirements was to open SharePoint site from same server. when i was trying to open site from the same server i was getting authentication prompts and the site was not opening. I have even added the URL entry in the local host file but the issue remains same. After googling i found one Microsoft KB on the same issue which stated that  “This issue occurs if you install Microsoft Windows XP Service Pack 2 (SP2) or Microsoft Windows Server 2003 Service Pack 1 (SP1). Windows XP SP2 and Windows Server 2003 SP1 include a loopback check security feature that is designed to help prevent reflection attacks on your computer. Therefore, authentication fails if the FQDN or the custom host header that you use does not match the local computer name.”

In order to fix this we need to disable loopback check on all web server which are facing this behavior. I have applied this fix on my both front-end servers and after restarting both front-end servers the issue solved and i can now open website on same server. Following is the fix i followed.

Click Start, click Run, type regedit, and then click OK.

In Registry Editor, locate and then click the following registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa

Right-click Lsa, point to New, and then click DWORD Value.

Type DisableLoopbackCheck, and then press ENTER.

Right-click DisableLoopbackCheck, and then click Modify.

In the Value data box, type 1, and then click OK.

Quit Registry Editor, and then restart your computer

 

Following is the KB for this issue;

http://support.microsoft.com/kb/896861

Command for shrinking Log files

Below command will shrink SQL log files and also release unused physical space on drives.

Use DatabaseName

Backup log databasename with no_log

DBCC SHRINKFILE (logfilename)

WSS search and SQL 2005 indexing issue

Recently we have migrated our SharePoint 2003 from one AD forest to another forest, apart of this change we have also upgraded our backend DB from  SQL 2000 to SQL 2005.  Recently i faced a very critical issue, one of the heavily used site is not showing search result and the search is timing out. In order to investigate this issue i have identified the query in the SQL profiler and ran it directly on the SQL management studio. It took around 10 minutes to show results in new DB Server (SQL 2005)  but the same query is returning result in 1 sec in the old DB server (SQL 2000) i ran the query again with the Query execution plan against both servers and compared it, i have seen the cost for referencing Clustered index is very high in new DB server as compared to old DB server.  After some researching i have planned to update DB statistics by using following command;

update statistics tablename

SharePoint create its own indexes which we cannot alter and it uses the hard coded name for indexes such as ix_databasename. As per my analysis after moving DBs from SQL 2000 to SQL 2005 the databases statistics were not updated and indexes were not using optimal path for searching for data that why it is showing results after long delays.

Exchange 2007 unable to view deleted mailbox

In exchange 2007 if you delete any mailbox or any AD user, the mailbox will be there for 30 days in disconnected mailbox view and you can reconnect it again within 30 days if you want to restore the mailbox to its original state. After 30 days the mailbox will be permanently deleted and you can then only restore it from the backup.

Recently i faced a strange issue, one of the mailbox user was deleted by mistake and i am unable to see this user in disconnected mailbox view. The mailbox was deleted 2 days back which is under 30 days threshold but still the mailbox was not visible. After done some research and some help from one of my friend i figured out that i have to run following command in order to make this mailbox visible in disconnected mailbox view

Clean-mailboxdatabase –identity Databasename

As per Microsoft,

“use the Clean-MailboxDatabase cmdlet to scan the Active Directory directory service for disconnected mailboxes that are not yet marked as disconnected in the Microsoft Exchange store and update the status of those mailboxes in the Exchange store. This cmdlet is not able to update the Exchange store unless the Microsoft Exchange Information Store service is running and the database is mounted.

A connected mailbox has two parts: the mailbox object in the Exchange store, and the user object with Exchange properties in Active Directory. A disconnected mailbox is the mailbox object in the Exchange store, but it is not connected to a user object in Active Directory

Under normal circumstances, it is not necessary to run Clean-MailboxDatabase because a mailbox is marked as disconnected immediately after the Disable-Mailbox or Remove-Mailbox command completes. If you used the Disable-Mailbox cmdlet or the Remove-Mailbox cmdlet while the Exchange Information Store service was stopped, or if a mailbox was disabled by an external means other than the Disable-Mailbox cmdlet or Remove-Mailbox cmdlet, you may want to use the Clean-MailboxDatabase cmdlet to scan for disconnected mailboxes.”

SQL 2005 services not starting

When you have SQL server running in clustering and you changed any startup parameters, it is possible that SQL services will not run due to any incorrect parameter  settings. As per Microsoft this is because the cluster service obtains an incorrect protocol setting from a cluster checkpoint when you start the SQL Server service.

In order to fix this issue you have to first remove the last check point by running this command (after making SQL server resource offline)

cluster res "SQL Server (Instancename)" /removecheck: "Software\Microsoft\Microsoft SQL Server\MSSQL.x\MSSQLSERVER

and then fix the startup parameters for all nodes and then run following commands

cluster res "SQL Server (Instancename)" /addcheck: "Software\Microsoft\Microsoft SQL Server\MSSQL.x\MSSQLSERVER"

Following is the Microsoft KB for more details.

http://support.microsoft.com/kb/912397