Mssearch.exe handle/memory leak windows 2008 sp2 sharepoint server

A sharepoint server was having domain connectivity issues and this issue was reoccurring every day during business hours causing impact to the business.
After analyzing the perfmon reports and resource monitor we have found MSSEARCH.exe causing handle leak on the server, the opened handles by this process gradually increases causing the server resource exhaustion.
Mssearch.exe had 49681 handles opened and it was consuming high memory, any process having such a high number of handles will cause performance issues. Thus causing server to loose connectivity(port block) with its domain controller.

If we restart/kill the msearch.exe process the server will be able communicate with domain controller and the domain user accounts will be able to login to this server

Lets try to understand what are Handles….(reference Windows Internals 6th Edition):

The kernel-mode core of Windows, which is implemented in Ntoskrnl.exe, consists of various subsystems such as the Memory Manager, Process Manager, I/O Manager, and Configuration Manager (registry), which are all parts of the Executive. Each of these subsystems defines one or more types with the Object Manager to represent the resources they expose to applications.

When an application wants to use one of these resources, it first must call the appropriate API to create or open the resource. For instance, the CreateFile function opens or creates a file, the RegOpenKeyEx function opens a registry key, and the CreateSemaphoreEx function opens or creates a semaphore. If the function succeeds, Windows allocates a reference to the object in the process’ handle table, which is maintained by the Executive, and returns the index of the new handle table entry to the application.

This handle value is what the application uses for subsequent operations on the resource. To query or manipulate the resource, the application passes the handle value to API functions such as ReadFile, SetEvent, SetThreadPriority, and MapViewOfFile. The system can look up the object the handle refers to by indexing into the handle table to locate the corresponding handle entry, which contains a pointer to the object. The handle entry also stores the accesses the process was granted at the time it opened the object, which enables the system to make sure it doesn’t allow the process to perform an operation on the object for which it didn’t ask permission. For example, if the process successfully opened a file for read access but tried to use the handle to write to the file, the function would fail. When a process no longer needs access to an object, it can release its handle to that object, typically by passing the handle value to the CloseHandle API.(Note that some resource managers provide a different API to release its resources.) When a process exits, any handles it still possesses are closed.

To understand more about handles please visit http://blogs.technet.com/b/markrussinovich/archive/2009/09/29/3283844.aspx

How did we fix the issue on sharepoint server??
There were a number of configurations we did within SharePoint:
1) Created a new Shared Service Provider
2) Created a new Search Content Database
3) Cleared the SharePoint Configuration Cache
4) Ran SharePoint STSADM command prompts to stop and restart the search service
5) Ran SharePoint configuration wizard on application and web front-end server to repair
6) Reset the search index
7) Initiated a full search crawl.

The errors what we were seeing were related to a problem writing to the search configuration database. Creating a new search content database and then stopping and restarting the mssearch.exe process got things working

Advertisements
Posted in Windows Troubleshooting | Tagged , , | Leave a comment

Troubleshooting a Hung Windows Server

Troubleshooting a hung or non-responsive Windows server can be a challenging endeavor. Simply hitting the reset button is no longer a tolerated option as an increasing number of these servers are used for business critical operations.

There are a variety of reasons why a server may hang including both hardware and software issues. For example bad NIC, device driver conflicts, resource pool depletion… etc

Here are some of the basic steps for troubleshooting an incident related to physical windows server in non-responsive/hung state

1. Try to Ping the server in hung state.
2. If server is reachable via ping try RDP to server and see if you are able to login.
3. If ping is working and RDP is not working then try to manage server remotely using ‘computer management’ or ‘pstools’ or “perfmon” from another server in same network.
4. If server is reachable via ping not via any other remote management tools then login to hardware remote console RSA/DRAC/ILO (if available).
5. Check the server status in hardware console and see if you are able to login.
6. If you are able to login to server console via RSA/DRAC/ILO then use perfmon to generate the performance report on server or use task manager and look for process that is consuming high CPU/memory, look for errors in event logs.
7. If server is non-responsive even from server console RSA/DRAC/ILO then perform NMI reboot. NMI generates a forced crash dump which may only be necessary if other means of troubleshooting prove unsuccessful.
8. NMI reboot option is available in power/diagnostics options in RSA/DRAC/ILO depending on the vendor (please see the below screenshots).
9. Before performing NMI reboot/generating memory dump make sure crash control is enabled in windows registry
10. After you have created crash dump file (memory.dmp), you are ready to begin using Windbg to determine what caused that server hung.

NOTE: Please take screenshots at each and every step you perform, screenshots are important for drafting a RCA.

Posted in Windows Troubleshooting | Leave a comment

PowerShell Script using WMI class to check uptime of the remote machines – Suptime.ps1

Hello.. Everyone,

Good Day !!!!

I have written one more script to check uptime of the server remote machines that uses PowerShell and WMI class “Win32_OperatingSystem” to check the uptime of the server

The script first checks if the server/computer is reachable then it tries to get uptime of that machine, if the computer/server is not reachable then the script throws a output stating “The server is not reachable”


Content of sutime.ps1
——————————————————————————–
# PS script to get uptime of the computers mentioned in computers.txt file
# Script first checks if the server is reachable then tries to get uptime of the server
# Function to get uptime

Function Get-HostUptime {
    param ([string]$ComputerName)
    $Uptime = Get-WmiObject -Class Win32_OperatingSystem -ComputerName $ComputerName
    $LastBootUpTime = $Uptime.ConvertToDateTime($Uptime.LastBootUpTime)
    $Time = (Get-Date) – $LastBootUpTime
    Return ‘{0:00} Days, {1:00} Hours, {2:00} Minutes, {3:00} Seconds’ -f $Time.Days, $Time.Hours, $Time.Minutes, $Time.Seconds
}

foreach ($computer in Get-Content “computers.txt”)
{
$c = Get-WmiObject Win32_PingStatus -f “Address=’$computer'”

if($c.StatusCode -eq 0) 
{

$sysuptime = Get-HostUptime -ComputerName $computer

write-host “$computer” “$sysuptime”
}

else
{

write-host “$computer is not reachable”

}
}
——————————————————————————–

Posted in Power-Shell | Leave a comment

Powershell – Ping servers using wmi class svr-ping.ps1

After a long time this is one new post from me… 😉 !!!!
 
Here is one more PowerShell script which uses WMI class PingStatus to check if server is reachable or not remotely 

The computer/server name should be mentioned in “Computers.txt” file

The script can use both IP address or hostname to check the ping status remotely

Ofcourse there are many other ways to ping servers however this is the easiest way I found to be quick way to check the server ping status remotely


Content of svr-ping.ps1

————————————————————————————
foreach ($computers in Get-Content “Computers.txt”)
{

$Computer = Get-WmiObject Win32_PingStatus -f “Address=’$computers'”

if($computer.StatusCode -eq 0) 
{
“{0,0} {1,5} {2,5}” -f$computer.Address, $computer.StatusCode, “Its pinging”
}
else
{
“{0,0} {1,5} {2,5}” -f $computer.Address, $computer.StatusCode, “Its not reachable”

}

————————————————————————————

Posted in Power-Shell | Leave a comment

Powershell – Invoke command used for Get-AppLockerFileInformation Windows 2008

This script which I have written can be used for getting app-locker event information from remote computers using power-shell on windows 2008 servers.

 

Invoke command calls applocker.ps1 on remote computer mentioned and exports the output to csv format, this can be converted to batch file to run on multiple servers
The .ps1 file and the .csv file will be on the source computer from which we are running this power-shell script. Before you run this script make sure power-shell remote management is enabled on the remote computers.

invoke-command -filepath C:\scripts\applocker.ps1 -computername servername | Export-csv c:\scripts\applocker\servername.csv

Content of applocker.ps1
———————————————————————————–
Import-Module AppLocker
Get-AppLockerFileInformation -EventLog -LogPath “Microsoft-Windows-AppLocker/EXE and DLL”
Get-AppLockerFileInformation -EventLog -LogPath “Microsoft-Windows-AppLocker/MSI and Script”
———————————————————————————–

Posted in Power-Shell | 1 Comment

PowerCLI script for deleting specified user account from ESX/ESXi hosts

vSphere PowerCLI: Windows PowerShell interface for managing vSphere is very powerful tool that can be used for managing large environments running ESX/ESXi servers.

I have tried writing a Powercli script for deleting specified user account from ESX host

This script reads the server names from servers.txt file and connects to each ESX server in txt file and removes the account mentioned in the script code, you can add additional commands to remove multiple users from a ESX/ESXi host.

Disadvantage of this script is that you need to enter root password every time it scans the each esx host in servers.txt file. Save this script as .ps1 file and enter the esx server names in servers.txt file

$hostx = get-content -path “c:\servers.txt”
foreach ($ESXhost in $hostx)
{
Connect-VIServer $ESXhost
Get-VMHostAccount -ID username | Remove-VMHostAccount -Confirm
}

Here “Username” is the user account to be deleted on the ESX servers mentioned in “Servers.txt” file

Posted in PowerCLI | Leave a comment

VSS – System writer not found in backup windows 2008 std sp2

We’ll discuss about a VSS backup issue on windows 2008 Sp2 virtual machine running on VMware platform

System writer is not found in the backup

Whenever we tried to run system state backup using wbadmin, the system state backup used to fail with the below error

WBADMIN error:
—————————–
U:\>wbadmin start systemstatebackup -backuptarget:e:
wbadmin 1.0 – Backup command-line tool
(C) Copyright 2004 Microsoft Corp.

Starting System State Backup [5/21/2012 8:21 PM]
Retrieving volume information…

This would backup the system state from volume(s) System(C:),Data(E:) to e:.
Do you want to start the backup operation?
[Y] Yes [N] No y

Creating the shadow copy of volumes requested for backup.

Summary of backup:
——————

Backup of system state failed [5/21/2012 8:22 PM]

Log of files successfully backed up
‘C:\Windows\Logs\WindowsServerBackup\SystemStateBackup 21-05-2012 20-22-43.log’

Log of files for which backup failed
‘C:\Windows\Logs\WindowsServerBackup\SystemStateBackup_Error 21-05-2012 20-22-43
.log’

System writer is not found in the backup.
—————————–


“vssadmin list writers” – all the writers were visible and stable

No errors in application or system event log. Tried procmon to see if any permission issues, no clue found in procmon

Server had symantec backup exec , SQL and IIS installed.

Tried the below article http://support.microsoft.com/kb/2009272, however this didnt worked.

Enabled vss tracing and found below messages in trace log “Error while obtaining an interface interface 0x80004002 HRESULT EXCEPTION CAUGHT: hr: 0x80042308”


Solution:

Check for any recent changes performed on the server. For example disk expansion, disk drive letter change, patch installation.. etc

After looking few days back we found there was a change performed on the server to change drive letter of the partition on which SQL was installled

During the above change the SQL services were still pointing to old partition and application team had uninstalled SQL manually and the OLD SQL services remained in registry pointing to old partition/drive letter.

We can identify non-present services in system information window — start –> all programs –> accesssories –> system tools –> system information –> Software environment –> services –> sort services by Error control

Reference MS article http://blogs.technet.com/b/askcore/archive/2010/06/18/reasons-why-the-error-enumeration-of-the-files-failed-may-occur-during-system-state-backup.aspx

Now we have identified the non-present services(services which are having invalid path or which doesnt exists at all but remain in registry), next step is to remove the non present services from registry and reboot server, To do this.. regedit –> HKLM –> SYSTEM –> Currentcontrolset –> Services . Identify the service and delete it from registry.


Last step is Remove any third party vss provider installed on server, Our server had symantec backup exec provider which got installed with Symantec backupexec

To remove symantec backup exec provider from registry perform these steps –> regedit –> HKLM –> SYSTEM –> Currentcontrolset –> Services –> vss –> Providers . Remove provider from symantec backup exec.

Once the above steps were performed MY SYSTEM STATE BACKUP WAS SUCCESSFULLY COMPLETED WITHOUT ANY ERRORS.

Posted in Windows-2008-VSS-issues | Leave a comment

Get CPU core information using Power-Shell

How to get cpu core information on windows 2003 and 2008 servers ?

The answer is here its just simple we can get CPU core information on all the servers provided in servers.txt file using power-shell script which uses WMI to get CPU core information from remote macines.

Save the below content to file coreinfo.ps1 and mention the server names in servers.txt file
———————————————————-

$Computers = Get-Content “C:\coreinfo\servers.txt”

ForEach ($cn in $computers)
{

Get-WmiObject Win32_Processor -ComputerName $cn | format-table SystemName, numberofcores, NumberOfLogicalProcessors –AutoSize
}

———————————————————-

The above script has been tested on windows 2008 servers and works perfectly fine

To run this script on Windows 2003 server machines you need to install below mentioned patches(whichever applies)

Posted in Power-Shell | Leave a comment