A sharepoint server was having domain connectivity issues and this issue was reoccurring every day during business hours causing impact to the business.
After analyzing the perfmon reports and resource monitor we have found MSSEARCH.exe causing handle leak on the server, the opened handles by this process gradually increases causing the server resource exhaustion.
Mssearch.exe had 49681 handles opened and it was consuming high memory, any process having such a high number of handles will cause performance issues. Thus causing server to loose connectivity(port block) with its domain controller.
If we restart/kill the msearch.exe process the server will be able communicate with domain controller and the domain user accounts will be able to login to this server
Lets try to understand what are Handles….(reference Windows Internals 6th Edition):
The kernel-mode core of Windows, which is implemented in Ntoskrnl.exe, consists of various subsystems such as the Memory Manager, Process Manager, I/O Manager, and Configuration Manager (registry), which are all parts of the Executive. Each of these subsystems defines one or more types with the Object Manager to represent the resources they expose to applications.
When an application wants to use one of these resources, it first must call the appropriate API to create or open the resource. For instance, the CreateFile function opens or creates a file, the RegOpenKeyEx function opens a registry key, and the CreateSemaphoreEx function opens or creates a semaphore. If the function succeeds, Windows allocates a reference to the object in the process’ handle table, which is maintained by the Executive, and returns the index of the new handle table entry to the application.
This handle value is what the application uses for subsequent operations on the resource. To query or manipulate the resource, the application passes the handle value to API functions such as ReadFile, SetEvent, SetThreadPriority, and MapViewOfFile. The system can look up the object the handle refers to by indexing into the handle table to locate the corresponding handle entry, which contains a pointer to the object. The handle entry also stores the accesses the process was granted at the time it opened the object, which enables the system to make sure it doesn’t allow the process to perform an operation on the object for which it didn’t ask permission. For example, if the process successfully opened a file for read access but tried to use the handle to write to the file, the function would fail. When a process no longer needs access to an object, it can release its handle to that object, typically by passing the handle value to the CloseHandle API.(Note that some resource managers provide a different API to release its resources.) When a process exits, any handles it still possesses are closed.
To understand more about handles please visit http://blogs.technet.com/b/markrussinovich/archive/2009/09/29/3283844.aspx
How did we fix the issue on sharepoint server??
There were a number of configurations we did within SharePoint:
1) Created a new Shared Service Provider
2) Created a new Search Content Database
3) Cleared the SharePoint Configuration Cache
4) Ran SharePoint STSADM command prompts to stop and restart the search service
5) Ran SharePoint configuration wizard on application and web front-end server to repair
6) Reset the search index
7) Initiated a full search crawl.
The errors what we were seeing were related to a problem writing to the search configuration database. Creating a new search content database and then stopping and restarting the mssearch.exe process got things working