FIM Service Management Agent (MIMMA) is not responsive and ends with a stopped-server error – [sadness noises]

The stopped-server error is difficult to troubleshoot because of its indistinct error reporting and multiple causes. It is basically saying that something has happened but Microsoft Identity Manager Synchronization Service (MIMSync) doesn’t know what exactly. It is indicating that the error is beyond the scope of MIMSync. We should focus on what the MIMMA is trying to do in order to troubleshoot this correctly.

Let’s take a look at the architecture of MIM in general. The architecture diagram is based on FIM but this still apply to MIM. As for naming convention we’ll continue to use MIMSync and MIMService.

We can see that the MIMSync is located in the application tier between the MIMService and the database servers. Most of the time the MIMService is also installed on the MIMPortal server but as shown in the diagram, they are two separate entities. This means that if I disable every Microsoft Internet Information Services (IIS) instances that is hosting MIMPortal, MIMService will continue to run as this is a stand-alone web service (Microsoft.ResourceManagement.Service.exe). As for the scope of this problem, we have established that the stopped-server error doesn’t involve MIMPortal nor IIS.

The MIMMA behaves like a SQLMA with additional ECMA/XMA properties. It connects to the database server and to the MIMService’s base address. The export run in particular connects to the MIMService. Should the MIMService endpoint be unavailable, the export will generate the following error.

An error occurred in executing a Web service object request. 
Type: System.ServiceModel.EndpointNotFoundException 

Message: There was no endpoint listening at http://mim.identandy.local:5725/ResourceManagementService/MEX that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.

Stack Trace:    at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
   at System.Net.HttpWebRequest.GetRequestStream()
   at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()

Inner Exception: Unable to connect to the remote server

Troubleshooting

There are multiple causes that could make the stopped-server error occur. I have categorized them as the following four suspects:

• The MIMService is unreachable. We need to establish what is preventing the connection between MIMSync and MIMService.
  • Check the Windows Services Manager (services.msc) whether or not MIMService is running.
  • If the MIMService isn’t running, check if the service account has sufficient rights.
  • Check the firewall whether or not port 5725 is blocked.
  • Check the connectivity to the MIMService endpoint using a browser. It should display a default WCF services page.

• The SQL Server is unreachable. We need to confirm that the management agent can connect to the SQL database and server.
  • Check the SQL Server Configuration Manager whether or not SQL Server and the concerning instance are running.
  • Check the firewall if the necessary ports are open.
  • Check the SQL Server Client Network Utility (cliconfg.exe) whether or not the correct network library and server name is being used. Normally I would use TCP/IP with “Dynamically determine port” enabled.
  • Check the SQL Server Management Studio (SSMS) whether or not the service account has login permissions.

• The SQL Server is unresponsive. We can assume that MIMSync can successfully (or partially) connect to the database server.
  • Check the Event Viewer for SQL timeout exceptions such as “Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding“.
    • This indicates that there are performance problems that needs to be addressed.
  • Check the Event Viewer for SQL permission exceptions such as “System.Data.SqlClient.SqlException: The EXECUTE permission was denied on the object 'StoredProcedureName', database 'FIMService', schema 'sync'.“.
    • This should be easily fixed by adding the service account to the db_owner role of the FIMService database.
  • Check the Event Viewer for SQL tempdb exceptions such as “Could not allocate space for object ‘dbo.SORT temporary run storage: ***************‘ in database ‘tempdb’ because the ‘PRIMARY’ filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.”.
    • While I haven’t fully established the source of this problem, I have managed to work around this by stopping the MIMSync and MIMService instances and clearing the TempDB’s by either a SQL Server restart or via SQL scripts.

• The MIMMA, MIMService or SQL Server is taking forever to complete requests. This one is by far the most difficult to troubleshoot.
  • Check if the implementation is following the best practices defined by Microsoft.
  • Check the SQL Server Management Studio (SSMS) to find deadlocks in the FIMService and FIMSynchronization databases.
  • Check the Management Policy Rules (MPR) in MIM Portal to find contradicting rules or erroneous workflows.
  • Check if the server infrastructure is adequately equipped. Ensure that the SQL Server has enough IOPS, memory and CPU to limit bottlenecking.

If you are still experiencing problems, consider getting a MIM health check done by a Microsoft Professional to discover underlying issues that might be the cause of this problem. Contact me for more information.