Ops Mgr Event ID 2115

June 24, 2008 · Filed Under Operations Manager · Comment

I finally got around to implementing reporting yesterday for my Operations Manager 2007 implementation. I had previously asked the DBA to create the OperationsManagerDW DB for me on the SQL 2005 cluster using the DBCreateWizard tool per the deployment documentation (page 65 in the OpsMgr2007_Deployment.doc file or online here)…of course, following the directions and expecting that things would be just fine. Uh huh.

I finished the reporting installation late the afternoon and left for the day, knowing that the reports would take some time to display in the console…30 minutes is what the documentation states. A few hours later, however, I started getting Alert generated by Send Queue % Used Threshold alerts from a large number of hosts.

Summary
This monitor measures the Health Service Management Groups\Send Queue % Used and generates the following states:

Monitor State
Send Queue % Used Threshold

Warning
50 %

Critical
60 %

Causes
This can be caused by a low bandwidth or high latency connection from this Health Service to its parent Management Server. This can also be caused by rules that are collecting more data than the parent Management Server can process; especially when the parent Management Server has many agents reporting to it sending large amounts of data.

In looking at the Operations Manager event log on the RMS, I noticed Event ID 2115 errors every minute.

Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date:  6/24/2008
Time:  6:44:00 AM
User:  N/A
Computer: SERVER
Description:
A Bind Data Source in Management Group SysCtrOpsMgr has posted items to the workflow, but has not received a response in 52139 seconds. This indicates a performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData
Instance    : server.local
Instance Id : {CCA6D079-B810-CFE6-9784-9A8B2E483432}

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date:  6/24/2008
Time:  6:43:27 AM
User:  N/A
Computer: SERVER
Description:
A Bind Data Source in Management Group SysCtrOpsMgr has posted items to the workflow, but has not received a response in 52077 seconds. This indicates a performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectEventData
Instance    : server.local
Instance Id : {CCA6D079-B810-CFE6-9784-9A8B2E483432}

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date:  6/24/2008
Time:  6:43:27 AM
User:  N/A
Computer: SERVER
Description:
A Bind Data Source in Management Group SysCtrOpsMgr has posted items to the workflow, but has not received a response in 52077 seconds. This indicates a performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectEntityHealthStateChange
Instance    : server.local
Instance Id : {CCA6D079-B810-CFE6-9784-9A8B2E483432}

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

There were also Event ID 11411 entries every 10 minutes.

Event Type: Warning
Event Source: Health Service Modules
Event Category: None
Event ID: 11411
Date:  6/24/2008
Time:  6:45:04 AM
User:  N/A
Computer: SERVER
Description:
Alert subscription data source module encountered alert subscriptions that were waiting for a long time to receive an acknowledgement.
Alert subscription ruleid, Alert subscription query low watermark, Alert subscription query high watermark:
5fcdbf15-4f5b-29db-ffdc-f2088a0f33b7,06/23/2008 21:18:58, 06/23/2008 21:21:58

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

And of course, no reports visible in the Reporting node of the Ops Mgr GUI. Not good. I found several references to Event ID 2115 and this exact error while searching, but they didn’t seem to correct the issue. We don’t have Exchange Server 2003, so Clive’s post was out the window. I did make the overrides that Kevin mentioned in his post, but that didn’t fix the errors either. So I was right back to where I started in thinking that something was wrong with the database.

The solution was found in KB 942865, although the symptoms in the article didn’t match up completely…at least the title didn’t. At any rate, the OperationsManagerDW DB was created with the DBCreateWizard tool, and the query Select * from MemberDatabasethat was run against the OperationsManagerDW DB didn’t return any rows.

Running the query EXEC MemberDatabaseAttach ‘dbserver\instanceName’, ‘datawarehouseDBname‘, 1, 1, 1 query in the KB article was the fix. (Be sure to change the SQL server and instance and DW DB names accordingly, as the article points out.) After that, running the query Select * from MemberDatabase returned the expected results, a single row that contained the SQL server name and DB name. Even better yet, event ID 31554 entries (not 31544 as indicated in the KB article) entries started to appear.

Event Type: Information
Event Source: Health Service Modules
Event Category: Data Warehouse
Event ID: 31554
Date:  6/24/2008
Time:  7:40:09 AM
User:  N/A
Computer: SERVER
Description:
Workflow succeeded storing data in the Data Warehouse

One or more workflows were affected by this.

Workflow name: Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData
Instance name: server.local
Instance ID: {CCA6D079-B810-CFE6-9784-9A8B2E483432}
Management group: SysCtrOpsMgr

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

If have previously seen the Event ID 11411 entries, you should also see an Event ID 11412 show up on your management server a short time later.

Event Type: Warning
Event Source: Health Service Modules
Event Category: None
Event ID: 11412
Date:  6/24/2008
Time:  7:54:04 AM
User:  N/A
Computer: SERVER
Description:
Alert subscription data source module has recovered from the long wait to receive acknowledgements for alert subscriptions.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

So that’s the fix…but what was the REAL problem? Nothing in the KB article discussed it, and I didn’t find it through searching after the fact. Anyone have any ideas?