BlackBerry® Enterprise Server Version 5.0

[gerek]

Last night our Blackberry 5.0.0 MR2 server locked up and no phones could send/receive email or browse the internet.  RDP wouldn't work either.  It also would not restart from the console requiring a forced shutdown with the power button.  Below are the errors from the event logs from when this issue began.

I am not very familiar yet with the Blackberry server as I have recently begun to help out our corporate IT with the management of it, but in this instance, my gut is telling me that this server (Win 2003SP2) has a memory leak.  Any thoughts would be appreciated.  After the Server was restarted (Ungracefully) everything was working again.

Event Type: Warning
Event Source: BlackBerry Controller
Event Category: None
Event ID: 20000
Date:  11/2/2009
Time:  6:50:32 PM
User:  N/A
Computer: MPLSBES
Description:
Unable to query service status of Dispatcher 'MPLSBES': 1726

Event Type: Warning
Event Source: BlackBerry Controller
Event Category: None
Event ID: 20000
Date:  11/2/2009
Time:  6:50:32 PM
User:  N/A
Computer: MPLSBES
Description:
Could not connect to Service Control Manager at \\127.0.0.1: 1721

Event Type: Warning
Event Source: BlackBerry Controller
Event Category: None
Event ID: 20000
Date:  11/2/2009
Time:  6:50:32 PM
User:  N/A
Computer: MPLSBES
Description:
Unable to reconnect to Dispatcher 'MPLSBES'

Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2019
Date:  11/2/2009
Time:  6:53:14 PM
User:  N/A
Computer: MPLSBES
Description:
The server was unable to allocate from the system nonpaged pool because the pool was empty.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 04 00 01 00 54 00   ......T.
0008: 00 00 00 00 e3 07 00 c0   ....ã..À
0010: 00 00 00 00 9a 00 00 c0   ....š..À
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........
0028: 01 00 00 00               ....   

Event Type: Warning
Event Source: BROWSER
Event Category: None
Event ID: 8021
Date:  11/2/2009
Time:  7:00:09 PM
User:  N/A
Computer: MPLSBES
Description:
The browser service was unable to retrieve a list of servers from the browser master \\DCName on the network \Device\NetBT_Tcpip_{80486837-DC1A-43F2-ADF6-FAFE411F1DB7}.
 
 Browser master: \\DCName
 Network: \Device\NetBT_Tcpip_{80486837-DC1A-43F2-ADF6-FAFE411F1DB7}
 
 This event may be caused by a temporary loss of network connectivity. If this message appears again, verify that the server is still connected to the network. The return code is in the Data text box.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: aa 05 00 00               ª...   

[knottyrope]

Did your domain controller GC reboot/update when it occured?

[gerek]

I confirmed with the Domain Administrators that they were not.

[jetpowercom]

Hmm... I'm wondering how similar our issue, which began today on our DC with BES 5.0, may be to this.  While I didn't get a chance to to see the event logs while the system was console-responsive, I did note that the page file was continuously maxed out by dozens of calhelper.exe instances, and consecutive Visual C++ runtime errors pop up.  Some BB users report email receipt and calendar update failure.

They refer to "D:\Program Files\Research In Motion\Black..." with the notation that "The application has requested the Runtime to terminate it in an unusual way.  Please contact the application's support team for more information."

My plan is to restart the DC as soon as feasible, and explore/document from there.

If this doesn't resemble any behavior of your instance, please feel free to ignore this note.  In any case, I'll post results of whatever I find later tonight or early tomorrow.

[jetpowercom]

Oops - "They" refers to the pop-ups, not the users, who know nothing of Runtime...

[jetpowercom]

Follow-up: 

Our issue turned out to be most likely related to a failed AV update file download, which apparently interfered with BES services.  Restarting fully cleared the issue, after which I excepted the BES folders' contents from virus-scanning on writes, in order to forestall future such events.

[knottyrope]

What AV are you running?

[jetpowercom]

Wow, sorry I missed your question until now!  We're running Network Associates VirusScan Enterprise, v7.0.0, scan engine 5.3, with daily updates.  On that day, a couple of users also reported strange behavior from their desktop AV (also VirusScan), so I attributed the failure to some issue with the download - bandwidth or download server issues.  We researched potential conflicts pretty well before acquiring BES.  Do you know of a BES 5.0 compatibility issue with VirusScan?

[gerek]

We are running Symantec AV v10.1.6.6.  I have excluded the folder and will see how that goes.

Thank you,

Jason