BlackBerry® Enterprise Solution

[GardBrown]

We were in the process if migrating our domain to Windows Server 2008 from Windows Server 2003.  At this stage, I have two 2008 DC's and one remaining 2003 DC.

My BES 4.0 is installed to a Windows 2000 member server.  I am ready to retire my 2003 Domain controller.  Yesterday, I did a 'test' shutdown on the 2003 DC to determine if any unexpected issues arose.  I then restarted all the member servers, including BES.  When the BES server came back up, all the services appeared to start, but none of the Blackberry devices would receive mail and there was an event id 10000 repeated in the event log which basically said:

'BES' Agent 1 failed to initialize within 30 seconds. 

That repeats about 10 times, then the log gets flooded with various 20049 and 16058 events relating to Exchange Connector and Blackberry Synchronization Service.

I restarted it a few times, and went through my BES configuration to see if anything appeared to be looking for the 2003 DC, but I found nothing referencing it.  The Exchange server is working normally.  So then I brought the 2003 DC back online, rebooted the BES server once more.  With the 2003 DC online, all is normal with BES again.  It's not a DNS issue because the BES Server's DNS points to the 2008 DC's, and has for some time.  

So, bottom line is with the 2003 DC out of the mix BES won't run.  It seems there is something about having the 2003 DC online that BES needs, but I don't know what. I am wondering if v4.0 of BES may be incompatible with a 2008 domain even though it's running on a Windows 2000 server.  Any help is appreciated!

-Gard Brown

Message Edited by GardBrown on 06-04-2009 05:12 PM

[AndyDufresne]

Did you move the FSMO roles off of the 2003 DC?

[GardBrown]

Yes, all the FSMO roles have been transferred to one of the 2008 DC's.  For what it's worth, this last 2003 DC was actually a secondary DC from the start.  The primary 2003 DC has already been decommissioned, and at that time the roles were transferred to one of the new 2008 DC's... the 2003 DC that remains never had any of the FSMO roles, but I just verified that to be sure.

Is it possible that BES 4.0 just isn't compatible with a 2008 Domain for some reason?  I would really like to know if anyone who still uses BES 4.0 is on a 2008 domain.

[AndyDufresne]

If you'd like, zip up a set of the days logs and email it to BBAndyDufresne@gmail.com. Please also make sure to include a link to this thread so I know what the message is referencing / where to respond.

If its too big for email please use something like yousendit.com and send me a link.

I'll post back any relevant content removing anything personally identifiable / confidential.

What forest and domain functional levels do you have set?

[GardBrown]

Thanks Andy,  are you referring to the Windows Application / System Logs, or the BES Logs?  

I'll have to actually do a test shutdown of the 2003DC again to reproduce the issue as everything is working fine as long as it's online.  So, that means that I will need to do it after hours, which is no problem, but it may be a day or two before I can send the logs.

The Forest and Domain functional levels are 2003.

[knottyrope]

He is asking for the BES logs.

[GardBrown]

I thought it might be a good idea to review the logs myself as well.  What would be the best place to start, I see there are a number of logs in the log folder, but I am unsure which ones may reveal issues with the BES startup process.

One other thing I just noticed that I thought I should mention.   The 2003 DC is also a WINS server.  I think it had WINS for a past project, and so we don't plan to have a WINS server going forward...  BES doesn't need WINS does it?

[AndyDufresne]

---

GardBrown wrote:

Thanks Andy,  are you referring to the Windows Application / System Logs, or the BES Logs?  

I'll have to actually do a test shutdown of the 2003DC again to reproduce the issue as everything is working fine as long as it's online.  So, that means that I will need to do it after hours, which is no problem, but it may be a day or two before I can send the logs.

The Forest and Domain functional levels are 2003.

---

The BES Logs ... however if you can also send over the app / system logs that would be of some use too.

[AndyDufresne]

look in the CTRL and MAGT logs at the times that the services are starting up.

No, BES does not need WINS.

[GardBrown]

Ok, here is the CTRL log from a restart when the 2003DC was offline.  I found nothing out of the ordinary in the MAGT log during this same time, but I can post it if needed.

[50000] (06/03 22:21:23):{0xDC0} Starting BlackBerry Controller Version 4.0.0.110
[30000] (06/03 22:21:23):{0xDC0} Found Dispatcher for server BES as 'BlackBerry Dispatcher' at \\127.0.0.1, PID=7300
[30000] (06/03 22:21:23):{0xDC0} 'BES': controlling 1 agents (1)
[30000] (06/03 22:21:23):{0xDC0} [CFG] Controller will restart BES Agent(s) on crash
[30000] (06/03 22:21:23):{0xDC0} [CFG] Controller will restart Dispatcher(s) on crash
[30000] (06/03 22:21:23):{0xDC0} [CFG] Controller will wait for WaitCount = 6 to restart BES Agent(s) on hung threads
[30000] (06/03 22:21:23):{0xDC0} [CFG] The number of BES Agent restarts is limited to 10 times per 24 hours
[20000] (06/03 22:21:23):{0xDC0} [CFG] Controller won't generate userdump files. Userdump.exe not found in system folder.
[30000] (06/03 22:21:23):{0xDC0} [CFG] No blackout period found for BES Agent restart on hung thread
[30000] (06/03 22:21:23):{0xDC0} BlackBerry Agent Controller Started
[50000] (06/03 22:21:23):{0x6B0} Waiting for syslog messages on port 4070
[30000] (06/03 22:21:24):{0x1138} 'BES' agent 1 started as process 4480
[10000] (06/03 22:21:55):{0x1138} 'BES' agent 1 failed to initialize within 30 seconds
[30000] (06/03 22:21:55):{0x1138} 'BES' agent 1 process terminated
[30000] (06/03 22:21:55):{0x1138} 'BES' agent 1 started as process 3372
[10000] (06/03 22:22:26):{0x1138} 'BES' agent 1 failed to initialize within 30 seconds
[30000] (06/03 22:22:26):{0x1138} 'BES' agent 1 process terminated
[30000] (06/03 22:22:26):{0x1138} 'BES' agent 1 started as process 6952
[10000] (06/03 22:22:57):{0x1138} 'BES' agent 1 failed to initialize within 30 seconds
[30000] (06/03 22:22:57):{0x1138} 'BES' agent 1 process terminated
[30000] (06/03 22:22:57):{0x1138} 'BES' agent 1 started as process 4600
[10000] (06/03 22:23:28):{0x1138} 'BES' agent 1 failed to initialize within 30 seconds
[30000] (06/03 22:23:28):{0x1138} 'BES' agent 1 process terminated
[30000] (06/03 22:23:28):{0x1138} 'BES' agent 1 started as process 6916
[10000] (06/03 22:23:59):{0x1138} 'BES' agent 1 failed to initialize within 30 seconds
[30000] (06/03 22:23:59):{0x1138} 'BES' agent 1 process terminated
[30000] (06/03 22:23:59):{0x1138} 'BES' agent 1 started as process 7568
[10000] (06/03 22:24:30):{0x1138} 'BES' agent 1 failed to initialize within 30 seconds
[30000] (06/03 22:24:30):{0x1138} 'BES' agent 1 process terminated
[30000] (06/03 22:24:30):{0x1138} 'BES' agent 1 started as process 7004
[10000] (06/03 22:25:01):{0x1138} 'BES' agent 1 failed to initialize within 30 seconds
[30000] (06/03 22:25:01):{0x1138} 'BES' agent 1 process terminated
[30000] (06/03 22:25:01):{0x1138} 'BES' agent 1 started as process 6856
[10000] (06/03 22:25:32):{0x1138} 'BES' agent 1 failed to initialize within 30 seconds
[30000] (06/03 22:25:32):{0x1138} 'BES' agent 1 process terminated
[30000] (06/03 22:25:32):{0x1138} 'BES' agent 1 started as process 1984
[10000] (06/03 22:26:03):{0x1138} 'BES' agent 1 failed to initialize within 30 seconds
[30000] (06/03 22:26:03):{0x1138} 'BES' agent 1 process terminated
[30000] (06/03 22:26:03):{0x1138} 'BES' agent 1 started as process 2096
[10000] (06/03 22:26:34):{0x1138} 'BES' agent 1 failed to initialize within 30 seconds
[30000] (06/03 22:26:34):{0x1138} 'BES' agent 1 process terminated
[30000] (06/03 22:26:34):{0x1138} 'BES' agent 1 started as process 6856
[10000] (06/03 22:27:05):{0x1138} 'BES' agent 1 failed to initialize within 30 seconds
[30000] (06/03 22:27:05):{0x1138} 'BES' agent 1 process terminated
[30000] (06/03 22:27:05):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:28:05):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:29:05):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:30:05):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:31:05):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:31:20):{0xDD0} Performing system health check (BlackBerry Controller Version 4.0.0.110)
[30000] (06/03 22:32:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:33:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:34:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:35:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:36:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:37:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:38:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:39:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:40:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:41:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:41:20):{0xDD0} Performing system health check (BlackBerry Controller Version 4.0.0.110)
[30000] (06/03 22:42:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:43:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:44:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:45:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:46:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:47:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:48:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:49:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:50:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours
[30000] (06/03 22:51:13):{0x1138} 'BES': will not restart - reached the maximum of 10 restarts per 24 hours

The entries from 22:41:20 thru 22:51:13 repeat until the server is restarted.  Of course, these events don't happen when the 2003 DC is online.