Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

BlackBerry® Enterprise Service 10

Reply
New Member
Posts: 1
Registered: ‎07-15-2014
My Device: BlackBerry Z10

BES 10: BB Device MGM Console no longer appears.

[ Edited ]

Environment:
------------------------------------------------
OS Type:    M$ Win Server 2012 R2 Datacenter
OS Version:    6.2, B9200, SPK Major 0, SPK Minor 0
Arch:        x86_64
CPU:        2 x Intel Xeon E3-1245 V2 @3.40GHz
RAM:        16GB
BES:        10.2.3 (B26)
J2EE/Java:    1.7.0 U55 32Bit/64Bit
MSSQL:        M$ SQL Server 2008 R2 Enterprise (local instance)


Dear BlackBerry Friends,

since about 2 weeks I am desperately struggling to make again our BES 10 instance administrative,
sadly, I came to a death end - that's why I am here, hoping some of you good guys has a hot idea.

The instance ran over months with under release V10.2.2 B19 without any problems.

The current situation is as follows:

After injection of an EVSSL with the config tool the certificate was applied correctly.
I then managed to install the Apple APN Cert in the UDS WebConsole and tried to activate an iPhone.

However, this failed again as usual.

After a machine reboot the BlackBerry Device Management console does not appear anymore, though the
Management Studio is accessible but only shows the details for UDS (iOS, Android)

Unfortunately, I don't know exactly what really caused the current situation. In fact, an upgrade to the latest MR 10.2.3 B26 didn't resolve the problem either. Therefore, I examined all the logfiles and found a "keystore tampered" message in relation to the MDS keystore.

After having removed the registry entry for the keystore and generated a new one involving the ConfigUtil, the problem persists: the "tampered keystore" message and as well the Device Management Console doesn't come up.

Only the part for UDS administration is working, I can no longer add any license keys and I can also not administrate nor setup any BlackBerry devices and users.

A complete re-install even having on the 3rd attempt removed all BlackBerry related entries from the registry didn't help as well.
However, the complete re-install with the setup option "Existing BlackBerry Domain" by referring to the available BESCFG-DB brings up the BES10 services, so that all current activated BlackBerry devices work without any reason to complain.

Even a full re-install under a new user with a blank database turns out the same result: No BlackBerry Device Management Console.

So I investigated in depth and found out, that for whatsoever reason the EMWS service starts, but after a few minutes shuts down but still indicates to be running under services.

 

All what appears in the Windows eventlog can be seen on the following screen shot.

 

 


I could also reveal, that the entire Tomcat instance - which hosts the BB Device MGM Console - doesn't even start. There is no open default port 37443 (https) and no open port 38180 (http). I can confirm, that there is no other application using these ports. As well a port change with the ConfigUtil has no impact either.

When I open a command shell and start C:\Program Files (x86)\Research In Motion\BlackBerry Enterprise Service 10\EMWS\bin\EMWS.exe, nothing appears. When I run java -jar bootstrap.jar from inside C:\Program Files (x86)\Research In Motion\BlackBerry Enterprise Service 10\EMWS\bin, a bunch of Java errors appears indicating amongst other messages, that the EMWSKeystore is missing.

So having created a EMWSKeystore under <inst root>/EMWS/bin has no impact.

In the BBMFS-Log I have a refused XML-SOAP connection error which looks as follows:

014-07-15 23:56:30.547    ForkJoinPool-1-worker-12    INFO     STDERR    System.err    Jul 15, 2014 11:56:30 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
    INFO: Creating Service {http://ws.rim.com/enterprise/admin}BWSUtilService from class com.rim.ws.enterprise.admin.BWSUtil
    
2014-07-15 23:56:30.548    ForkJoinPool-1-worker-13    INFO     STDERR    System.err    Jul 15, 2014 11:56:30 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
    INFO: Creating Service {http://ws.rim.com/enterprise/admin}BWSUtilService from class com.rim.ws.enterprise.admin.BWSUtil
    
2014-07-15 23:56:33.591    ForkJoinPool-1-worker-12    INFO     STDERR    System.err    Jul 15, 2014 11:56:33 PM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging
    WARNING: Interceptor for {http://ws.rim.com/enterprise/admin}BWSUtilService#{http://ws.rim.com/enterprise/admin}getAuthenticat... has thrown exception, unwinding now
    org.apache.cxf.binding.soap.SoapFault: Error writing to XMLStreamWriter.
        at org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor.writeSoapEnvelopeStart(SoapOutInterceptor.java:175)
        at org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor.handleMessage(SoapOutInterceptor.java:81)
        at org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor.handleMessage(SoapOutInterceptor.java:61)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:255)
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516)
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
        at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
        at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
        at com.sun.proxy.$Proxy45.getAuthenticators(Unknown Source)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$8$$anonfun$10.apply(BWSDomain.scala:384)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$8$$anonfun$10.apply(BWSDomain.scala:384)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$execInternal$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(BWSDomain.scala:217)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$execInternal$1$$anonfun$apply$mcV$sp$1.apply(BWSDomain.scala:216)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$execInternal$1$$anonfun$apply$mcV$sp$1.apply(BWSDomain.scala:216)
        at com.rim.fuse.server.net.Node.exec(Node.scala:548)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$execInternal$1.apply$mcV$sp(BWSDomain.scala:215)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$execInternal$1.apply(BWSDomain.scala:204)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$execInternal$1.apply(BWSDomain.scala:204)
        at com.rim.fuse.server.bws.ThreadLogger$.use(ThreadLogger.scala:31)
        at com.rim.fuse.server.bws.BWSDomain.execInternal(BWSDomain.scala:204)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$exec$3.apply(BWSDomain.scala:327)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$exec$3.apply(BWSDomain.scala:327)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$8.apply(BWSDomain.scala:384)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$8.apply(BWSDomain.scala:380)
        at com.rim.fuse.server.util.CacheableValue.apply(CacheableValue.scala:25)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$11.apply(BWSDomain.scala:414)
        at com.rim.fuse.server.bws.BWSDomain$$anonfun$11.apply(BWSDomain.scala:414)
        at com.rim.fuse.server.util.CacheableValue.apply(CacheableValue.scala:25)
        at com.rim.fuse.server.bws.BWSDomain.getAuthenticationTypes(BWSDomain.scala:420)
        at com.rim.fuse.server.Fusion$$anonfun$getAvailableAuthenticationTypes$1.apply(Fusion.scala:93)
        at com.rim.fuse.server.Fusion$$anonfun$getAvailableAuthenticationTypes$1.apply(Fusion.scala:93)
        at com.rim.fuse.server.dist.Distributor$$anonfun$distributed$2$$anonfun$apply$2.apply(Distributor.scala:67)
        at com.rim.fuse.server.util.FusionPredefs$.outcomeOf(FusionPredefs.scala:372)
        at com.rim.fuse.server.dist.Distributor$$anonfun$distributed$2.apply(Distributor.scala:67)
        at com.rim.fuse.server.dist.Distributor$$anonfun$distributed$2.apply(Distributor.scala:66)
        at com.rim.fuse.server.dist.RequestExecutor$$anonfun$act$1$$anonfun$apply$1.apply(RequestExecutor.scala:38)
        at com.rim.fuse.server.dist.RequestExecutor$$anonfun$act$1$$anonfun$apply$1.apply(RequestExecutor.scala:33)
        at scala.actors.ReactorTask.run(ReactorTask.scala:31)
        at scala.actors.Reactor$class.resumeReceiver(Reactor.scala:129)
        at com.rim.fuse.server.SomeDeviceDomain.scala$actors$ReplyReactor$$super$resumeReceiver(SomeDeviceDomain.scala:25)
        at scala.actors.ReplyReactor$class.resumeReceiver(ReplyReactor.scala:68)
        at com.rim.fuse.server.SomeDeviceDomain.resumeReceiver(SomeDeviceDomain.scala:25)
        at scala.actors.Actor$class.searchMailbox(Actor.scala:500)
        at com.rim.fuse.server.SomeDeviceDomain.searchMailbox(SomeDeviceDomain.scala:25)
        at scala.actors.Reactor$$anonfun$startSearch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Reactor.scala:117)
        at scala.actors.Reactor$$anonfun$startSearch$1$$anonfun$apply$mcV$sp$1.apply(Reactor.scala:114)
        at scala.actors.Reactor$$anonfun$startSearch$1$$anonfun$apply$mcV$sp$1.apply(Reactor.scala:114)
        at scala.actors.ReactorTask.run(ReactorTask.scala:33)
        at scala.concurrent.forkjoin.ForkJoinPool$AdaptedRunnable.exec(ForkJoinPool.java:611)
        at scala.concurrent.forkjoin.ForkJoinTask.quietlyExec(ForkJoinTask.java:422)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.mainLoop(ForkJoinWorkerThread.java:340)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:325)
    Caused by: javax.xml.stream.XMLStreamException: java.net.ConnectException: Connection refused: connect
        at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.writeStartElement(Unknown Source)
        at org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor.writeSoapEnvelopeStart(SoapOutInterceptor.java:122)
        ... 52 more
    Caused by: java.net.ConnectException: Connection refused: connect


In fact is a BlackBerry Service running under port 9091 not available.

The EMWS-Log file shows another error:

[WARN ] (07/15 23:53:44:837):{AsynchronousPushServiceThread} AsynchronousPushServiceUtils:Failed to send heartbeat poke to MDS-CS.
[DEBUG] (07/15 23:54:49:876):{AsynchronousPushServiceThread} EMWSSmiley Frustratedent Poke to: http://169.254.19.176:9080/push2 with query parameters:
[DEBUG] (07/15 23:54:49:876):{AsynchronousPushServiceThread} EMWS:X-Rim-Push-Id: null

(...)

[DEBUG] (07/12 02:53:33:809):{AsynchronousPushServiceThread} EMWS:X-Rim-Push-Id: null
[DEBUG] (07/12 02:53:33:809):{AsynchronousPushServiceThread} EMWSSmiley Frustratedtatus Code was: 200
[INFO ] (07/12 02:54:19:163):{Stop Thread} EMWS:[CFG] BlackBerry Enterprise Management Web Service is stopping
[INFO ] (07/12 02:54:19:163):{Stop Thread} EMWSSmiley Frustratedtopping LicensesManagerThread.
[INFO ] (07/12 02:54:19:163):{Stop Thread} EMWSSmiley Tongueausing EMWS Service.
[INFO ] (07/12 02:54:19:177):{Stop Thread} EMWSSmiley Frustratedhutting down EMWS Service.
[INFO ] (07/12 02:54:19:191):{Stop Thread} EMWS:[CFG] BlackBerry Enterprise Management Web Service has stopped



From the WebServer-Log file:

Jul 15, 2014 11:54:28 PM org.apache.coyote.http11.Http11NioProtocol init
SEVERE: Error initializing endpoint
java.io.IOException: Keystore was tampered with, or password was incorrect
    at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source)
    at sun.security.provider.JavaKeyStore$JKS.engineLoad(Unknown Source)
    at java.security.KeyStore.load(Unknown Source)
    at org.apache.tomcat.util.net.NioEndpoint.init(NioEndpoint.java:834)
    at org.apache.coyote.http11.Http11NioProtocol.init(Http11NioProtocol.java:136)
    at org.apache.catalina.connector.Connector.initialize(Connector.java:1022)
    at org.apache.catalina.startup.Embedded.start(Embedded.java:830)
    at net.rim.application.ipproxyservice.TomcatThread.startTomcat(Unknown Source)
    at net.rim.application.ipproxyservice.TomcatThread.run(Unknown Source)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
    ... 9 more

What I am really concerned about is that even a fresh install under a different user ends up with the same phenomena, no matter if using an existing config DB or creating a new one. Since I am unable to access the BB Device console, I can not retrieve the default keystore password, though I can change all passwords with the ConfigUtil, I am wondering also, if the default keystore password is not only stored in the registry but in the BES config DB as well, and the services query the DB password on run-up which is of course different from the registry, and as well from the new applied keystore passwords.

I would deeply appreciate any comment or idea of you where else to look at and what are the necessary tasks in this instance.

Thank you very much in advance &
Best from Germany! :-)

Dave.