09-07-2010 07:37 AM
On keyboard models, an app can automate menu selection using input simulation -- simulate a Menu key press, then the first letter of the item, then simulate Click.
On the Storm, this does not work, either with KeyCodeEvent or KeyEvent.
If the menu list itself could be retrieved, and the default selection, then a NavigationEvent could be used to move up or down the exact # of items to select the desired item. This would also enable automation of menus where more than one item starts with the same letter.
Does anyone know how to do this?
Otherwise, I'll have to resort to the ridiculously complex procedure of taking screenshots and using OCR to read the menu, scroll it, read the rest of it, and then use simulated touch events, just to play out a UI macro. Themes can complicate this by inverting colors, changing fonts, etc.
The menu list could be had if we could get a reference to any MenuItem, the Menu itself, the Screen, the UiApplication, or the foreground app's UiEngine. Using the automating app's UiEngine does not work, nor does creating a background or invisible-popup screen with a ScreenUiEngineAttachedListener to pick up the screen of the app to be automated when it's opened.
I could get the Menu if I could just find a way to positively select a temporary ApplicationMenuItem, but that's not working because, again, it can't be selected by a shortcut key, and cannot be placed in any guaranteed position such as at the top or bottom. There again, unless I resort to taking and interpreting screenshots, I can't find a guaranteed way to get the menu list and select the right items.
Any help would be appreciated, I've spent a lot of time on this already.
09-07-2010 07:40 AM
which screen are you talking about?
09-07-2010 10:11 AM - edited 09-07-2010 10:15 AM
I'm talking about (in general) pre-programmed automation of procedures through simulating user input, including menu selections, for apps and phone functions that I did not build. I don't want to give any ideas away but the only remaining problem I'm up against right now is how to automate the menu selections. Can't assume positions, because software versions can change and utilities can be installed that insert menu items at unknown positions.
09-07-2010 10:20 AM
unless you can get hold of the native screen (as it is possible for the phone screen) this is not going to work.
09-07-2010 10:56 AM
Well, it would work if I could get ahold of the screen, or the uiengine, menu, or any menuitem, because from any of those you can get to the menu list and default item.
It is possible, because it has been done. Log messages of the apps that do it indicate an interesting approach - the target app has its menu triggered by simulated input, then immediately loses the focus to a NullField of the manipulating app, which is followed by a sizing exception and the manipulator closing an invisible screen resulting in a UiEngine fault that reveals the target app's Screen and Application object handles. Then the target app and its menu regain the focus, and voila, you've got everything.
Attempts to replicate this have not worked -- I can't get a NullField to generate an error by doing a setExtent beyond the display dimensions, nor does the expected UIE error occur when I pop the app's only screen and then keep the app running. The log message always has a specific GUID, BEF92E11214401C3, so I was hoping to reliably trigger and retrieve that logged error and get the Screen and Application handles from there. The entry looks like this:
UIE: Focus - target lost, prev=, input=targetApp.Screen@ABCD1234, app=targetApp.main@EF567890
09-07-2010 11:05 AM - edited 09-07-2010 11:11 AM
Something else I investigated was whether any global events are triggered by UI operations that could be used for this, but all I see in that respect is one type of globalevent indicating that a PopupScreen was pushed (with just the process ID), and another type indicating that a PopupScreen was popped, which includes the area of the display that has to be refreshed and the process ID. Those may be useful for other things, but not for this, and I don't know of any other global events that occur spontaneously relating to UI operations.
09-07-2010 11:10 AM
sure sounds interesting, but not in my field of expertise.
i somehow doubt that if somebody did succeed in something like this he would tell it here.
(if you find it, i doubt that you will, either)
i am happy with executing the phone apps menu items without event injection.
09-07-2010 11:13 AM
Something I'm trying to do requires it, and if I can get that working it opens up a few more interesting prospects, maybe even more than a few. I was going to figure it out and keep it under my hat, but at this point I don't care if the whole developer community is in on the solution as long as it's solved and I can move on from banging my head against this here brick wall...