Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

Native Development

Reply
New Contributor
joshua_wong
Posts: 8
Registered: ‎10-27-2011
My Device: Q10
Accepted Solution

Embedded Python import issues (numpy/scipy/h5py) in release mode

Background:

I am working on a scientific application which uses numpy/scipy (Python modules) in the backend for computation and h5py for data collection.  My application consists of a UI layer written in QML/C++ which invokes an embedded Python interpreter for numerical computations.

 

I've successfully cross-compiled the following libraries:

BLAS, LAPACK, HDF5, Python3.2, numpy, scipy, and h5py

 

The application directory structure under app/native is as follows:

+assets

+qm

+usr

  +include

  +lib

    +pkgconfig

    +python3.2  <- numpy/scipy/h5py exists within the site-packages

     libblas.so

     libgfortran.so.3

     libhdf5_hl.so.8

     libhdf5.so.8

     liblapack.so

     libpython3.2m.so

     libpython3.2m.so.1.0

     libpython3.so

 

So far, I am able to run my application in debug mode and was able to import various Python modules (including extension modules) without any problems.

 

Problem:

As I am approaching completion, I started testing in release mode and noticed the followings:

 

1) loading external libraries packaged in the bar file works (ie:  I am able to call routines from the Fortran BLAS/LAPACK or HDF5 libraries).

 

2) running the embedded Python interpreter works (in the main thread or in a separate thread)

 

Thus, I am able to confirm that the external libraries and the Python interpreter work correctly in release mode as well.

 

Problems occur when I tried to import Python modules:

3) importing Python builtin modules/packages works without issues

 

4) importing modules/packages with relative path fails.  Example, if I issue the command:

import numpy

This will work in debug mode but not in release mode.  Upon closer examination, it seems that when importing packages, any import statement with relative path will fail.  In release mode, the above example failed at:

from . import scimath as emath

However, the following statement works:

from .scimath import *

 

I have two questions:

1) Any hypothesis on whether this is a Python issue or a release mode issue?  I am leaning towards release mode since the same application works in debug mode.  I also looked at the python source code and couldn't see anything esoteric file I/O operations.

 

2) Based on the observation, it seems that relative paths are handled differently between Debug and Release modes.  Could someone please point me to some documentations explaining the differences?  And if such documentation does not exist, perhaps provide a brief summary?

 

Thanks,

Josh

New Contributor
joshua_wong
Posts: 8
Registered: ‎10-27-2011
My Device: Q10

Re: Embedded Python import issues (numpy/scipy/h5py) in release mode

Ok.  I think I just solved my issue.  I will list the solution first and then other observations that could be helpful to anyone in a similar situation.

 

Bottom Line Solution (if you don't want to read the rest):

Apparently, I compiled my LAPACK library with extended floating point precision (128-bit) which is not supported on the Cortex A9.  I recompiled LAPACK with double precision.  Running and importing the application in release mode now works. :Clap:

 

Interesting Observations:

At first, I thought that release mode handles dynamic library loading differently than debug mode since over the course of development I never ran into issues.  As a result, I decided to directly link all the libraries that numpy uses into my application (even though the application itself doesn't use them).  Compiling and linking is all fine, but of course during execution the numpy module failed to load.

 

So I decided to run this in debug mode as a sanity check.  Surprisingly, the application fails at the linking stage!!!  It is giving unresolved symbols trying to call extended precision routines from the XBLAS library (which as mentioned before does not work on the Cortex A9).  At this stage, I am completely dumbfounded as I have been running the application using LAPACK in debug mode without ever seeing this message for almost a year. :Oops:

 

I asked myself how can this be??  I also use scipy's linear algebra module which calls LAPACK/BLAS routines without problems over the course of the development.  In debug mode, I never see this failed or any messages indicating that there are unresolved symbols in my the LAPACK library.  Furthermore, I cross check all computations against MATLAB/Octave and they are correct.

 

Conclusions:

Here's my conclusions and if someone from BlackBerry can confirm this I will be very grateful:

 

1) In release mode, the dynamic linker will check every symbol in a shared library at runtime.  If any symbols cannot be resolved regardless of whether your application uses them or not, the library will fail to load.  This does not occur in debug mode.

 

2) There is a potential bug (not sure) in the linker that also behaves differently between debug mode and release mode.  In debug mode, the linker will fail to link a shared library if any symbols cannot be resolved despite the executable not referencing them.  In release mode, the linker will NOT FAIL.

 

If the above two statements are true, then I would like to make a suggestion to change the behaviour such that build stage and execution stage behaviours are aligned with each other.  Please let me know if this is intentional or if I should file a bug report.

 

Thanks,
Josh

 

PS:  Btw, I also built a Fortran cross compiler in case you are wondering how I managed to compile BLAS and LAPACK for BlackBerry.

New Developer
trevornunes
Posts: 6
Registered: ‎12-23-2011
My Device: Playbook

Re: Embedded Python import issues (numpy/scipy/h5py) in release mode

Just and FYI,

 

The VFP unit is in 'runfast mode and there might be issues with denormalization with high precision. This is true for all blackberry 10 based devices and Playbook.  It might be worth getting a QNX developer to see if we could implement support of the full compliancy mode in order to support scientific apps like this , I think it's worth it but it's a QNX OS decision:

 

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0344h/Chdiihcd.html

 

Developer
madkristoff
Posts: 58
Registered: ‎11-30-2012
My Device: BlackBerry Z10

Re: Embedded Python import issues (numpy/scipy/h5py) in release mode

Hey, sort of off topic, but could you provide tips on how you went about cross compiling? I sent you a PM too.
__________________________________________________________

If my post helped, click the like button!

Developer of Reader|YC , an open source Hackernews client