Integrating accessories with BlackBerry 10 using audio as a data carrier

by Retired on ‎07-04-2013 10:06 AM (3,166 Views)

Introduction

 

This article was originally published as a blog post in the BlackBerry® developer blog (http://devblog.blackberry.com) and has been transformed into a Knowledge Base article to make it easier to locate and use in the future. Readers of this article ought to be familiar with BlackBerry 10 Native development, including Qt® and QML, and should be familiar with C++ in order to appreciate the sample application that is used to demonstrate the concepts.

 

The sample application "AudioAccessory" has been published on GitHub® as Open Source and a link is provided at the end of this article.

 

In this article, we look at how you can use audio and the headphone socket on your BlackBerry 10 smartphone as an interface to other devices. Now this may sound like a far-fetched idea. But it really isn’t and there are a number of commercial accessories in the market which use this very approach. This is an integration method that is out there and in use today, and, there’s no reason why you couldn’t consider it for your accessory and BlackBerry 10 should you have good reason not to use an alternative such as Bluetooth®, for example.

 

We'll start with some theory before turning our attention to a real, sample application.

 

The Authors

 

This article was co-authored by Martin Woolley and John Murray both of whom work in the BlackBerry Developer Relations team. Both Martin and John specialize in the application of proximity radio technology on BlackBerry devices including Bluetooth and NFC (amongst other things). And sometimes they look at other interesting things like audio :-)

 

The Theory Part

 

So how can you use audio to send and/or receive digital data? Well, this is precisely what modems do. Modems take digital data and turn it into an analogue representation that can be transmitted as an audio signal. This is called “modulation”. In addition, on receipt of such a signal, they can reverse the process and “demodulate” the analogue signal back into the digital data, hopefully the same digital data that was originally transmitted.

 

Modems are generally pieces of hardware. But, you can perform the same modulation/demodulation process in software. Those of you who like to dabble with an Arduino, may be familiar with the SoftModem Library for example.

 

To use this technique in a BlackBerry 10 application so that it can communicate with an external device, via the headphone socket, you need a number of key ingredients in or available to your application:

 

  1. Access to the BlackBerry 10 audio sub-system via an API
  2. A suitable modulation/demodulation  algorithm
  3. A data coding 

 We can depict these three magic ingredients, together with the application data, as layers in a stack:

 

stack.png

 

Figure 1 - Simplified Stack

 

Encoding and Modulation/Demodulation Choices

 

There are lots of ways you could choose to encode text as byte values and with a suitable error detection/correction mechanism and equally, there are many modulation schemes that you could consider. We'll look at the way we handled this in our sample application soon.

 

Data Coding Scheme

 

To encode our simple text in such a way that we could safely transmit it, we used something called Baudot code which represents characters with 5 bits only and uses 2 bits to supply a parity checking capability.

 

Let’s take a look at an example:

 

Imagine we receive the following byte value: 00011010. How would we decode it according to our chosen scheme?

 

The first thing to know is that the data is transmitted with least significant bit first. So reordering, what we really have is 01011000.

 

Bit 7 in our reordered data bit is always 0 in this schema; the data payload is 5-bits long and is 11000 whilst the 10 bits contain our parity check bits. For an even number of 1-bits in the message (11000) the check bits must be 10 whilst for an odd number they must be 01.

So, in this example we see that the checksum is correct and the message is 11000 or 0x14 in Hex.

There’s a final twist in that this value represents a 5-bit encoding of an ASCII characters with its code-point (value) derived as an offset from the code point value for ASCII “0”, which is 0x30. So, to find out what ASCII letter 0x14 represents you have to add 0x30 to 0x14 and that gives us 0x44 which is (drum roll)..... “D”.

 

Modulation / Demodulation

 

In our sample app we used something called “AFSK” (Audio Frequency Shift Keying). You’ll find a reference to frequency shift keying at the end of this post. But basically, it involves the transmission of sine waves for a given length of time, at one frequency to represent a binary 1 and at another frequency to represent a binary 0.  We used 3150Hz, a high tone, to represent a 1 and 1575Hz, a lower frequency to represent a 0. Here’s an image taken from an audio application, Audacity, which shows this:

 

bit_frequency_graph.png

Figure 2 - Frequency Analysis

 

 

Now sending a single cycle of a sine wave for a single bit value will not work. The tone’s duration would be far too short for reliable decoding. So in practice we send a series of sine wave cycles at the required frequency such that the tone is emitted for a greater length of time. The following image depicts this:

 

series_of_pulses.png

Figure 3 - A series of audio tones

 

It’s pretty easy to appreciate what’s going on in that last image:

 

  • Time progresses from left to right
  • On the left there is a long preamble of a high frequency tone (HIGH)
  • This is followed by 4 intervals of a lower frequency tone (LOW)
  • Then 2 HIGH
  • Then 1 LOW
  • Then 1 HIGH
  • Then 1 LOW
  • Then 2 HIGH
  • Then “silence”

The pre-amble of high frequencies (1s) is a technique for indicating to the receiver that some data is about to be transmitted to it. The first 0 after this, or low frequency pulse, indicates that the next 8 bits contain data. It’s called the START bit for that reason. This comes from something called “Asynchronous Serial Data Transmission”

 

Audio

BlackBerry 10 has extensive APIs for working with the audio system on the device. I’ve included a reference below to the API documentation.

 

We have a couple of key considerations that we need to address in the context of our application and use of audio for data communications. Firstly, when capturing or transmitting, we need to work at a particular sample rate. So what I sampling and what is a sample rate exactly?

 

With our modulation/demodulation strategy, we’re converting digital data, characterised by discreet values in a fixed range, to analogue data which is characterised by being “continuously variable”. I think of it like this; analogue systems are capable of nice smooth curves, whereas digital systems can only manage straight lines or blocks. So if I want to express the idea of a curve in the digital world, I use a series of short, straight lines and if those lines are short enough then when placed next to each other, my creation will look pretty much like a curve. Here’s a crude representation of that concept with the red curve representing an analogue signal and my blocks representing a digital approximation of that curve:

  

sampling.png

Figure 4 - Sampled digital representation vs analogue representation of an audio tone

 

Hopefully you can see at a glance that if I use a small number of “large grained” digital values to represent my analogue curve, I will end up with a very poor approximation indeed. But if I use a very large number of very fine, lines or blocks I will produce a good likeness.

 

And this is what sampling is about. Imagine a sound that lasts for 1 second. If I divide that sound into 4, quarter second segments (and I’d say my sample rate here was therefore “4”), and I demodulate  each quarter second segment, producing a single digital value that corresponds to each of the measured signals during each quarter second, then this is just like producing a crude, blocky approximation of the original sound. But if (plucking a number out of the air), I was to “sample” the analogue signal, say 44,100 times a second, I will end up with a quite fine and accurate representation of my analogue sound, albeit requiring a large number of digital values as a result.

 

So a high sample rate will give us a good quality digital representation of a sound.

 

Now in our case, as you’ll see when we look at the code behind our application, we’re going to be generating our high frequency (0x01) and low frequency (0x00) tones using quite a low level API and we’ll be providing those digital values which will then get converted to an analogue signal. We need to vary those values at the required frequency to get the tone we want. But in what manner? We could toggle abruptly from (+amplitude) to (-amplitude) at that frequency if we wanted to, giving a square wave representation. Or, as was our choice, we could vary our amplitude values along a mathematical curve such as that which we’d recognise as a “sine wave”.

This sample based approach to modulation and demodulation is known as Pulse Code Modulation or PCM for short.

 

The Practical Bit - The AudioAccessory Application

 

The sample application "AudioAccessory" was developed to illustrate the basic approach to using audio as a means of integrating devices as described in “The Theory Part” above. AudioAccessory lets you type some text and transmit this text as data, encoded as a series of audible beeps via the headphone socket. It can also receive audio via the microphone and if it’s correctly encoded, convert the audio into text and display it on the UI.
We didn’t develop every line of code in AudioAccessory from scratch. In fact we plundered a couple of juicy sample applications that our colleagues Roberto Speranza and Gurtej Sandhu have already published in our GitHub repositories. I’ve included links to those applications at the end of this article.

 
The Physical and Electrical Layers

 

Before we get into the detail of our AudioAccessory application, let's fill in a couple of gaps in the theory. I presented a stack in the theory section but it wasn’t the complete stack. At the very bottom are of course the electrical and physical layers. If you intend to plug a gizmo into a BlackBerry 10 smart phone’s headphone socket, you’ll need to do so with the right kind of plug and for that plug to exhibit the right electrical properties.
The physical part is easy enough; the correct type of plug is a 4 ring “TRRS” plug. Here’s a picture one in case you’re not familiar with the term TRRS:

 

4_ring_plug.png

Figure 5 - TRRS 3.5mm plug

 

If what you want to do is output audio then there are no particular considerations regarding the electrical properties. But if you’re intending to use the headphone socket as an external microphone socket and receive audio for decoding, then for the BlackBerry 10 device to detect what it believes to be a microphone plugged into that socket, it must find a specific set of impedances between the pins. Values of:  32ohm / 32ohm / 2kohm between ground and each of the T, R1 and S pins, where ground is Ring 2 are the nominal values for this to work.

As you’ll see in the accompanying video, we found a way of cheating here which didn’t require any soldering.

  

The AudioAccessory Application

Here’s what the UI for AudioAccessory looks like:

 

IMG_00000320_360.png

Figure 6 - AudioAccessory UI

 

If you want to see the application in action, there's a demonstration included in the YouTube video I linked to above.

 

AudioAccessory Architecture

 

AudioAccessory uses a form of the Producer/Consumer pattern to decouple input and output processing, each of which works at quite different rates.

 

Capturing audio

 

When capturing audio, one thread, the “capture thread”, reads PCM audio samples from the BlackBerry 10 audio system, and deposits the data into a circular buffer. Meanwhile, another thread, the “demodulate thread” performs blocking reads on the circular buffer, demodulates data that appears there and hands it to the higher layers of the application via call backs.

 

capture.png

Figure 7 - Capturing audio

 

Emitting Tones

When we emit text encoded as audio tones, our text gets split into single character “messages” and placed into a queue by one thread, whilst another thread fetches messages from the queue and performs the encoding, modulation and emission of the tone via the BlackBerry audio APIs.

 

emit.png

Figure 8 - Emitting audio tones

 

Let’s follow through the steps required to take some text from the UI, entered by the user, through to the point at which it is transmitted as a series of audio tones. As mentioned in the architecture section, we use a producer consumer style pattern and the text entered by the user gets stored in a queue, one character at a time. Given the 5 bit encoding scheme we’re using, we make sure all characters are upper case as we do this.

 

void AudioService::addMessageToQueue(const QString &text)
{
	for (int i = 0; i < text.length(); i++) {
		addMessage(text.at(i).toUpper().toAscii());
	}
}

Adding text “messages” to processing queue

 

You’ll find the addMessage function in our AudioPcm.c source file and if you take a look you’ll see how it stores our characters away in memory in a linked list created using C pointers. If you prefer, I see no reason why you could not use one of the higher level Qt classes for this.

 

So, we accumulate characters in our queue. Meanwhile, in the generateMessages() function, we wait for characters to arrive in queue and then encode, modulate and transmit our data as audio. Here are a few of the key lines of code:

 

if (got_message_to_send) {
    message_to_send -= 0x30; // map ascii "0" to 0x00, etc. primitive encoding scheme
    modulateAndTransmit(baudotEncoding(message_to_send));    
    last_message_processed = currentMillis();
    free(message_to_generate);
}

Encode, modulate and transmit characters

 

Data Coding

 

In the baudotEncoding function, we convert a single character into a byte with a value according to the Baudot Code. 

 

static int baudotEncoding(int number)
{
   int character_to_send = number - 0x30; // map ascii "0" to 0x00, etc. 
   int check_sum = checkSum(character_to_send);
   int message = character_to_send + check_sum;
   LOG("createMessage() - character_to_send=%d, check_sum=%d, message=%d\n", character_to_send, check_sum, message);
   return message;
}

Applying Baudot coding to a character

 

Modulation

 

In the modulateAndTransmit(....) function we implement the Asynchronous Serial Data Transmission (ASDT) functionality, creating sine waves of one of two frequencies and for a given duration, to create a modulated, analogue representation of each of the bits in a Baudot encoded character, plus additional bits (e.g start and stop bits) required by ASDT. Having done so, we use the audio APIs to emit tones corresponding to those audio waves.

Here’s how we generate the data representing our sine waves in AudioAccessory:

 

for (index = 0; index < length; index ++) {

  current_angle = fmod(((double)index * 2.0 * M_PI * ((double) frequency / (double) 
                    SAMPLING_FREQUENCY )), 2.0 * M_PI);

  start[index] = (int16_t)(phase * amplitude * sin(current_angle));

}

Generating a sine wave

 

You’ll find this code in the generateTone function of AudioPcm.c. The variable frequency takes one of two values, FREQUENCY_HIGH or FREQUENCY_LOW according to whether we’re modulating a 1 or a 0 of course.

 

Emitting Audio

 

In modulateAndTransmit, you’ll see that the final steps are to emit tones, each representing a single bit and overall, comprising the bits required by the Asynchronous Serial Data Transmission scheme.

 

// Asynchronous Serial Data Transmission
writeTone(silence, silence_size);
writeTone(carrier, carrier_size);
writeTone(start_bit, start_size);
writeTone(data_bits, data_size);
writeTone(stop_bit, stop_size);
writeTone(end_bit, end_size);
writeTone(silence, silence_size);

Transmission of ASDT tone sequence

 

 

In the writeTone function, we use the asound library functions to prepare and then emit a tone.

 

error = snd_pcm_plugin_write (g_pcm_handle_p, start, size);

Emitting a tone using the asound library

 

snd_pcm_plugin_write takes three parameters, comprising a handle to an audio device and the start address and size of a buffer containing our PCM sample.

 

That’s all folks!

 

So that’s it for our introduction to the world of audio based accessory integration. We hope you’ve found this interesting and look forward to seeing what you use these techniques for in your own applications.

 

Contacts

 

You can contact Martin or John either through the BlackBerry support forums or through Twitter®:

  

     
Martin mwoolley @mdwrim
John jomurray @jcmrim

 

References

 

Video Presentation on this topic:

http://youtu.be/cQzL-5b1P_g

 

AudioAccessory application code:

https://github.com/blackberry/Cascades-Community-Samples/tree/master/AudioAccessory

 

BlackBerry 10 Bluetooth LE developer resources:

http://supportforums.blackberry.com/t5/Native-Development/BlackBerry-10-Bluetooth-LE-resource-index/...

 

BlackBerry NFC developer resources:

http://supportforums.blackberry.com/t5/Java-Development/NFC-Article-and-Code-Index/ta-p/1538775

 

Software Modems:

http://en.wikipedia.org/wiki/Softmodem

 

BlackBerry 10 audio APIs:

http://developer.blackberry.com/native/reference/bb10/audio_libref/topic/summary.html

 

BlackBerry 10 audio applications:

https://github.com/blackberry/Core-Native-Community-Samples/tree/master/AudioLoopBackSample  by Gurtej Sandhu

https://github.com/blackberry/Core-Native-Community-Samples/tree/master/ToneGenerator  by Roberto Speranza

 

Baudot code:

http://en.wikipedia.org/wiki/Baudot_code

 

Frequency Shift Keying:

http://en.wikipedia.org/wiki/Frequency-shift_keying

 

ASDT:

http://www.electronics.dit.ie/staff/tscarff/serial_comms/serial_comms.htm  

 

Pulse Code Modulation:

https://en.wikipedia.org/wiki/Pulse-code_modulation