You can see all of the repositories using this link:
http://github.com/boundarydevices

The most active at the moment are:

Linux on i.MX6, branch boundary-imx-android-r13.2.

U-Boot on i.MX6, branch boundary-imx-android-r13.2

As you can probably guess from the branch names, these were based on Freescale’s Android R13.2 release. We hope to be moving to main-line U-Boot shortly.

In this blog post, we’ll explain three key environment variable conventions used on the Nitrogen6X (and Sabre Lite before it).

For the impatient:

• Both of these devices ship with U-Boot configured to load and run a boot script named 6q_upgrade from the root of either SD card.
• A pre-configured command upgradeu can be used to upgrade U-Boot in serial flash
• Another pre-configured command clearenv will erase the environment variables in serial flash.

For those that want an explanation:

Both of these boards boot to SPI EEPROM (serial flash), and both U-Boot and its’ environment variables are stored there.

Convention for many modern ARM boards, including i.MX in recent years has been to boot directly to SD card, either by using unformatted areas of the disk or carefully placing special files in prescribed areas of a filesystem.

We jokingly refer to this as a game of “Hide the U-Boot”, and don’t want to play.

Instead, we added a serial flash chip and store the boot loader there. The flash device size is 2MB, which is plenty of space for the boot loader and environment, but not enough for a complete O/S, and we defer that to SD card, which brings us to the first environment variable:

bootcmd

We’ve pre-configured the bootcmd environment variable to load and launch a boot script from the root directory of either SD card slot by using the looping features of the Hush parser:

for disk in 0 1 ; do
	mmc dev ${disk} ;
        for fs in fat ext2 ; do
                ${fs}load mmc ${disk}:1 10008000 /6q_bootscript &&
                source 10008000 ;
        done ;
done

If you’re not familiar with U-Boot boot scripts, we wrote a backgrounder in this blog post.

To make compiling boot scripts a bit easier, we created a web-based tool to compile them for you. There’s also a bash snippet to make this easy from the command-line.

upgradeu

Once you’re running U-Boot from serial EEPROM, the question is how do you upgrade it?

The easy answer is that you ask U-Boot to do it:

MX6Q SABRELITE U-Boot > run upgradeu
...bunch of debug spew
## Executing script at 10008000
check U-Boot
Loading file "u-boot.bin" from mmc device 1:1 (xxb1)
179632 bytes read
...more debug spew...
SUCCESS
Total of 179632 bytes were the same
------- U-Boot versions match

If it’s not entirely obvious to you what this example illustrated, let’s dig a little deeper.

If you run print at the U-Boot prompt, you’ll see that upgradeu is defined to contain some U-Boot commands:

MX6Q SABRELITE U-Boot > print
bootdelay=3
...
upgradeu=for disk in 0 1 ; do mmc dev ${disk} ;for fs in fat ext2 ; do ${fs}load mmc ${disk}:1 10008000 /6q_upgrade && source 10008000 ; done ; ...
Environment size: 908/8188 bytes

That’s a pretty long command-line, but is essentially the same loop as we described in the bootcmd section of this post. It will try to read a script named 6q_upgrade from either SD card using either the FAT or ext2/3/4 filesystem and source it. The source command simply executes a boot script.

If you’ve followed along this far, you may have just realized that we haven’t really followed either the full boot path or the upgrade path because we’ve waved our hands over what the script does. We’ll address that in a later section.

clearenv

The final convention may be the most useful for the programmer. If a Sabre Lite or Nitrogen6X board spends much time on your desk, you’ll invariably set something in the U-Boot environment. Perhaps you’re bypassing all of these conventions and booting over NFS (the topic of another blog post). Maybe you’re booting to a SATA drive or a USB stick or tweaking some kernel parameters.

If so, you may ultimately want to put things back the way you found them before you hand the device off to someone else. If you know the layout of the serial EEPROM, you can simply erase the sector(s) containing the environment variables and go back to the compiled-in defaults.

I’m assuming you don’t (know or remember the flash layout) because I don’t and I helped come up with it. This is where the clearenv variable or command comes into play. Going back to the print example will tell you:

MX6Q SABRELITE U-Boot > print
bootdelay=3
...
clearenv=sf probe 1 && sf erase 0xc0000 0x2000 && echo restored environment to factory default
...
Environment size: 908/8188 bytes

Aha! The environment is stored at offset 0xc0000 (768k) and is 0x2000 (8192) bytes long!

And if you want to clear it, you can simply run run clearenv from the shell.

Unless you’re as much of a U-Boot geek as I am, you’re probably wishing you’d stopped at the for the impatient section, but you may someday need the details, so I’m going to describe a handful of additional things that will help in the use and deployment of boot scripts for your use.

To begin with, I promised the details of the boot script itself, but I lied.

I can’t really give that to you because the things needed really depend on your environment. If you’ve spent time with various embedded Linux distributions, you’ll find that many of them require different sorts of kernel command-line parameters. Also, some distributions, like Android require the use of a RAM disk, but others don’t. These differences (additions) are best expressed in the actual userspace image, so we add them there.

Here’s an example of what we shipped on SabreLite boards with Ubuntu:

Ubuntu boot script

set bootargs $bootargs root=/dev/mmcblk0p1 rootwait fixrtc ;
ext2load mmc ${disk}:1 10800000 /boot/uImage && bootm 10800000 ;
echo "Error loading kernel image"

If you look closely, you’ll see that it’s adding a couple of things to the bootargs environment variable, then loading the kernel from /boot/uImage and booting it using bootm and that it’s printing an error message in case something fails. The bootm command will normally not return.

Also note that this script uses the disk variable that was defined in the loop from which it was called, so the boot script must reside on the same disk as the kernel.

Android boot script

Here’s another example of loading Android ICS on an i.MX6:

${fs}load mmc ${disk}:1 10800000 uimage &&
${fs}load mmc ${disk}:1 12800000 uramdisk.img &&
bootm 10800000 12800000 ;
echo "Error loading kernel image"

This one doesn’t add anything to the bootargs, and uses the && operators liberally, but the primary thing it does differently from the Ubuntu script is load a ram disk (uramdisk.img). It also uses the fs parameter from the calling loop in the same way the Ubuntu script used the disk variable.

A lot more could be done inside this boot script. The Android boot script could (and probably should) handle recovery mode. Both of them should probably loop in the case of errors, although we don’t yet have display support in i.MX6 U-Boot.

We should also add support for SATA into our default environment, but those are topics for another day and perhaps another blog post.

Upgrade script

The content of the 6q_upgrade script is a little more complex, but shouldn’t be too surprising. It tries to load a file named u-boot.bin from the same partition in which the script was found and compares it against the content of the serial EEPROM. If the two are different, it will wait for 10 seconds before (re)programming the flash to match the content of the SD card. It will also perform a verification pass before telling you that things are good:

echo "check U-Boot" ;
if ${fs}load mmc ${disk}:1 12000000 u-boot.bin ; then
      echo "read $filesize bytes from SD card" ;
      if sf probe 1 27000000 ; then
	   echo "probed SPI ROM" ;
           if sf read 0x12400000 0 $filesize ; then
               if cmp.b 0x12000000 0x12400000 $filesize ; then
                   echo "------- U-Boot versions match" ;
               else
                   echo "Need U-Boot upgrade" ;
                   echo "Program in 10 seconds" ;
                   for n in 9 8 7 6 5 4 3 2 1 0 ; do
                        echo $n ;
                        sleep 1 ;
                   done
		   echo "erasing" ;
                   sf erase 0 0x40000 ;
		   # two steps to prevent bricking
		   echo "programming" ;
                   sf write 0x12000000 0 $filesize ;
		   echo "verifying" ;
                   if sf read 0x12400000 0 $filesize ; then
                       if cmp.b 0x12000000 0x12400000 $filesize ; then
                           while echo "---- U-Boot upgraded. reset" ; do
				sleep 120
			   done
                       else
                           echo "Read verification error" ;
                       fi
                   else
                        echo "Error re-reading EEPROM" ;
                   fi
               fi
           else
               echo "Error reading boot loader from EEPROM" ;
           fi
      else
           echo "Error initializing EEPROM" ;
      fi ;
else
     echo "No U-Boot image found on SD card" ;
fi

Recap and best practices

I should probably move this section to the top of the document, but here’s some additional rationale for how and why we use the boot script scheme.

Two-file U-Boot upgrades

To build an SD card to upgrade U-Boot, you can copy just two files (6q_upgrade and u-boot.bin)to a fresh, out-of-the-box SD card (which tend to come formatted as FAT32), insert the card and run run upgradeu.

Single partition Ubuntu

To run Ubuntu, LTIB or Timesys single-partition images, you can create an SD card by simply formatting an ext2/3/4 partition and copy your filesystem image, boot script and kernel using cp. There’s no need to use dd or any other special formatting commands.

Windows CE, too

I guess I should say Windows Embedded 7. As the name implies, U-Boot is the Universal Boot Loader, and it can load and launch Windows Embedded using a boot script in a very similar manner as Linux. In this environment, you shouldn’t need to format the SD card at all: FAT32 should work.

And NFS

It may not be clear, but the use of a boot script on SD can also be useful in booting over NFS. This may sound like an oxymoron, but it’s often more convenient to write a network load script and copy it to an SD card than it is to hook up a serial port and tweak settings there.

This is especially true for those that infrequently use a serial port or when devices are enclosed in a manner that makes access to the serial port difficult.

Best practices

If U-Boot and environment variables are all stored on SD card, you don’t have much choice about what goes where, and environment variables are often used and changed to reflect the combination of distribution-related variables like whether or not a RAM disk is loaded and machine-specific variables like ethernet addresses or display settings.

When the two are separated as they are in SabreLite and Nitrogen6 boards, you should try to use environment variables for only those things which are tied to the board, such as the type of display. Any distribution-related bootargs should be added by the boot script.

If we haven’t yet convinced you that this is a good idea, I hope that at least you’ll understand what you’re seeing when you receive a board on your desk. At that point, I’ll refer you to the Linaro way, which allows you to override all of this and play “Hide the U-Boot” using a small shim in SPI ROM.

For what it’s worth, you can load and program that using the upgradeu command and the 6q_upgrade script by naming the shim u-boot.bin.

Since that time, we’ve been shipping demo software based on LTIB and Qt with our Nitrogen53 and Nitrogenboards, and the lack of documentation shows.

In order to remedy that and allow our customers to replicate our results, this post will walk through the steps to build an Ubuntu-based VirtualBox image that contains LTIB and Qt. A set of future blog posts will detail the steps we’ve used to translate that build into our standard Qt demo package.

Since developer machines come and go, the VM image is a good way to control our build process and minimize the number of unknowns when deploying systems.

Step 1: Build the VM

We built the VirtualBox image by starting with the Ubuntu Minimal CD to try and keep things small. Our purpose is setting up a build machine, so we started with the “OpenSSH Server” profile, which allows us to fork the VM off into a second workspace and primarily access the device over SSH.

We chose the 32-bit PC (x86) distribution and used 10.04 “Lucid Lynx”.

We created a single user: boundary, password boundary.

Step 2: Configure with pre-requisites

In order to simplify things, we captured the full set of Ubuntu packages included on the build machine and placed on our web-site at
http://boundarydevices.com/ltib-installed-packages. You can copy these by piping them into apt-get install like so:

boundary@ubuntu:~$ wget http://boundarydevices.com/ltib-installed-packages
--2011-12-10 18:10:08--  http://boundarydevices.com/ltib-installed-packages
Resolving boundarydevices.com... 69.89.31.84
Connecting to boundarydevices.com|69.89.31.84|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6093 (6.0K) 
Saving to: `ltib-installed-packages'

100%[======================================>] 6,093       --.-K/s   in 0.04s

2011-12-10 18:10:08 (144 KB/s) - `ltib-installed-packages' saved [6093/6093]

boundary@ubuntu:~$ cat ltib-installed-packages | xargs sudo apt-get install
[sudo] password for boundary:
Reading package lists... Done
Building dependency tree
Reading state information... Done
adduser is already the newest version.
apparmor is already the newest version.
...

Step 3: Install LTIB

In this distribution, we started with the package L2.6.35_11.09.01_ER_source_bundle.tar.gz from Freescale, and the L2.6.35_11.09.01_ER_source.tar.gz tar-ball within it.

After extracting, we installed LTIB in the normal way into /home/boundary/ltib:

boundary@ltib-qt-4:~$ sudo mkdir -p /opt/freescale/
boundary@ltib-qt-4:~$ sudo chown boundary.boundary /opt/freescale
boundary@ltib-qt-4:~$ cd L2.6.35_11.09.01_ER_source/
boundary@ltib-qt-4:~/L2.6.35_11.09.01_ER_source$ ls
EULA     ltib.tar.gz           pkgs                tftp.zip
install  package_manifest.txt  redboot_201003.zip
boundary@ltib-qt-4:~/L2.6.35_11.09.01_ER_source$ ./install

You are about to install the LTIB (GNU/Linux Target Image Builder)

Before installing LTIB, you must read and accept the EULA
(End User License Agreement) which will be presented next.

Do you want to continue ? Y|n
Y

              FREESCALE SEMICONDUCTOR SOFTWARE LICENSE AGREEMENT

IMPORTANT. Read the following Freescale Semiconductor Software License Agreement
("Agreement") completely. By selecting the "I Accept" button at the end of this
page, you indicate that you accept the terms of this Agreement. You may then
download the file.

...

I have read and accept the EULA (yes|no):
yes

The LTIB files are extracted from a tar file which includes the
prefix ltib.  After installation you will find LTIB in:
/home/boundary/L2.6.35_11.09.01_ER_source/ltib

Where do you want to install LTIB ? (/home/boundary/L2.6.35_11.09.01_ER_source)
/home/boundary
Making target directory /home/boundary/ltib
Installing LTIB to /home/boundary/ltib
ltib/
ltib/hash
ltib/doc/
...

Following the LTIB directions, we also added this line to /etc/sudoers:

boundary ALL= NOPASSWD: /usr/bin/rpm, /opt/freescale/ltib/usr/bin/rpm

And edited ltib as described here.

    no_sudo_check => 0,  # fc9 work-around

Step 4: Run LTIB

In the earlier posts on bringing up LTIB, I described running ltib -m config to configure the toolchain and packages for LTIB. In other words, I described an interactive process.

For production use, it’s better to simply save and restore the configuration file produced by the U/I.

The LTIB configuration used in this VirtualBox build is available here. In order to restore it, use the ltib --selecttype option as shown below.

boundary@ubuntu:~$ wget http://boundarydevices.com/ltib_11.03.config
--2011-12-10 19:44:54--  http://boundarydevices.com/ltib_11.03.config
Resolving boundarydevices.com... 69.89.31.84
Connecting to boundarydevices.com|69.89.31.84|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1675 (1.6K) 
Saving to: `ltib_11.03.config.3'

100%[======================================>] 1,675       --.-K/s   in 0s

2011-12-10 19:44:54 (173 GB/s) - `ltib_11.03.config.3' saved [1675/1675]

boundary@ubuntu:~$ cd ltib && ./ltib --selectype

After running selectype, LTIB will run for a while. Go get a cup of coffee and check in with it periodically.

Note that SSH is up and running now, and if you configure the VM with a bridged network adapter, you can run ifconfig eth0 in the virtual machine to find its’ IP address and use SSH to access it. Because you can run this in a console window on your development machine, Cut and Paste and screen scraping becomes much easier. Once things are configured and I know the IP address of the VM, I typically stash the VM console window on an unused Workspace on my Ubuntu development machine.

If everything proceeds as planned, your LTIB build will complete with this happy banner:

Started: Sun Dec 11 11:42:07 2011
Ended:   Sun Dec 11 11:42:21 2011
Elapsed: 14 seconds

Build Succeeded

The output of the build is placed in /home/boundary/ltib/rootfs. The size is fairly
large, but can be trimmed down later (a topic that deserves more than one blog post of its own).

boundary@ltib-qt:~$ du -hs /home/boundary/ltib/rootfs
663M	/home/boundary/ltib/rootfs

Step 5: Set up symlink for Qt

If you’ve looked at the other post on building Qt, or if you’ve taken a look at the qmake.conf file, you should notice that the include and linker specs are specified using the path /tftpboot/ltib. This is done primarily to avoid having references to home directories and to allow multiple instances of LTIB to be placed on a development machine.

To set this up, you should simply create a /tftpboot directory and a symlink from /tftpboot/ltib to the LTIB root filesystem. In this post, that directory is /home/boundary/ltib/rootfs.

boundary@ltib-qt:~$ sudo mkdir /tftpboot
boundary@ltib-qt:~$ sudo ln -sf /home/boundary/ltib/rootfs /tftpboot/ltib

Step 6: Configure Qt

Now that LTIB is installed, you can move on to configuring Qt.

To make things easy and reproducible, we placed some configuration files in this zip file. It consists of a configuration shell script (do_config_qt-4.7.1) and the LTIB configuration directory mkspecs/linux-mxc-g++/.

You can grab it and the Qt sources themselves using wget.

boundary@ltib-qt:~$ wget http://get.qt.nokia.com/qt/source/qt-everywhere-opensource-src-4.7.1.tar.gz
--2011-12-11 11:51:59--  http://get.qt.nokia.com/qt/source/qt-everywhere-opensource-src-4.7.1.tar.gz
Resolving get.qt.nokia.com... 208.111.145.207, 69.28.182.56
Connecting to get.qt.nokia.com|208.111.145.207|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 211768512 (202M) [application/octet-stream]
Saving to: `qt-everywhere-opensource-src-4.7.1.tar.gz'

 2% [                                       ] 4,239,502    416K/s  eta 8m 45s

boundary@ltib-qt:~$ tar zxvf qt-everywhere-opensource-src-4.7.1.tar.gz
qt-everywhere-opensource-src-4.7.1/
qt-everywhere-opensource-src-4.7.1/qmake/
qt-everywhere-opensource-src-4.7.1/qmake/property.cpp
qt-everywhere-opensource-src-4.7.1/qmake/meta.h
qt-everywhere-opensource-src-4.7.1/qmake/Makefile.win32-g++
...
boundary@ltib-qt:~$ cd qt-everywhere-opensource-src-4.7.1
boundary@ltib-qt:~/qt-everywhere-opensource-src-4.7.1$ wget http://boundarydevices.com/qt-config-mx5x.zip
--2011-12-11 12:01:46--  http://boundarydevices.com/qt-config-mx5x.zip
Resolving boundarydevices.com... 69.89.31.84
Connecting to boundarydevices.com|69.89.31.84|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3324 (3.2K) [application/zip]
Saving to: `qt-config-mx5x.zip'

100%[======================================>] 3,324       --.-K/s   in 0.001s

2011-12-11 12:01:47 (3.45 MB/s) - `qt-config-mx5x.zip' saved [3324/3324]

boundary@ltib-qt:~/qt-everywhere-opensource-src-4.7.1$ unzip qt-config-mx5x.zip
Archive:  qt-config-mx5x.zip
  inflating: do_config_qt-4.7.1
   creating: mkspecs/linux-mxc-g++/
  inflating: mkspecs/linux-mxc-g++/qplatformdefs.h
  inflating: mkspecs/linux-mxc-g++/qmake.conf

If you take a look at the file do_config_qt-4.7.1, you’ll see that it sets a few environment variables, so you’ll want to run it with the source or dot operator:

boundary@ltib-qt:~/qt-everywhere-opensource-src-4.7.1$ . ./do_config_qt-4.7.1
Determining system architecture... (Linux:2.6.32-36-generic:i686)
    32-bit Intel 80x86 (i386)
    'arm' is supported
    'i386' is supported
System architecture: 'arm'
Host architecture: 'i386'

You have asked to use pkg-config and are cross-compiling.
Please make sure you have a correctly set-up pkg-config
environment!

... lots of spew here...

Qt is now configured for building. Just run 'make'.
Once everything is built, you must run 'make install'.
Qt will be installed into /usr/local/Trolltech/Qt-4.7.1/

Step 7: Build and install Qt

Compared with the steps above, this step is really easy:

boundary@ltib-qt:~/qt-everywhere-opensource-src-4.7.1$ make
cd src/tools/bootstrap/ && make -f Makefile
make[1]: Entering directory `/home/boundary/qt-everywhere-opensource-src-4.7.1/src/tools/bootstrap'
...
make[3]: Leaving directory `/home/boundary/qt-everywhere-opensource-src-4.7.1/demos/spectrum/app'
make[2]: Leaving directory `/home/boundary/qt-everywhere-opensource-src-4.7.1/demos/spectrum'
make[1]: Leaving directory `/home/boundary/qt-everywhere-opensource-src-4.7.1/demos'
boundary@ltib-qt:~/qt-everywhere-opensource-src-4.7.1$ sudo make INSTALL_ROOT=~/ltib/rootfs/ install
[sudo] password for boundary:
cd src/tools/bootstrap/ && make -f Makefile install
make[1]: Entering directory `/home/boundary/qt-everywhere-opensource-src-4.7.1/src/tools/bootstrap'
...
cp -f -r /home/boundary/qt-everywhere-opensource-src-4.7.1/mkspecs/wincewm60standard-msvc2008 /home/boundary/ltib/rootfs//usr/local/Trolltech/Qt-4.7.1//mkspecs/
cp -f -r /home/boundary/qt-everywhere-opensource-src-4.7.1/mkspecs/wincewm65professional-msvc2005 /home/boundary/ltib/rootfs//usr/local/Trolltech/Qt-4.7.1//mkspecs/
cp -f -r /home/boundary/qt-everywhere-opensource-src-4.7.1/mkspecs/wincewm65professional-msvc2008 /home/boundary/ltib/rootfs//usr/local/Trolltech/Qt-4.7.1//mkspecs/
boundary@ltib-qt:~/qt-everywhere-opensource-src-4.7.1$

As shown above, this configuration installed Qt libraries into /home/boundary/ltib/rootfs, and
specifically, into the usr/local/ tree within it.

If you’ve followed this post to this point, congratulations! You now have a usable build of
Linux and Qt. I’ll follow up in future posts with how you can create a production-ready distribution
based on this core.

Wrapping it up

This VirtualBox image is available here.

You can use it in a number of different ways:

1. You can install the nfs-kernel-server package, export it and boot a Nitrogen53 directly to the VM image.
2. You can grab the /opt/freescale and /home/boundary/ltib directories and use them to compile and link your applications.
3. You can use it as a quick-start to cross-compilation without going through all of these steps yourself if you’re evaluating a Nitrogen or Nitrogen53 board.
4. You can use it for comparison with your own build.

The last of these is really the intent of this post.

<intent-filter>
   <action android:name="android.intent.action.MAIN" />
   <category android:name="android.intent.category.HOME" />
   <category android:name="android.intent.category.DEFAULT" />
   <category android:name="android.intent.category.MONKEY"/>
</intent-filter>

My early debug steps involved instrumenting the kernel, but I really wanted the quicker cycle time of the i2c-tools package. It turns out that this was trivially easy to do. The i2c-tools package is very clean and the Android build system makes generating a makefile simple.

I simply copied my source tree into hardware/i2c-tools-3.0.2, added the following Android.mk, and it worked! The following is the content of Android.mk that I came up with by a quick scan of the Makefile.

LOCAL_PATH:= $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE_TAGS := eng
LOCAL_C_INCLUDES += $(LOCAL_PATH) $(LOCAL_PATH)/$(KERNEL_DIR)/include
LOCAL_SRC_FILES := tools/i2cbusses.c tools/util.c
LOCAL_MODULE := i2c-tools
include $(BUILD_STATIC_LIBRARY)

include $(CLEAR_VARS)

LOCAL_MODULE_TAGS := eng
LOCAL_SRC_FILES:=tools/i2cdetect.c
LOCAL_MODULE:=i2cdetect
LOCAL_CPPFLAGS += -DANDROID
LOCAL_SHARED_LIBRARIES:=libc
LOCAL_STATIC_LIBRARIES := i2c-tools
LOCAL_C_INCLUDES += $(LOCAL_PATH) $(LOCAL_PATH)/$(KERNEL_DIR)/include
include $(BUILD_EXECUTABLE)

include $(CLEAR_VARS)

LOCAL_MODULE_TAGS := eng
LOCAL_SRC_FILES:=tools/i2cget.c
LOCAL_MODULE:=i2cget
LOCAL_CPPFLAGS += -DANDROID
LOCAL_SHARED_LIBRARIES:=libc
LOCAL_STATIC_LIBRARIES := i2c-tools
LOCAL_C_INCLUDES += $(LOCAL_PATH) $(LOCAL_PATH)/$(KERNEL_DIR)/include
include $(BUILD_EXECUTABLE)

include $(CLEAR_VARS)

LOCAL_MODULE_TAGS := eng
LOCAL_SRC_FILES:=tools/i2cset.c
LOCAL_MODULE:=i2cset
LOCAL_CPPFLAGS += -DANDROID
LOCAL_SHARED_LIBRARIES:=libc
LOCAL_STATIC_LIBRARIES := i2c-tools
LOCAL_C_INCLUDES += $(LOCAL_PATH) $(LOCAL_PATH)/$(KERNEL_DIR)/include
include $(BUILD_EXECUTABLE)

include $(CLEAR_VARS)

LOCAL_MODULE_TAGS := eng
LOCAL_SRC_FILES:=tools/i2cdump.c
LOCAL_MODULE:=i2cdump
LOCAL_CPPFLAGS += -DANDROID
LOCAL_SHARED_LIBRARIES:=libc
LOCAL_STATIC_LIBRARIES := i2c-tools
LOCAL_C_INCLUDES += $(LOCAL_PATH) $(LOCAL_PATH)/$(KERNEL_DIR)/include
include $(BUILD_EXECUTABLE)

diff --git a/AndroidManifest.xml b/AndroidManifest.xml
index 606f6f8..84ee599 100644
--- a/AndroidManifest.xml
+++ b/AndroidManifest.xml
@@ -50,7 +50,6 @@
         android:label="@string/permlab_write_settings"
         android:description="@string/permdesc_write_settings"/>

-

diff --git a/src/com/android/launcher2/LauncherApplication.java b/src/com/androi
index 85a9b9d..eda92d9 100644
--- a/src/com/android/launcher2/LauncherApplication.java
+++ b/src/com/android/launcher2/LauncherApplication.java
@@ -23,16 +23,10 @@ import android.content.IntentFilter;
 import android.database.ContentObserver;
 import android.os.Handler;
 import dalvik.system.VMRuntime;
-import android.os.PowerManager;
-import android.app.Activity;
-import android.content.Context;
-import android.content.Context;
-import android.content.Context;
--- a/AndroidManifest.xml
diff --git a/AndroidManifest.xml b/AndroidManifest.xml
diff --git a/AndroidManifest.xml b/AndroidManifest.xml
index 606f6f8..84ee599 100644
--- a/AndroidManifest.xml
+++ b/AndroidManifest.xml
@@ -50,7 +50,6 @@
         android:label="@string/permlab_write_settings"
         android:description="@string/permdesc_write_settings"/>

-

diff --git a/src/com/android/launcher2/LauncherApplication.java b/src/com/android/launcher2/LauncherApplication.java
index 85a9b9d..eda92d9 100644
--- a/src/com/android/launcher2/LauncherApplication.java
+++ b/src/com/android/launcher2/LauncherApplication.java
@@ -23,16 +23,10 @@ import android.content.IntentFilter;
 import android.database.ContentObserver;
 import android.os.Handler;
 import dalvik.system.VMRuntime;
-import android.os.PowerManager;
-import android.app.Activity;
-import android.content.Context;
-import android.os.Bundle;
-import android.os.PowerManager;

 public class LauncherApplication extends Application {
     public LauncherModel mModel;
     public IconCache mIconCache;
-    protected PowerManager.WakeLock mWakeLock;

     @Override
     public void onCreate() {
@@ -58,10 +52,6 @@ public class LauncherApplication extends Application {
         ContentResolver resolver = getContentResolver();
         resolver.registerContentObserver(LauncherSettings.Favorites.CONTENT_URI, true,
                 mFavoritesObserver);
-
-        final PowerManager pm = (PowerManager) getSystemService(Context.POWER_S
-        this.mWakeLock = pm.newWakeLock(PowerManager.SCREEN_DIM_WAKE_LOCK, "My Tag");
-        this.mWakeLock.acquire();
     }

     /**

if ((mPowerState & (SCREEN_ON_BIT | SCREEN_BRIGHT)) != 0) {
    Slog.d(TAG, "No sleep till Brooklyn!");
    return;
}

    private void userActivity(long time, long timeoutOverride, boolean noChangeLights,
            int eventType, boolean force) {

        ...

        synchronized (mLocks) { 

            ...

            if (mLastEventTime <= time || force) {
                mLastEventTime = time;
                if ((mUserActivityAllowed && !mProximitySensorActive) || force) {
                    // Only turn on button backlights if a button was pressed
                    // and auto brightness is disabled
                    if (eventType == BUTTON_EVENT && !mUseSoftwareAutoBrightness) {
                        mUserState = (mKeyboardVisible ? ALL_BRIGHT : SCREEN_BUTTON_BRIGHT);
                    } else {
                        // don't clear button/keyboard backlights when the screen is touched.
                        mUserState |= SCREEN_BRIGHT;
                    }

                    int uid = Binder.getCallingUid();
                    long ident = Binder.clearCallingIdentity();
                    try {
                        mBatteryStats.noteUserActivity(uid, eventType);
                    } catch (RemoteException e) {
                        // Ignore
                    } finally {
                        Binder.restoreCallingIdentity(ident);
                    }
                    mWakeLockState = mLocks.reactivateScreenLocksLocked();
                    setPowerState(mUserState | mWakeLockState, noChangeLights,
                            WindowManagerPolicy.OFF_BECAUSE_OF_USER);
                    setTimeoutLocked(time, timeoutOverride, SCREEN_BRIGHT);
                }
            }
        }

        if (mPolicy != null) {
            mPolicy.userActivity();
        }
    }

Well, on an app-by-app basis, you could always update the manifest file. There’s a tag in there that defines the activity’s requested orientation. This isn’t really the best way to go about this, though, especially if you’re planning on using lots of apps this way.

Credit goes to android-porting group for establishing the right direction for people in our situation. The answer is in frameworks/base/policy/src/com/internal/policy/impl/PhoneWindowManager.java (whew). Among other things, this file provides rotation settings to the rest of the system. Check out rotationForOrientationLw():

    public int rotationForOrientationLw(int orientation, int lastRotation,
            boolean displayEnabled) {

        if (mPortraitRotation < 0) {             // Initialize the rotation angles for each orientation once.
            Display d = ((WindowManager)mContext.getSystemService(Context.WINDOW_SERVICE))
                .getDefaultDisplay();
            if (d.getWidth() > d.getHeight()) {
                mPortraitRotation = Surface.ROTATION_90;
                mLandscapeRotation = Surface.ROTATION_0;
                mUpsideDownRotation = Surface.ROTATION_270;
                mSeascapeRotation = Surface.ROTATION_180;
            } else {
                mPortraitRotation = Surface.ROTATION_0;
                mLandscapeRotation = Surface.ROTATION_90;
                mUpsideDownRotation = Surface.ROTATION_180;
                mSeascapeRotation = Surface.ROTATION_270;
            }
        }

        synchronized (mLock) {
            switch (orientation) {
                case ActivityInfo.SCREEN_ORIENTATION_PORTRAIT:
                    //always return portrait if orientation set to portrait
                    return mPortraitRotation;
                case ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE:
                    //always return landscape if orientation set to landscape
                    return mLandscapeRotation;
                case ActivityInfo.SCREEN_ORIENTATION_REVERSE_PORTRAIT:
                    //always return portrait if orientation set to portrait
                    return mUpsideDownRotation;
                case ActivityInfo.SCREEN_ORIENTATION_REVERSE_LANDSCAPE:
                    //always return seascape if orientation set to reverse landscape
                    return mSeascapeRotation;
                case ActivityInfo.SCREEN_ORIENTATION_SENSOR_LANDSCAPE:
                    //return either landscape rotation based on the sensor
                    mOrientationListener.setAllow180Rotation(false);
                    return getCurrentLandscapeRotation(lastRotation);
                case ActivityInfo.SCREEN_ORIENTATION_SENSOR_PORTRAIT:
                    mOrientationListener.setAllow180Rotation(true);
                    return getCurrentPortraitRotation(lastRotation);
            }

            mOrientationListener.setAllow180Rotation(
                    orientation == ActivityInfo.SCREEN_ORIENTATION_FULL_SENSOR);

            // case for nosensor meaning ignore sensor and consider only lid
            // or orientation sensor disabled
            //or case.unspecified
            if (mLidOpen) {
                return mLidOpenRotation;
            } else if (mDockMode == Intent.EXTRA_DOCK_STATE_CAR && mCarDockRotation >= 0) {
                return mCarDockRotation;
            } else if (mDockMode == Intent.EXTRA_DOCK_STATE_DESK && mDeskDockRotation >= 0) {
                return mDeskDockRotation;
            } else {
                if (useSensorForOrientationLp(orientation)) {
                    return mOrientationListener.getCurrentRotation(lastRotation);
                }
                return Surface.ROTATION_0;
            }
        }
    }

The first block defines the rotation angles, and then the rest administrates them on request. Most utility apps don’t request a particular rotation, however; instead, they leave it up to the motion sensor. So, look in the last block, which dictates behavior when the orientation sensor is disabled. Since we aren’t worried about dock mode, car mode, or open lid, the last line is what gets called. That last line (line 59 in the snippet) is what ends up determining the rotation that gets dictated to most apps in our situation. Try changing ROTATION_0 to ROTATION_270, rebuild and see what happens.

Cool! Portrait mode!

EDIT: One thing I found out while working on an upcoming post… The screen orientation defined in Launcher2′s manifest file is “nosensor”, which is why the process I described above works.  This sets the behavior for activities opened on top of Launcher — most of the default apps don’t even have a defined orientation.  If you are set up to open an app other than Launcher on boot, you will need to define the screenOrientation in their manifest, otherwise you’ll get the default landscape orientation. If you made the above changes, “nosensor” should work fine.

Before we start, you’ll want to look at the javadoc for the Camera and MediaRecorder classes.  You’ll see step-by-step instructions for using each in your application.  What follows provides context, so you can see how they are used, but ultimately, your app might look different.  So, check out the above links first and you’ll know what to look for in the code snippets below.

First, a few notes about structure. The main activities are Camera and VideoCamera. The actual camera device is wrapped up in a CameraHolder class which is designed to alleviate the performance impact from having to juggle repeated calls to Camera.open() and Camera.release() as the user switches between the video and still camera activities. There is also a ImageCapture class (a subclass of Camera) which abstracts the entire process of taking a still picture and saving it locally. So an ImageCapture object represents the picture you are taking or have just finished taking.

By the way, the methods presented in the code snippets below don’t necessarily appear in that order in the source code.  Here, they are arranged for clarity, with methods that call preceding the methods they are calling.

Initialization

    public void onCreate(Bundle icicle) {
        super.onCreate(icicle);

        setContentView(R.layout.camera);
        mSurfaceView = (SurfaceView) findViewById(R.id.camera_preview);
        ...

In the onCreate() method, which is called when the activity starts, you can see that it immediately goes and finds the SurfaceView it will be posting previews to. The activity itself implements SurfaceHolder.Callback, meaning that the SurfaceHolder held by that View gets handed back to the app via the surfaceChanged() callback. The important part is that there’s a reference to it for later.

Then it gets some settings from the framework, and spins off a preview thread. Inside the preview thread, the startPreview() method is called.

    private void startPreview() throws CameraHardwareException {
        if (mPausing || isFinishing()) return;

        ensureCameraDevice();

        // If we're previewing already, stop the preview first (this will blank
        // the screen).
        if (mPreviewing) stopPreview();

        setPreviewDisplay(mSurfaceHolder);
        Util.setCameraDisplayOrientation(this, mCameraId, mCameraDevice);
        setCameraParameters(UPDATE_PARAM_ALL);

        mCameraDevice.setErrorCallback(mErrorCallback);

        try {
            Log.v(TAG, "startPreview");
            mCameraDevice.startPreview();
        } catch (Throwable ex) {
            closeCamera();
            throw new RuntimeException("startPreview failed", ex);
        }
        mPreviewing = true;
        mZoomState = ZOOM_STOPPED;
        mStatus = IDLE;
    }

    private void ensureCameraDevice() throws CameraHardwareException {
        if (mCameraDevice == null) {
            mCameraDevice = CameraHolder.instance().open(mCameraId);
            mInitialParams = mCameraDevice.getParameters();
        }
    }

    private void setPreviewDisplay(SurfaceHolder holder) {
        try {
            mCameraDevice.setPreviewDisplay(holder);
        } catch (Throwable ex) {
            closeCamera();
            throw new RuntimeException("setPreviewDisplay failed", ex);
        }
    }

CameraHolder’s open() method returns an instance of android.hardware.Camera, which is saved locally. It sets the Camera’s preview display. Finally, it calls the device’s startPreview() method (and by the way, the Camera can only take pictures once startPreview() returns successfully). At this point, the camera preview picture should show up on the app’s UI. Easy!

Taking a picture

The code to actually take a picture is in the doSnap() method, called from callbacks associated with the shutter button and the UI capture button. It calls the onSnap() method of a previously instantiated ImageCapture object, which in turn calls its initiate() method, which then calls its capture() method. The capture() method sets some parameters such as orientation and GPS information (for geotagging), then finally calls the takePicture() method of the Camera object.

    private class ImageCapture {
        ...
        public void onSnap() {
            // If we are already in the middle of taking a snapshot then ignore.
            if (mPausing || mStatus == SNAPSHOT_IN_PROGRESS) {
                return;
            }
            mCaptureStartTime = System.currentTimeMillis();
            mPostViewPictureCallbackTime = 0;
            mHeadUpDisplay.setEnabled(false);
            mStatus = SNAPSHOT_IN_PROGRESS;

            mImageCapture.initiate();
        }

        public void initiate() {
            if (mCameraDevice == null) {
                return;
            }

            capture();
        }

        private void capture() {
            mCaptureOnlyData = null;

            // See android.hardware.Camera.Parameters.setRotation for
            // documentation.
            int rotation = 0;
            if (mOrientation != OrientationEventListener.ORIENTATION_UNKNOWN) {
                CameraInfo info = CameraHolder.instance().getCameraInfo()[mCameraId];
                if (info.facing == CameraInfo.CAMERA_FACING_FRONT) {
                    rotation = (info.orientation - mOrientation + 360) % 360;
                } else {  // back-facing camera
                    rotation = (info.orientation + mOrientation) % 360;
                }
            }
            mParameters.setRotation(rotation);

            // Clear previous GPS location from the parameters.
            mParameters.removeGpsData();

            // We always encode GpsTimeStamp
            mParameters.setGpsTimestamp(System.currentTimeMillis() / 1000);

            // Set GPS location.
            Location loc = mRecordLocation ? getCurrentLocation() : null;
            if (loc != null) {
                double lat = loc.getLatitude();
                double lon = loc.getLongitude();
                boolean hasLatLon = (lat != 0.0d) || (lon != 0.0d);

                if (hasLatLon) {
                    mParameters.setGpsLatitude(lat);
                    mParameters.setGpsLongitude(lon);
                    mParameters.setGpsProcessingMethod(loc.getProvider().toUpperCase());
                    if (loc.hasAltitude()) {
                        mParameters.setGpsAltitude(loc.getAltitude());
                    } else {
                        // for NETWORK_PROVIDER location provider, we may have
                        // no altitude information, but the driver needs it, so
                        // we fake one.
                        mParameters.setGpsAltitude(0);
                    }
                    if (loc.getTime() != 0) {
                        // Location.getTime() is UTC in milliseconds.
                        // gps-timestamp is UTC in seconds.
                        long utcTimeSeconds = loc.getTime() / 1000;
                        mParameters.setGpsTimestamp(utcTimeSeconds);
                    }
                } else {
                    loc = null;
                }
            }

            mCameraDevice.setParameters(mParameters);

            mCameraDevice.takePicture(mShutterCallback, mRawPictureCallback,
                    mPostViewPictureCallback, new JpegPictureCallback(loc));
            mPreviewing = false;
        }

Notice that it passes four objects as parameters, each of which wraps a callback method as defined in the Camera class: ShutterCallback, which is called immediately after the image capture is complete; RawPictureCallback, which is called as soon as uncompressed data is available; PostViewPictureCallback, as soon as a scaled postview image (i.e., the image you see after taking the picture, which you can choose to discard or save) is available, and JpegPictureCallback, when compressed JPEG image data is available. In this case, the raw and postview callbacks only contain logging data, and the Shutter callback just tells the app that it doesn’t need to auto-focus anymore.  The JpegPictureCallback, however, is important.

    private final class JpegPictureCallback implements PictureCallback {
        Location mLocation;

        public JpegPictureCallback(Location loc) {
            mLocation = loc;
        }

        public void onPictureTaken(
                final byte [] jpegData, final android.hardware.Camera camera) {
            if (mPausing) {
                return;
            }

            mJpegPictureCallbackTime = System.currentTimeMillis();
            // If postview callback has arrived, the captured image is displayed
            // in postview callback. If not, the captured image is displayed in
            // raw picture callback.
            if (mPostViewPictureCallbackTime != 0) {
                mShutterToPictureDisplayedTime =
                        mPostViewPictureCallbackTime - mShutterCallbackTime;
                mPictureDisplayedToJpegCallbackTime =
                        mJpegPictureCallbackTime - mPostViewPictureCallbackTime;
            } else {
                mShutterToPictureDisplayedTime =
                        mRawPictureCallbackTime - mShutterCallbackTime;
                mPictureDisplayedToJpegCallbackTime =
                        mJpegPictureCallbackTime - mRawPictureCallbackTime;
            }
            Log.v(TAG, "mPictureDisplayedToJpegCallbackTime = "
                    + mPictureDisplayedToJpegCallbackTime + "ms");
            mHeadUpDisplay.setEnabled(true);

            if (!mIsImageCaptureIntent) {
                // We want to show the taken picture for a while, so we wait
                // for at least 1.2 second before restarting the preview.
                long delay = 1200 - mPictureDisplayedToJpegCallbackTime;
                if (delay < 0) {
                    restartPreview();
                } else {
                    mHandler.sendEmptyMessageDelayed(RESTART_PREVIEW, delay);
                }
            }
            mImageCapture.storeImage(jpegData, camera, mLocation);
            ...
        }
    }

The Jpeg callback instructs the app to leave the postview picture on the screen for a short time, and then calls the ImageCapture.storeImage() method. This method passes all of the image data to the ImageManager class for storage, which simply writes the JPEG data straight to file. It also adds an entry for it to Android's MediaStore provider, which keeps a list of all images and other media on the device – that way it can be viewed in the Gallery right away.

Auto-focus is managed by the Camera class too. The app simply calls autoFocus() when it's ready to start focusing, and registers a callback for when focus is achieved.

    private void autoFocus() {
        // Initiate autofocus only when preview is started and snapshot is not
        // in progress.
        if (canTakePicture()) {
            mHeadUpDisplay.setEnabled(false);
            Log.v(TAG, "Start autofocus.");
            mFocusStartTime = System.currentTimeMillis();
            mFocusState = FOCUSING;
            updateFocusIndicator();
            mCameraDevice.autoFocus(mAutoFocusCallback);
        }
    }

It's also worth noting that there is a Camera.Parameters class (you probably noticed mParameters or mInitialParams in the code snippets) that bundles up all of the details, such as manual focus, zoom, antibanding, and basic photo effects.  The camera app keeps a Parameters instance locally for changes to be made easily, then passes them down to the Camera framework via Camera.setParameters().

Video capture

There's a switch on the UI that lets the user pick whether they want to capture video or still images, and it simply switches between the Camera and VideoCamera activities. So, let's look at the VideoCamera activity now.

The fundamental difference, obviously, is in what happens when you press the shutter button. This calls the startVideoRecording() method.

    private void startVideoRecording() {
        ...
        initializeRecorder();
        ...
        try {
            mMediaRecorder.start(); // Recording is now started
        } catch (RuntimeException e) {
            Log.e(TAG, "Could not start media recorder. ", e);
            releaseMediaRecorder();
            return;
        }
        ...
        keepScreenOn();
    }

    private void initializeRecorder() {
        ...
        mMediaRecorder = new MediaRecorder();

        // Unlock the camera object before passing it to media recorder.
        mCameraDevice.unlock();
        mMediaRecorder.setCamera(mCameraDevice);
        mMediaRecorder.setAudioSource(MediaRecorder.AudioSource.CAMCORDER);
        mMediaRecorder.setVideoSource(MediaRecorder.VideoSource.CAMERA);
        mMediaRecorder.setProfile(mProfile);
        mMediaRecorder.setMaxDuration(mMaxVideoDurationInMs);

        // Set output file.
        if (mStorageStatus != STORAGE_STATUS_OK) {
            mMediaRecorder.setOutputFile("/dev/null");
        } else {
            // Try Uri in the intent first. If it doesn't exist, use our own
            // instead.
            if (mVideoFileDescriptor != null) {
                mMediaRecorder.setOutputFile(mVideoFileDescriptor.getFileDescriptor());
                try {
                    mVideoFileDescriptor.close();
                } catch (IOException e) {
                    Log.e(TAG, "Fail to close fd", e);
                }
            } else {
                createVideoPath();
                mMediaRecorder.setOutputFile(mVideoFilename);
            }
        }

        mMediaRecorder.setPreviewDisplay(mSurfaceHolder.getSurface());
        ...
        try {
            mMediaRecorder.setMaxFileSize(maxFileSize);
        } catch (RuntimeException exception) {
            // We are going to ignore failure of setMaxFileSize here, as
            // a) The composer selected may simply not support it, or
            // b) The underlying media framework may not handle 64-bit range
            // on the size restriction.
        }

        // See android.hardware.Camera.Parameters.setRotation for
        // documentation.
        int rotation = 0;
        if (mOrientation != OrientationEventListener.ORIENTATION_UNKNOWN) {
            CameraInfo info = CameraHolder.instance().getCameraInfo()[mCameraId];
            if (info.facing == CameraInfo.CAMERA_FACING_FRONT) {
                rotation = (info.orientation - mOrientation + 360) % 360;
            } else {  // back-facing camera
                rotation = (info.orientation + mOrientation) % 360;
            }
        }
        mMediaRecorder.setOrientationHint(rotation);
        mOrientationHint = rotation;

        try {
            mMediaRecorder.prepare();
        } catch (IOException e) {
            Log.e(TAG, "prepare failed for " + mVideoFilename, e);
            releaseMediaRecorder();
            throw new RuntimeException(e);
        }

        mMediaRecorder.setOnErrorListener(this);
        mMediaRecorder.setOnInfoListener(this);
    }

This first calls an initializeRecorder() method, which creates a MediaRecorder object, and sets it up for camera recording. For this to happen, it needs to know what the video and audio sources are, the details for file recording such as format and encoding, and the name of the output file. Finally, its prepare() method is called, telling it to get ready to record – if this fails, the camera cannot record video.

Once the MediaRecorder is set up, its start() method is called, and recording begins. Later on, when the activity is sent to the background or the user hits the shutter button again, the stopVideoRecording() method is called, which in turn calls the MediaRecorder's own stop() method. It then goes and registers the video with the content database, and finally releases the MediaRecorder object.

That pretty much covers the guts of your basic Camera app. There's more details, of course, but we could spend a lot of time going over every single option exposed by the Camera and MediaRecorder classes. Armed with what we've already discussed and the source code for the Camera app, you should be able to roll your own with little effort. Good luck!

"We need to go deeper"

That’s kind of how I felt as a naive software intern thumbing through the vast amount of information in the Android source code. There’s a lot to take in: Android covers an enormous set of use cases, and it follows a secure and extensible model that requires function calls to dive through layer after layer of abstraction to reach the bottom. Perhaps not so complicated as some heavyweight operating systems that we know of, but the user-focused nature of Android demands that the experience be seamless, so it’s important for engineers involved in porting to a new device to understand how it all fits together.

There’s documentation, of course, but most of Google’s material is focused on the applications-level developer. Whether or not you’re interested in messing with the lower layers, it helps to have an understanding of what happens beneath the surface of your application.

In this post, we’re going to try and provide a brief overview of the Android 2.3 (aka “Gingerbread”) camera subsystem on the i.MX51, which we’re in the process of configuring for our Nitrogen boards.  If you don’t already have it, get it here.

First, an architectural diagram:


It’s color-coded by origin, by the way.  Blue is Boundary Devices (either software we’ve written or pieces of the Nitrogen board itself), orange is Freescale, yellow is Linux, green is Google, and purple is alsa.

Anyway, now you have some idea of how it’s all laid out from a bird’s eye view. Next we’ll go over the pieces.

Application layer

The heart of any camera app is the Camera class. This is an abstraction for the Camera device itself and everything that sits between it and the Android API. It provides methods to set parameters, auto-focus, and, of course, take a picture. When you initialize a Camera object, you’ll pass a SurfaceView that represents the area of the UI that will contain the camera preview image.  Then you’re ready to take pictures with the takePicture() method.  Pretty self-explanatory!

To make your app record video, you’ll need a MediaRecorder object as well. This hooks into the media framework (which we’ll discuss below) and abstracts the task of recording audio and video. It expects a Camera object and a file descriptor for the output file, as well as a number of parameters, but again, using it is very simple. When an application developer needs to use the Camera, he grabs a Camera object; when he needs to record something, he grabs a MediaRecorder object, and that’s that. Works out well for an independent app developer working on his own time.

We’ll show some examples on how to use these classes in the next post. What we’re really interested right now is what’s underneath. For brevity’s sake, we should mention that source files in the following sections (until we get to Freescale proprietary components) are listed are under frameworks/base/, unless otherwise stated.  Also, if we link to the implementation but not the header, you can assume that the header is somewhere under include/.

JNI layer

The Java Native Interface (JNI) is what allows Android classes to use the native C++ libraries from within the Dalvik virtual machine. Look in frameworks/base/core/jni/ or frameworks/base/media/jni/ and you’ll see C++ implementation files corresponding to many of the Android-specific Java classes – for example, android_hardware_camera.cpp. These contain methods for passing messages between their Java counterparts running inside a Dalvik virtual machine and a native C++ implementation of that class.

Native layer glue classes

Okay, here’s where things get really interesting. Below the JNI layer, there is a complex set of processes that does the dirty work of the Android operating system as a whole. You can actually see these processes if you open up an adb shell and run ps to find the list of running processes.

Binder interface

The way these processes communicate is through the Binder system. The binder system was designed by Google as a custom inter-process communication system for Android. With adbshell, you can see that there is a binder driver under /dev/. This driver is sort of like a mailbox that every process hooks into, and is responsible for passing messages between service providers and service clients.

You will frequently see objects in the native libraries with names like ICamera, ICameraService, IMediaRecorder, and so on (look here for the ICamera example). These are objects that implement the Binder interfaces and thus represent proxy objects that marshal data across process boundaries. Each header/implementation pair usually contains an interface class, IObject as well as a BpObject class and a BnObject class. Bp stands for binder proxy, the class that sits in the application process, and Bn stands for binder native, the class that sits in the remote process (such as the MediaServer or SurfaceFlinger) that is basically invisible to the end user. Binder proxy objects call a transact() method which passes messages to the binder native object, which handles them with an onTransact() callback. transact() never returns until onTransact() returns, so they are synchronous.

Native proxies

The Camera and MediaRecorder objects, as well as the Surface object that is held by the SurfaceHolder you may have passed to the Camera object, have native proxy objects with the same name sitting below the JNI layer; for example, the Camera object. One of these will be created for each Java object you instantiate on the top layer. They implement the native methods in the Java object, but in this case it really means that they wrap a binder proxy object and call methods on it when told to by the JNI layer, which in turn become transactions across the process boundary to the binder native object. Once the binder proxy receives a response from the other side, it passes it back up to the native proxy, which passes it back up through the JNI layer to you.

MediaServer

The MediaServer process is the heart of Android’s media framework. It wraps up everything that is necessary to make media playback and recording possible, such as codecs, file authoring, and connections to hardware abstraction layers (HALs).

Upon startup, MediaServer launches server threads for each of its major functions, including the CameraService and the MediaPlayerService. Within the media server, our binder native objects correspond to client objects, which connect to their corresponding services.

The camera service handles clients which sit on top of the hardware abstraction layer and correspond to Camera objects. Actually, they sit on top of an interface which itself abstracts the HAL (since the HAL will be different depending on the target system’s own hardware). Its primary purpose is to manage synchronization among clients – in other words, while several can be connected to the service, only one can actually use the camera at any given time. The classes for both client and server are in the CameraService header and implementation under services/camera/libcameraservice/.

We’ll mention that MediaPlayerService actually creates clients that connect to MediaRecorders, but the MediaPlayerService doesn’t actually handle any of the recording duties. That’s handled in the next layer within the MediaServer process.

Stagefright, OpenCore, and OpenMAX

Stagefright is a new addition to the Android source – although pieces of it have been in the source since Eclair, it was only fully implemented in Gingerbread. The job of Stagefright is to abstract (again with the abstraction!) the codec library. There are a number of other interesting parts and pieces that are more or less outside of our scope here, but the idea is that Stagefright is where all the encoding and decoding takes place above the platform-specific level.  For some good block diagrams, detailing the Stagefright architecture, look here and here (last one is in Chinese, so use Google Translate). Most of the Stagefright files are in media/libstagefright/ and include/media/stagefright/.

Prior to Gingerbread, the primary vehicle for encoding/decoding was PacketVideo’s OpenCORE framework. Unlike Stagefright, it was particularly well-documented and if you need to know how it works, the information isn’t nearly as difficult to find. Stagefright is basically Google’s in-house version of OpenCORE, created with help from PV. It is not well-documented at all, mainly because the average app developer doesn’t really need to know how it works. However, my understanding is that it is simpler and more extensible than OpenCORE, and with Google’s attention it will evolve quickly and efficiently.

By the way, it is possible to swap Stagefright or OpenCORE out for an alternate media framework, such as Gstreamer, which Eric has covered in previous blog posts. That’s a serious undertaking, though, and definitely out of the scope of this post!

The central class in the recording subsystem is StagefrightRecorder (header and implementation, confusingly under media/libmediaplayerservice/). A reference to a StagefrightRecorder object is bundled into initialized MediaRecorderClient objects. The job of StagefrightRecorder is to take the short list of calls passed in from upper layers and dissect them. Given an encoding format, output format, and list of parameters, it selects an encoder and a MediaWriter object, as well as MediaSources representing data sources like the camera and microphone, then manages them as the MediaRecorder object tells it to (or complains if a bad combination of codec and file format were supplied at any point in the call stack).

MediaWriter is actually an interface that we use as a simplification for a wide array of file authoring classes which implement it. These classes call on the codec library to encode media coming in from the camera and microphone and then write it to file in a particular format. There is an MPEG4Writer, an AMRWriter, an ARTPWriter, and so on with each of the implemented file formats. Notice that this is the endpoint for file authoring: each of these classes has all it needs to write a video file in the correct format and store it to the SD card.

The great thing about Stagefright (and OpenCORE, for that matter) is that it makes it easy for codecs to be implemented. The codecs used by Stagefright are expected to adhere to the OpenMAX (OMX) standard, a system for developing codecs that are interchangeable between media frameworks. So, in theory, if you have an OMX-compliant codec, you should be able to directly plug it into Stagefright and expect it to work. More information about that here.

Freescale proprietary components

Android is portable, so it doesn’t provide the actual glue to the hardware. Instead, it has a handful of interfaces, which the hardware designer can write implementations for. Notice on the diagram that none of the actual Google code interfaces directly with the kernel layer – this has to be done by Freescale-provided components. (by the way, the source for proprietary components isn’t available via AOSP’s gitweb, so follow along with your own local copy of the Freescale release).

The camera, for example, is hidden under a hardware abstraction layer, the CameraHal class (under hardware/mx5x/libcamera/). The HAL actually contains the name of the camera driver (as well as other important system files it needs to query), and calls IOCTLs on it to set it up and use its functions. It also performs other important functions, such as encoding the picture in JPEG format or converting between color schemes.

Then there’s the codec libraries. If you look on your Android device in /system/lib/ you’ll see a pile of .so files. These are shared libraries, compiled when you first built Android for your device, and are referenced by running processes such as MediaServer. You can look in /proc/maps to see which libraries a particular process is using. Among these are a handful of precompiled libraries provided by Freescale – back on your host machine, these are under device/fsl/proprietary/. You won’t be able to see the source code for what’s in these libraries, because they’re closed-source; however, you can get an idea of who is calling on who with objdump.

Ultimately, each of the OMX codecs are linked within these libraries to a VPU codec, which itself connects to libvpu.so. The codecs are hardware-accelerated, meaning that since they perform a complex job, it’s easier to offload it to the VPU to do it. The proprietary code in libvpu does exactly that. We’ll talk more about the VPU later.

Audio subsystem

Audio input is handled via an AudioRecord object that is abstracted within Stagefright as an AudioSource object. AudioRecord manages the buffers and parameters for recording on a single audio channel.

The system’s audio channels themselves are managed by AudioFlinger, which is another service exposed by the MediaServer. AudioFlinger manages and mixes all of the audio channels in the system. This is more important for other use cases, such as being able to listen for calls while streaming music, and games with complex, multi-track audio; the important part is that AudioFlinger acts as the interface between alsa and the rest of Android.

alsa, if you aren’t aware, is the Advanced Linux Sound Architecture. alsa provides the drivers and virtual devices necessary to handle quality sound on most Linux systems, and is part of the kernel. The user-space alsa library is in external/alsa-lib.

Overlays

On the other side of the diagram is the Surface. Android uses Surface objects as a wrapper for graphics buffers, and has a Surface for every UI window you see on your Android device. You can directly manipulate a Surface from a topside application by using a SurfaceHolder, exposed to the UI via a SurfaceView object. More important to this discussion is you pass a SurfaceHolder object to your Camera to tell it where to post preview frames within your app’s UI.

Below the surface, Surfaces are typically managed by SurfaceFlinger. SurfaceFlinger’s job is to organize all the Surfaces thrown at it by Android and organize them for display. So, it figures out what parts of which windows are hidden by the one on top. To SurfaceFlinger, our preview frame is just another UI element, which it happily sorts with all the other Surfaces and posts to the display hardware.

Actually, this isn’t what happens on the i.MX51. Android also provides the pipework for surfaces to be posted as a video overlay. Overlaying allows the Surfaces to be posted directly to the hardware, without having to go through SurfaceFlinger. Since preview windows are constantly being updated with complex data, using overlays makes sense – less overhead, better performance, happier user. There are three layers here: an Overlay class which abstracts the hardware from the rest of the system, an interface header (hardware/libhardware/include/hardware/overlay.h) which connects to the hardware, and then Freescale’s Overlay library in hardware/mx5x/liboverlay/. The final layer connects to the overlay device, /dev/video16, and posts YUV-formatted overlays to it directly.

Kernel components

That about does it for the Android userland. Whew.

Android runs on a Linux kernel, and follows most of the rules that apply to normal Linux systems. So to communicate with the hardware, Android processes talk to device drivers, exposed as usual in /dev/.

First, you should know about V4L2. It’s the official Linux API for developing video drivers and applications to work with Linux. If you have any recent experience developing video applications for Linux then you’ve probably worked with V4L2. It makes sense that Android would utilize it. At any rate, both the camera and overlay drivers are V4L2-compliant. The device drivers are exposed in the filesystem as /dev/video0 (for the camera) and /dev/video16 (for the overlay device).  /dev/fb0 is the actual display device driver.

The camera driver itself is in android/kernel_imx/drivers/media/video/boundary/. It’s a V4L2 module that is designed to work with the OV5642 cameras that we use with our Nitrogen boards, and makes calls on the IPU driver to empty buffers that are filled by the camera. Eric has already said a lot about it in previous posts here and here.

We’ve already mentioned the binder driver and ALSA. You should also know that there is an mxc_vpu driver which corresponds to the VPU, and is responsible for filling and emptying buffers between the codec subsystem and the VPU itself. The source is in kernel_imx/drivers/mxc/vpu. There is also an IPU driver in kernel_imx/drivers/mxc/ipu, which does essentially the same thing for the IPU.

I.MX51 components

It also helps to know a few things about the relevant i.MX51 subsystems that are involved in this process.

The Image Processing Unit is the component that connects to both the camera and display. The IPU automates the process of managing the display. It actually provides a lot of cool features such as hardware image conversion, but they aren’t really used by Freescale’s Android build.

More important to understand is the Video Processing Unit, which implements complex encoding and decoding algorithms in hardware, greatly speeding up the transcoding process. So, we say that the codecs are hardware-accelerated. According to Freescale, all of the codecs offered by their Android build (RTP and HTTP streaming excepted) are VPU-accelerated, with H.263 and 264 being available for recording and a number of others for playback. The VPU itself is capable of working with a lot more formats, but not all of them are supported under Freescale’s Android release yet.

You should know that there is documentation included with the Freescale Android release (look in android_source_folder/doc/SDK_UserGuides; unfortunately, they are not available on the web) that describes an API for the i.MX51 VPU and IPU. So, support is there should you ever find yourself needing to write new kernel code (such as a hardware-accelerated codec) to work with the i.MX51 media system.

Camera

Finally there’s the camera itself. We use an Omnivision 5642 camera with our Nitrogen boards. Again, most of what you need to know about this is covered in either the datasheet or Eric’s previous posts.

Overview: Function call lifecycle

So to recap, let’s follow a few of the important function calls through the flowchart above.

Camera.takePicture(…)

Callbacks are registered, and the takePicture() native method is called through the JNI interface. The native Camera object then starts a transaction with its corresponding CameraService::Client through the Binder interface, notifying it that a takePicture request has been made. CameraService::Client calls its takePicture() method, which in turn calls the takePicture() method of the CameraHardwareInterface.

At the hardware abstraction layer, the cameraTakePicture() method is called, which sets up a buffer and calls IOCTLs on the driver to fill it from memory held by the IPU’s DMA controller. That data is converted to JPEG data and written to memory for later retrieval.  A status message is returned to each of the upper layers, up to the native layer, at which point the topmost function returns and you have your picture.

MediaRecorder.start()

Once again, the start() native method is called through the JNI interface, and the native MediaRecorder object starts a transaction with its corresponding MediaRecorderClient class, notifying it that a start request has been made. The MediaRecorderClient calls its start() method, which in turn calls the start() method of the wrapped StagefrightRecorder class. Supposing that we have chosen to record an MPEG4 video, the startMPEG4Recording() method is called, and a new MPEG4Writer object is created with the file descriptor we previously passed to the top-level MediaRecorder object.

The MPEG4Writer is set up with some initial parameters, including an encoder of some type, and then its own start() method is called. The parameters are copied, some header information is written, and then a writer thread is started, which calls the start() methods of a number of Tracks which wrap the AudioSource and CameraSource which were passed in previously. Each has its own start() method called – in the case of the CameraSource, it calls on the startRecording() method of the Camera object which it wraps, and that call proceeds down the chain as described above. At the CameraHAL layer, buffers are set aside and are filled with preview frames. The information is made available to the writer thread as a pointer to the memory where these frames can be found.

A discussion on how writing works, in detail, is probably beyond the scope of this article, since it is very complex, but the gist of it is that data comes in through the source pipes, gets directed to the VPU for hardware-accelerated encoding, and is written to file by the MPEG4Writer or whatever other Writer class is being employed.

That about covers it!  In the next post we’ll go over the AOSP Camera app in detail, to give you an idea of how to write your own.