October 4, 2013

i.MX6 Video acceleration on Ubuntu Raring (and Debian)

When we recently posted Linaro Raring images for i.MX6, we were surprised at the number responses regarding hardware acceleration, and especially video acceleration.

For a long time now, we’ve been saying that it could be done, but leaving it as an exercise for the reader. In other words, go read the manuals and figure it out. Based on the response, it’s probably time to show some details.

It’s not very pretty, but the following will show you how it can be done. I recommend a full cup of coffee and a SATA drive, because almost all of this will occur on the device, and native compilation is very slow on SD card.

Also note that these notes apply and have been tested on Debian Wheezy, except that the names of some of the prerequisites are different. Also note that I’m starting with raring-nano to make sure I capture all of them.

First, the pre-requisites

First, make sure your system is up-to-date:
root@linaro-nano:~# apt-get update
Then you’ll need a compiler:
root@linaro-nano:~# apt-get install build-essential
And the ability to compile gstreamer plugins:
root@linaro-nano:~# apt-get install libgstreamer-plugins-base0.10-dev
root@linaro-nano:~# apt-get build-dep gst-plugins-base0.10
During the steps above, you’re going to get messages like this:
/sbin/ldconfig.real: /usr/lib/libOpenVG.so is not a symbolic link
In English, this is a warning that the Vivante libraries we installed don’t follow proper Linux shared-library versioning standards (where .so files are supposed to be symlinks to .so.version).

You can safely ignore them.

Then get the packages

The packages required for building the gstreamer plugins are as follows:
  • A set of kernel headers
  • firmware-imx-3.5.7-1.0.0
  • imx-lib-3.5.7-1.0.0
  • fsl-alsa-plugins-3.5.7-1.0.0
  • libfslcodec-3.5.7-1.0.0
  • libfslparser-3.5.7-1.0.0
  • libfslvpuwrap-3.5.7-1.0.0
  • gst-fsl-plugins-3.5.7-1.0.0

Kernel headers

Because some of these packages use hardware-specific APIs, they expect some i.MX6-specific header files to be present in /usr/include on the build machine.

There are two simple ways to accomplish this:
  1. If you’ve already built the kernel, you can use the headers_install target in the kernel build tree to create a set of headers and simply copy them into /usr/include, or
  2. You can use the Debian approach of make-kpkg to build a linux-headers-versionnum.deb package and use dpkg -i to install it.
To make this easy, we’ve uploaded a package to our cloud storage site with a recent version of the tar-ball version that you can grab and extract as shown below. We had some difficulty with make-kpkg generating an armhf package with a cross-compiler.
root@linaro-nano:~# wget http://commondatastorage.googleapis.com/boundarydevices.com/linux-headers-3.0.35-02829-gac24896_4.1.0.tar.gz
root@linaro-nano:~# tar zxvf linux-headers-3.0.35-02829-gac24896_4.1.0.tar.gz -C /usr

Getting the other packages from Freescale – the LTIB way

There are two routes to the other packages, which are distributed by Freescale. The first is to download two packages from the Freescale 3.0.35_4.1.0 release page:

Inside the L3.0.35_4.1.0_ER_SOURCE_BSP package, you’ll find tar-balls for both the imx-lib-3.0.35-4.1.0 and firmware-imx-3.0.35-4.1.0 packages, and you’ll find tar-balls of the other packages inside L3.0.35_4.1.0_MM_CODECS package in the ltib codecs directory. Note that although there are .deb packages inside the codec package, we weren’t successful in compiling from them.

Getting the other packages from Freescale – the Yocto way

Another way to get the codec and BSP packages is to download them from the Freescale Yocto repository at http://www.freescale.com/lgfiles/NMG/MAD/YOCTO/. This location allows for automated download, but because some of the sources require acceptance of a EULA, they’re packaged as self-extracting shell scripts (.bin files).

Getting the packages is very easy using this route:
# first, the open-source codec package:
root@linaro-nano:~# wget http://www.freescale.com/lgfiles/NMG/MAD/YOCTO/gst-fsl-plugins-3.5.7-1.0.0.tar.gz
# then the restricted files
root@linaro-nano:~# for pkg in firmware-imx 
                             libfslvpuwrap ; do 
                         wget http://www.freescale.com/lgfiles/NMG/MAD/YOCTO/${pkg}-3.5.7-1.0.0.bin ; 
But extracting is a bit more complicated, requiring you to accept the EULA for each:
root@linaro-nano:~# tar zxf gst-fsl-plugins-3.5.7-1.0.0.tar.gz
root@linaro-nano:~# for pkg in firmware-imx 
                             libfslvpuwrap ; do 
                         sh ${pkg}-3.5.7-1.0.0.bin ; 

Building the packages

Building (or installing) the packages is relatively straightforward once you know the flags required for each.

The firmware-imx package simply requires copying files to the /lib/firmware directory:
root@linaro-nano:~# cp -ravf firmware-imx-3.5.7-1.0.0/firmware/* /lib/firmware/
The imx-lib package seems to have failures building the rng library, so we’ve skipped it. It also requires the PLATFORM=IMX6Q flag:
root@linaro-nano:~# cd imx-lib-3.5.7-1.0.0/
root@linaro-nano:~/imx-lib-3.5.7-1.0.0# rm -rf rng
root@linaro-nano:~/imx-lib-3.5.7-1.0.0# make PLATFORM=IMX6Q all && make PLATFORM=IMX6Q install
The fsl-alsa-plugins package requires some autotools-fu before compilation:
root@linaro-nano:~# cd fsl-alsa-plugins-3.5.7-1.0.0/
root@linaro-nano:~/fsl-alsa-plugins-3.5.7-1.0.0# aclocal && autoconf && automake --add-missing
root@linaro-nano:~/fsl-alsa-plugins-3.5.7-1.0.0# ./configure --prefix=/usr
root@linaro-nano:~/fsl-alsa-plugins-3.5.7-1.0.0# make all && make DESTDIR=/ install
The libfslparser, libfslvpuwrap, and gst-fsl-plugins packages each have an autogen.sh script to take care of the autotools stuff:
root@linaro-nano:~# cd libfslcodec-3.5.7-1.0.0
root@linaro-nano:~/libfslcodec-3.5.7-1.0.0# ./autogen.sh --prefix=/usr && make all && make DESTDIR=/ install
root@linaro-nano:~/libfslcodec-3.5.7-1.0.0# cd ../libfslparser-3.5.7-1.0.0
root@linaro-nano:~/libfslparser-3.5.7-1.0.0# ./autogen.sh --prefix=/usr && make all && make DESTDIR=/ install
root@linaro-nano:~/libfslparser-3.5.7-1.0.0# cd ../libfslvpuwrap-3.5.7-1.0.0
root@linaro-nano:~/libfslvpuwrap-3.5.7-1.0.0# ./autogen.sh --prefix=/usr && make all && make DESTDIR=/ install
root@linaro-nano:~/libfslvpuwrap-3.5.7-1.0.0# cd ../gst-fsl-plugins-3.5.7-1.0.0
root@linaro-nano:~/gst-fsl-plugins-3.5.7-1.0.0# ./autogen.sh PLATFORM=MX6 --prefix=/usr
root@linaro-nano:~/gst-fsl-plugins-3.5.7-1.0.0# make all && make DESTDIR=/ install

Testing things out

To test things out, we downloaded the Sintel 1080P video file from the Blender Foundation’s Sintel web-site and played the video using gplay. Note that before doing so, we also installed the base, good, bad, and ugly gstreamer plugins, and that we flushed the disk cache prior to starting playback:
root@linaro-nano:~# for t in base good bad ugly ; do 
         apt-get install gstreamer0.10-plugins-${t} ; 
root@linaro-nano:~# echo 3 > /proc/sys/vm/drop_caches
root@linaro-nano:~# gplay Sintel.2010.1080p.mkv
We also tested out camera capture for both our OV5642 5MP parallel camera and OV5640 5MP MIPI camera using the mfw_v4lsrc element:
root@linaro-nano:~# modprobe mxc_v4l2_capture
root@linaro-nano:~# gst-launch-0.10 mfw_v4lsrc capture-mode=4 ! mfw_v4lsink

Binary distribution

I’m sure a lot of you patient enough to get this far saying “I just want it to work!”.

I hear you, and I’m getting there.

If you look closely above, you’ll see that everything except firmware-imx uses make install to install things into the root filesystem, and that most of the packages specify the DESTDIR variable to say where the output should go. The one exception is imx-lib, which wants the PLATFORM=IMX6Q flag and a destination directory of DEST_DIR.

We can use this to create an overlay of the binary bits in a temporary directory:
root@linaro-nano:~# mkdir -p /root/overlay/lib/firmware/
root@linaro-nano:~# cp -ravf firmware-imx-3.5.7-1.0.0/firmware/* 
root@linaro-nano:~# for pkg in fsl-alsa-plugins 
      libfslvpuwrap ; do 
      cd ${pkg}-3.5.7-1.0.0 ; 
      make DESTDIR=/root/overlay/ install ; 
      cd ../ ; 
root@linaro-nano:~# cd imx-lib-3.5.7-1.0.0
root@linaro-nano:~/imx-lib-3.5.7-1.0.0# make DEST_DIR=/root/overlay PLATFORM=IMX6Q install
And create a tar-ball of the result like so:
root@linaro-nano:~# cd overlay
root@linaro-nano:~/overlay# tar czvf ../imx6-gstreamer-plugins-raring-20131003.tar.gz *
We’ve uploaded the result of this, and you can grab it at: If you extract this on top of one of the previously uploaded images, you’ll need to install gstreamer and the gst-plugins- packages as described above.

Recap and commentary

To recap, this post described the steps needed to get, build, and install most of the gstreamer plugins for i.MX6.

Note that there is one component not covered: the OpenGL image sink. That will have to wait for another day.

We also haven’t yet tested all of the functionality to look for gaps between this and other platforms. It’s entirely possible that there’s some breakage because we haven’t included patches like this one to the core GStreamer build.

Comments 33

  1. merics

    I noticed that L3.0.35_4.1.0_MM_CODECS sources includes gst-fsl-plugins, libfslcodec and libfslparser version 3.0.7-2, not 3.5.7-1.0.0. which version is correct to use with 3.0.35-4.1.0 kernel release? I’m kinda confused.

    1. Post

      Hi Meric,

      Unfortunately, I’m probably going to make things worse…

      • The 3.5.7-1.0.0 release actually contains version 3.0.8 of the package, and
      • There’s a new 3.10.9 release (3.10.9 refers to the kernel version) which is in alpha that reportedly is backward-compatible and will be used as the baseline going forward
  2. danielo

    The MIPI Camera only supports auto exposure at the beginning of the gstreamer pipeline. After the brightness is constant.

  3. Dataedge

    Hi all,

    I am trying to configure Xorg in portrait mode, because our product will have it’s display rotated counter clockwise CCW. My system ia a BD-SL-i.MX6 sabrelite used as development board, connected with NHD-5.0-800480TF-ATXI-CTP newhaven display :


    Like pointed out in that freescale thread: https://community.freescale.com/thread/310542
    I can confirm that vivante driver doesn’t seem to support any display rotation, and I’m forced to use the generic Xorg fbdev video driver.

    Has anyone ever faced this issue ?

  4. johncst

    I downloaded and installed the kernel headers as described above, but they don’t seem to be a complete set of kernel headers. For example, this fails with a “file not found”:

    #include <linux/module.h>

    Do I need to build the kernel from source to get this file, or??? Thanks…

    1. Post

      Hi John,

      You shouldn’t be including <linux/module.h> from a userspace program.

      Are you trying to build a driver? If so, you’ll need the full kernel sources.

    1. Post
    1. Post

      Note that you may consider cross-compiling. It’s much faster.

      Recent Ubuntu releases contain a nice package gcc-arm-linux-gnueabihf that makes this painless.

    1. Post

      Thanks Jasbir,

      No dice… glxgears seems to run with software rendering using a kernel that has acceleration in a Yocto image.

      I added some notes in a comment on your blog post.

  5. Bai Shi

    Hi Eric,

    Do you happen to have an idea where I can get the tvsrc source for gstreamer? One of our adv7180 board only works with tvsrc but not with mfw_v4lsrc. When I gst-launch mfw_v4lsrc the image seems having syncing problem and the frames will span to two buffers and either buffer will contain two different frames which make the image totally unusable.

    I straced tvsrc call and followed exactly the same sequence of VIDIOC ioctl calls. However I noticed that my result of v4l2-ctl –all shows different width and hight and bytes per field after call. I believe there is something wrong with my struct v4l2_format data, how I couldn’t pinpoint what’s wrong. Do you have any clue about it?

  6. Benjamin Federau


    I have followed your instructions to get the Freescale Libs and GStreamer plugins compiled on Ubuntu 12.04/armhf. But if I execute “gst-inspect | grep imx” I only get these elements

    aiur.imx: aiurdemux: aiur universal demuxer
    aiur.imx: webm: webm
    isink.imx: mfw_isink: IPU-based video sink
    ipucsc.imx: mfw_ipucsc: IPU-based video converter
    v4lsink.imx: mfw_v4lsink: v4l2 video sink
    vpu.imx: vpuenc: VPU-based video encoder
    vpu.imx: vpudec: VPU-based video decoder
    v4lsrc.imx: mfw_v4lsrc: v4l2 based camera src

    The beep.imx elements are completely missing. With GST_DEBUG I see that GStreamer blacklists the beep plugin, but I don’t why. There are no more error messages …
    Does the beep plugin run on Ubuntu raring for you?

    1. Post

      Hi Benjamin,

      Yes. I’m seeing beep.imx.

      root@linaro-alip:~# gst-inspect-0.10  | grep -i imx
      isink.imx:  mfw_isink: IPU-based video sink
      v4lsrc.imx:  mfw_v4lsrc: v4l2 based camera src
      v4lsink.imx:  mfw_v4lsink: v4l2 video sink
      ipucsc.imx:  mfw_ipucsc: IPU-based video converter
      beep.imx:  beepdec.aac: AAC LC decoder
      beep.imx:  beepdec.mp3: MP3 decoder
      beep.imx:  beepdec.wma: WMA decoder
      beep.imx:  beepdec.ac3: AC3 decoder
      beep.imx:  beepdec.vorbis: Vorbis decoder
      beep.imx:  beepdec: beep audio decoder
      beep.imx: 3ca: ac3
      beep.imx: ac3: ac3
      vpu.imx:  vpuenc: VPU-based video encoder
      vpu.imx:  vpudec: VPU-based video decoder
      aiur.imx:  aiurdemux: aiur universal demuxer
      aiur.imx: webm: webm
    1. Post
  7. lisarden

    I did everything step by step to the “Testing things out” and tried to test the OV5640 camera immediately and got this:
    root@linaro-nano:/mnt/Projects/gst-fsl-plugins-3.5.7-1.0.0# lsmod
    Module Size Used by
    mxc_v4l2_capture 22642 1
    ipu_fg_overlay_sdc 4868 1 mxc_v4l2_capture
    ipu_csi_enc 2937 1 mxc_v4l2_capture
    ipu_prp_enc 4685 1 mxc_v4l2_capture
    ipu_still 1703 1 mxc_v4l2_capture
    ipu_bg_overlay_sdc 3925 1 mxc_v4l2_capture
    ov5640_camera_mipi 52023 0

    root@linaro-nano:~# gst-launch-0.10 mfw_v4lsrc capture-mode=4 ! mfw_v4lsink
    MFW_GST_V4LSRC_PLUGIN 3.0.8 build on Apr 8 2014 06:52:28.
    MFW_GST_V4LSINK_PLUGIN 3.0.8 build on Apr 8 2014 06:52:01.
    Setting pipeline to PAUSED …
    Pipeline is live and does not need PREROLL …
    Setting pipeline to PLAYING …
    New clock: GstSystemClock
    full screen size:1920×1080
    [V4L Update Display]: left=0, top=0, width=1920, height=1080
    set v4l display crop sucessfully
    set v4l rotate sucessfully
    full screen size:1920×1080
    >>V4L_SINK: Actually buffer status:
    hardware buffer : 12
    software buffer : 0
    mxc_sdc_fb mxc_sdc_fb.0: Unable to allocate framebuffer memory
    mxc_v4l2_output mxc_v4l2_output.0: ERR:config_disp_output fb_set_var ret:-12
    mxc_v4l2_output mxc_v4l2_output.0: Config display output failed

    Can you help to resolve this issue?

    1. lisarden

      I found what was wrong at my side, now the camera works. Just a few annoying issues with it:
      1) weird auto-focus behavior. The auto-focus works only on the startup and not during the capturing. For example when the camera starts it tunes the auto-focus to some state and when I move it around the focus stays where it was and a picture is fuzzy.

      2) Next annoying issue is that with the command “gst-launch-0.10 mfw_v4lsrc capture-mode=4 ! mfw_v4lsink” the captured image is blended with a picture on the screen. Is there any way to show the captured image exclusively?

      3) And finally with capture-mode=5 the framerate is only 15fps, however I expected it was possible to achieve 30.

  8. lisarden

    Hi Eric!

    please recommend how to fix 6x_bootscript to enable 1080p resolution on HDMI. I tried to set 1920×1080 instead of 1280×720 but HDMI output becomes blank (even a display does not want to switch to HDMI input). Here is my cmdline:
    enable_wait_mode=off video=mxcfb0:dev=hdmi,1920x1080MR@60,if=RGB16 fbmem=48M console=ttymxc1,115200 vmalloc=400M consoleblank=0 mxc_hdmi.only_cea=1

    1. Post

      The problem in your video= clause is the if=RGB16. This clause refers to the interface between the IPU and HDMI transmitter, and should always be if=RGB24 for HDMI.

      Note that this doesn’t affect the in-memory framebuffer format. The bpp clause is used for that.

      1. lisarden

        here is a new cmdline:
        video=mxcfb0:dev=hdmi,1920x1080M@60,if=RGB24 fbmem=32M console=ttymxc1,115200 vmalloc=400M consoleblank=0 mxc_hdmi.only_cea=1

        still no HDMI output. I have compiled the kernel myself using your instructions
        I have Sabre Lite rev-D 5-9-12

        1. Post

          Hi Lisarden,

          This looks good, though I generally use 1920x1080MR@60 which uses reduced margins and a slightly slower video clock.

          Can you check to see what modes your monitor supports and what mode was selected through /sys?

          ~/$ cat /sys/class/graphics/fb0/modes
          ~/$ cat /sys/class/graphics/fb0/mode

          > w
          The modes element will give you the modes reported by your monitor and the mode will tell you what was selected.

          Also, do you see penguins during the boot process?

          1. lisarden

            rebooted with this cmdline:
            enable_wait_mode=off video=mxcfb0:dev=hdmi,1920x1080MR@60,if=RGB24 fbmem=32M console=ttymxc1,115200 vmalloc=400M consoleblank=0 mxc_hdmi.only_cea=1

            root@linaro-nano:~# cat /sys/class/graphics/fb0/modes
            root@linaro-nano:~# cat /sys/class/graphics/fb0/mode
            root@linaro-nano:~# w
            16:10:44 up 1 min, 2 users, load average: 3.57, 1.36, 0.50
            root ttymxc1 16:09 0.00s 0.11s 0.03s w
            root tty1 16:09 1:35 0.04s 0.02s -bash

            No penguins at all. What penguins should I see if my display does not want to choose HDMI input?

  9. lisarden


    I’ve noticed that when HDMI fails there are lines “mxc_sdc_fb mxc_sdc_fb.0: timeout when waiting for flip irq” in the dmesg. Please have a look:

    camera ov5640_mipi is not found
    mxc_sdc_fb mxc_sdc_fb.0: timeout when waiting for flip irq
    mxc_sdc_fb mxc_sdc_fb.0: timeout when waiting for flip irq
    eth0: Freescale FEC PHY driver [Micrel KSZ9021 Gigabit PHY] (mii_bus:phy_addr=1:06, irq=284)
    ADDRCONF(NETDEV_UP): eth0: link is not ready
    mxc_sdc_fb mxc_sdc_fb.0: timeout when waiting for flip irq
    EXT2-fs (mmcblk0p1): warning: mounting unchecked fs, running e2fsck is recommended
    PHY: 1:06 – Link is Up – 1000/Full
    ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    init: failsafe main process (318) killed by TERM signal
    init: plymouth-stop pre-start process (447) terminated with status 1
    init: tty1 main process (448) killed by TERM signal
    eth0: no IPv6 routers present
    imx-ipuv3 imx-ipuv3.0: IPU Warning – IPU_INT_STAT_5 = 0x10800000
    imx-ipuv3 imx-ipuv3.0: IPU Warning – IPU_INT_STAT_10 = 0x00180000
    ipu_init_sync_panel: disp=1, pixel_clk=148500000 148500000
    mxc_sdc_fb mxc_sdc_fb.0: timeout when waiting for flip irq
    mxc_sdc_fb mxc_sdc_fb.0: timeout when waiting for flip irq

    1. lisarden

      ok, so I tried to boot in 720p mode and then issue the following command:
      echo S:1920x1080p-60 > /sys/class/graphics/fb0/mode
      the system booted fine in 720p mode and after the previous command the display went blank and here is the dmesg output

      ipu_init_sync_panel: disp=1, pixel_clk=148500000 148500000
      imx-ipuv3 imx-ipuv3.0: IPU Warning – IPU_INT_STAT_10 = 0x00080000
      imx-ipuv3 imx-ipuv3.0: IPU Warning – IPU_INT_STAT_5 = 0x10000000
      imx-ipuv3 imx-ipuv3.0: IPU Warning – IPU_INT_STAT_5 = 0x00800000
      imx-ipuv3 imx-ipuv3.0: IPU Warning – IPU_INT_STAT_10 = 0x00100000
      mxc_sdc_fb mxc_sdc_fb.0: timeout when waiting for flip irq
      eth0: no IPv6 routers present
      mxc_sdc_fb mxc_sdc_fb.0: timeout when waiting for flip irq
      mxc_sdc_fb mxc_sdc_fb.0: timeout when waiting for flip irq

        1. Post
  10. Sridhar

    Instead of using software encode/decode in gstreamer, I would like to use Hardware-accelerated video Encoder/decoder like H.264 in SabreLite vivante GPU. Please help me how to get started Video encode using GPU.
    I have trying to install gstreamer-imx plugin (refer below link)


    Is this a correct way to do ?

  11. shondll

    Hello boundary-devices-fanatics,

    This is how I managed to setup gstreamer plugins on a Debian distro running on a Nitrogen6x board. It is a rework of this instructable mainly, just updated with tips from a recent Yocto build (I took versions from there). It goes like…

    1. Prerequisites:
    apt-get install build-essential libtool-bin
    apt-get install gstreamer1.0-plugins-good gstreamer1.0-plugins-bad
    apt-get install libgstreamer-plugins-base1.0-dev
    apt-get build-dep libgstreamer-plugins-base1.0

    2. Download stuff:
    git clone https://github.com/Freescale/gstreamer-imx.git
    wget http://www.freescale.com/lgfiles/NMG/MAD/YOCTO/firmware-imx-3.14.28-1.0.0.bin
    wget http://www.freescale.com/lgfiles/NMG/MAD/YOCTO/imx-lib-3.14.28-1.0.0.tar.gz
    wget http://www.freescale.com/lgfiles/NMG/MAD/YOCTO/imx-vpu-5.4.28.bin
    wget http://www.freescale.com/lgfiles/NMG/MAD/YOCTO/libfslvpuwrap-1.0.58.bin

    3. Extracting packages:
    chmod +x *.bin
    – Bin files are executable self-extracting archives, you just have to accept EULA. For the rest of the files, just untar.

    4. Get the kernel headers:
    – From the compiled kernel tree get the headers only with
    make headers_install
    – You will get header files in usr/include. Copy the files in /usr/include of the rootfs of the board
    cp -r usr/include/* /work/nitrogen6X/fs/debian/usr/include/

    5. Building it:
    – firmware-imx
    sudo cp -ravf firmware-imx-3.14.28-1.0.0/firmware/* /lib/firmware/

    – imx-lib-3.14.28-1.0.0
    make PLATFORM=IMX6Q all
    sudo make install

    – imx-vpu-5.4.28
    make PLATFORM=IMX6Q all
    sudo make install

    – libfslvpuwrap-1.0.58
    ./autogen.sh –prefix=/usr
    sudo make install

    – gstreamer-imx
    ./waf configure
    sudo ./waf install
    sudo ln -s /usr/local/lib/gstreamer-1.0/libgstimx* /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/

    *NOTE: maybe there is a more elegant way to pit .so files in place, bu this is quich enough

    You can now start gst-inspect-1.0 to find the imx related plugins are in place.

Leave a Reply