me

A fully open source stack for MIPI cameras

Many recent Intel laptops have replaced the standard UVC USB camera module with a raw MIPI camera-sensor connected to the IPU6 found in recent Intel laptop chips.

Both the hw interface of the ISP part of the IPU6 as well as the image processing algorithms used are considered a trade secret and so far the only Linux support for the IPU6 relies on an out of tree kernel driver with a proprietary userspace stack on top, which is currently available in rpmfusion.

Both Linaro and Red Hat have identified the missing ISP support for various ARM and X86 chips as a problem. Linaro has started a project to add a SoftwareISP component to libcamera to allow these cameras to work without needing proprietary software and Red Hat has joined Linaro in working on this.

FOSDEM talk

Bryan O'Donoghue (Linaro) and I are giving a talk about this at FOSDEM.

Fedora COPR repository

This work is at a point now where it is ready for wider testing. A Fedora COPR repository with a patched kernel and libcamera is now available for users to test, see the COPR page for install and test instructions.

This has been tested on the following devices:
  • Lenovo ThinkPad X1 yoga gen 8 (should work on any ThinkPad with ov2740 sensor)
  • Dell Latitude 9420 (ov01a1s sensor)
  • HP Spectre x360 13.5 (2023 model, hi556 sensor)

Description of the stack
  1. Kernel driver for the camera sensor, for the ov2740 used on current Lenovo designs (excluding MTL) I have landed all necessary kernel changes for this upstream.
  2. Kernel support for the CSI receiver part of the IPU6 Intel is working on upstreaming this and has recently posted v3 of their patch series for this upstream and this is under active review.
  3. A FOSS Software ISP stack inside libcamera to replace the missing IPU6 ISP (processing-system/psys) support. Work on this is under way. I've recently send out v2 of the patch-series for this.
  4. Firefox pipewire camera support and support for the camera portal to get permission to access the camera. My colleague Jan Grulich has been working on this, see Jan's blogpost. Jan's work has landed in the just released Firefox 122.

me

Fix Fedora IPU6 not working on Dell laptops with kernel 6.5

There is an issue with the rpmfusion packaged IPU6 camera stack for Fedora is not working on many Dell laptop models after upgrading the kernel to a 6.5.y kernel.

This is caused by a new mainline ov0a10 sensor driver which takes precedence over the akmod ov0a10 driver but lacks VSC integration.

This can be worked around by running the following command:
sudo rm /lib/modules/$(uname -r)/kernel/drivers/media/i2c/ov01a10.ko.xz; sudo depmod -a

After the rm + depmod run:
sudo rmmod ov01a10; sudo modprobe ov01a10

Or reboot. After this your camera will hopefully work again.

I have submitted a pull-request to disable the mainline kernel's non working ov01a10 driver, so after the next Fedora kernel update this workaround should no longer be necessary.
me

Fedora IPU6 black image issue

I have just become aware that Fedora users using the packaged IPU6 camera stack are having an issue where the output from the camera is black. We have been updating the stack and the new version of ipu6-camera-bins has hit the stable updates repo, while the matching new version of ipu6-camera-hal is currently in the updates-testing repo.

This causes the versions of ipu6-camera-bins vs ipu6-camera-hal to get out of sync (unless you have updates-testing enabled) which leads to the camera output being all black

You can fix this issue by running the following command:

sudo dnf update --enablerepo=rpmfusion-nonfree-updates-testing 'ipu6-camera-*'

Sorry about the inconvenience, we'll make sure to push both packages at the same time for the next set of updates.

I have tagged all the new ipu6-camera-hal builds to be moved to the stable update repositories, so on the next rpmfusion updates push this should be resolved.

me

Fedora IPU6 camera support now available in rpmfusion-nonfree

Installation

I am happy to announce that Intel's IPU6 camera stack has been packaged in rpmfusion and now can be installed under Fedora 37 and newer with a single `dnf install` command.

Note since this uses an out of tree kernel module build as unsigned akmod you need to disable secureboot for this to work; or alternatively sign the kmod with your own local key (instructions here).

First enable both the rpmfusion-free and rpmfusion-nonfree repositories, for instructions see https://rpmfusion.org/Configuration

The IPU6 support requires kernel >= 6.3.1 which is in updates-testing for now and v4l2loopback also needs to be updated to the latest version (in case you already have it installed):

sudo dnf update \
  --enablerepo=updates-testing \
  --enablerepo=rpmfusion-free-updates-testing \
  --enablerepo=rpmfusion-nonfree-updates-testing \
  'kernel*' '*v4l2loopback'

And now things are ready to install the IPU6 driver stack:

sudo dnf install \
  --enablerepo=updates-testing \
  --enablerepo=rpmfusion-free-updates-testing \
  --enablerepo=rpmfusion-nonfree-updates-testing \
  akmod-intel-ipu6

After this command reboot and you should be able to test your camera with https://mozilla.github.io/webrtc-landing/gum_test.html under firefox now.

This relies on Intel's partly closed-source hw-enablement for the IPU6, as such this known to not work on laptop models which are not covered by Intel's hw-enablement work. If your laptop has an option to come with Linux pre-installed and that SKU uses the IPU6 cameras then this should work. Currently known to work models are:

  • Dell Latitude 9420 (ov01a1s sensor)

  • Dell Precision 5470 (ov01a10 sensor)

  • Dell XPS 13 Plus 9320 (ov01a10 sensor)

  • Lenovo ThinkPad X1 Carbon Gen 10 (ov2740 sensor)

  • Lenovo ThinkPad X1 Nano Gen 2 (ov2740 sensor)

  • Lenovo ThinkPad X1 Yoga Gen 7 (ov2740 sensor)


If the IPU6 driver works for you on an unlisted model please drop mean email at so that the above list can be updated.

Description of the stack

The IPU6 camera stack consists of the following layers:

  1. akmod-intel-ipu6 the IPU6 kernel drivers. These are currently out of tree. Work is ongoing on getting various IO-expander, sensor drivers and the CSI2 receiver patches upstream. This is a slow process though and currently there is no clear path to getting the actual ISP part of the IPU supported upstream.

  2. ipu6-camera-bins this is a set of closed-source userspace libraries which the rest of the userspace stack builds on top of. There is a separate set of libraries for each IPU6 variant. Currently there are 2 sets, "ipu6" for Tiger Lake and "ipu6ep" for Alder Lake.

  3. ipu6-camera-hal this is a library on top of the set of libraries in ipu6-camera-bins. This needs to be built separately for the "ipu6" and "ipu6ep" library sets from ipu6-camera-bins.

  4. gstreamer1-plugins-icamerasrc a gstreamer plugin built on top of ipu6-camera-hal. This allows using the camera through gstreamer.

  5. akmod-v4l2loopback + v4l2-relayd. Most apps don't use gstreamer for camera access and even those that do don't know they need to use the icamerasrc element. v4l2-relayd will monitor a v4l2loopback /dev/video0 node and automatically start a gstreamer pipeline to send camera images into the loopback when e.g. firefox opens the /dev/video0 node to capture video.


Packaging challenges and technical details

  1. akmod-intel-ipu6: There were 2 challenges to overcome before the IPU6 kernel drivers could be packaged:

    1. The sensor drivers required patches to the main kernel package, specifically to the INT3472 driver which deals with providing GPIO, clk, regulator and LED resources to the sensor drivers. Patches have been written for both the main kernel, including some LED subsystem core additions, as well as patches to the IPU6 sensor drivers to bring them inline with mainline kernel conventions for GPIOs, clks and LEDs. All the necessary patches for this are upstream now, allowing the latest ipu6-drivers code to work with an unmodified mainline kernel.

    2. Until now the IPU6 drivers seem to have been used with a script which manually loads the modules in a specific order. Using automatic driver loading by udev exposed various probe-ordering issues. Requiring numerous patches (all upstreamed to Intel's github repo) to fix.


  2. ipu6-camera-bins: Since there were 2 sets of libraries for different IPU6 versions, these are now installed in separate /usr/lib64/ipu6[ep] directories and the headers and pkgconfig files are also installed in 2 different variants.

  3. ipu6-camera-hal: This needs to be built twice against the 2 different sets of ipu6-camera-bins libraries. Letting the user pick the right libcamhal.so build to install is not very user friendly, to avoid the user needing to manually chose:

    1. Both builds are installed in separate /usr/lib64/ipu6[ep] directories.

    2. The libcamhal.so under these directories is patched to include the directory it is installed in as RPATH, so that dynamic-linking against that libcamhal.so will automatically link against the right set of ipu6-camera-bins libraries.

    3. To make all this all work transparently the actual /usr/lib64/libcamhal.so is a symlink to /run/libcamhal.so and /run/libcamhal.so is set by udev rules to point to /usr/lib64/ipu6[ep]/libcamhal.so depending on the actual hw. The /run/libcamhal.so indirection is there so that things will also work with an immutable /usr .

    4. ipu6-camera-hal's udev rules also set a /run/v4l2-relayd config file symlink to configure the gstreamer pipeline use by v4l2-relayd to match the ipu6 vs ipu6ep capabilities.


  4. akmod-v4l2loopback + v4l2-relayd: Getting this to work with Firefox was somewhat tricky, there were 2 issues which had to be solved:

    1. Firefox does not accept the NV12 image format generated by ipu6ep pipelines. To work around this a conversion to YUV420 has been added to the v4l2-relayd pipeline feeding into v4l2loopback. This workaround can be dropped once Firefox 114, which will have NV12 support, is released.

    2. Gstreamer sends v4l2-buffers with a wrong bytesused field value into v4l2loopback causing Firefox to reject the video frames. A patch has been written and merged upstream to make v4l2loopback fix up the bytesused value, fixing this.



Many thanks to my colleague Kate Hsuan for doing most of the packaging work for this.

And also a big thank you to the rpmfusion team for maintaining the rpmfusion repo and infrastructure which allows packaging software which does not meet Fedora's strict guidelines outside of the Fedora infra.
me

Second request for Linux backlight testing for changes planned for 6.2

As mentioned in my previous blog post, I have written a new patch series for 6.2 to try to avoid having multiple entries in /sys/class/backlight for a single panel again.

This new series might cause regressions on a different set of even older laptop models then the one affected by the 6.1 backlight work. So I'm again looking for people willing to run a few quick tests.

To see if your laptop is possibly affected by the 6.2 change please run:

  • ls /sys/class/backlight


if the output of this command contains both:

  1. A GPU native backlight device, with intel/nv/nvidia/amd/radeon/psb/oaktrail in the name; and

  2. A vendor backlight device, with either a model-series: eeepc, ideapad, thinkpad, etc; or a vendor-name: acer, asus, dell, toshiba, sony, etc. in the name


then 6.2 will cause a behavior change on your device, it will hide the vendor backlight device in preference of the native backlight device.

If your laptop shows only a native or only a vendor backlight device (possibly in combination with another type of backlight device such as acpi_video#), then your laptop will not be affected by the planned changes for 6.2.

Note that I expect only very old models to be affected, e.g. the Sony Vaio PCG-FRV3
from 2003 is known to be affected.

If your laptop has both native + vendor backlight devices, then please do 2 things:


  1. Please run the following commands:

    1. ls /sys/class/backlight > ls-backlight.txt

    2. sudo dmesg > dmesg.txt

    3. sudo dmidecode > dmidecode.txt

    4. sudo acpidump -o acpidump.txt


  2. Please test if the native backlight interface works, the example below assumes the native backlight is called "intel_backlight":

    1. cd /sys/class/backlight/intel_backlight

    2. cat max_brightness

    3. <the "cat max_brightness" will show the maximum brightness value supported>

    4. echo $max_brightness_value > brightness

    5. echo $half-of-max_brightness_value > brightness


Where if for example cat max_brightness returns 255 then $max_brightness_value
is 255 and $half-of-max_brightness_value is 127. And then check if the brightness of the backlight actually changes when you do this ?


After generating the 4 .txt files and running the native backlight tests, please send me an email about this at hdegoede@redhat.com with the results of the native backlight tests (panel brightness changes when echo-ing values to brightness or not?) and with the 4 generated .txt files attached.

If the native backlight interface works then things should keep working fine with 6.2 and typically you will get more fine-grained brightness control as an added bonus. Please send me an email with the test results even if the native backlight interface works.
me

Results of requested backlight testing for 6.1

I have received quite a few test reports in response to my previous blog post. Many thanks to everyone who has run the tests and send me their results!

These tests show that as a result of the current 6.1 changes quite a few laptop models will end up with an empty "/sys/class/backlight", breaking users ability to control their laptop panel's brightness.

I have submitted a patch-set for 6.1 upstream to fix this.

More detailed summary/analysis of the received test reports:

  • Known Windows laptop models affected by this:

    • Acer Aspire 1640

    • HP Compaq nc6120

    • IBM ThinkPad X40

    • System76 Starling Star1


  • Known MacBook models affected by this:

    • Apple MacBook 2.1

    • Apple MacBook 4.1

    • Apple MacBook Pro 7.1


  • 30 unaffected models

  • The Dell Inspiron N4010 has acpi_video support and acpi_osi_is_win8() returns false, so acpi_video_get_backlight_type() returns acpi_video, but acpi_video fails to register a backlight device due to a _BCM eval error. This model already has a DMI quirk for this, so it is unaffected.

  • The following laptop models use vendor backlight control method, while also having a native backlight entry under /sys/class/backlight:

    • Asus EeePC 901 (native backlight confirmed to work better then vendor)

    • Dell Latitude D610 (native backlight confirmed to also work)

    • Sony Vaio PCG-FRV3 (native backlight control does not work, BIOS from 2003!)



The fixes for 6.1 restore the behavior where userspace can see multiple entries under "/sys/class/backlight" for a single panel and the kernel leaves figuring out which one actually works up to userspace. This is undesirable and having more then 1 backlight device for a single panel also blocks the new backlight userspace API work which I have planned.

This first round of testing has shown that native works well even on systems so old that they don't have acpi_video backlight control support.

So I have prepared a patch series to try again with 6.2 by making native be preferred over vendor, which should avoid the problems seen with the 6.1 changes before the fixes.
me

Kernel 6.1-rc# might break backlight control on old/weird laptops, please test

I have landed a large(ish) refactor of the ACPI/x86 backlight detection code in the kernel for 6.1. I have been very careful to try and not break things but there is a special group of laptops where the ability to control the backlight brightness may disappear because of this.

The most likely laptops to be hit by this are laptops which are either pretty old and or which are weird in some other way (e.g. flashed with coreboot, did not ship with Windows as factory os, ...). Note Chromebooks are affected by this too, but that special category has already been fixed.

You can check if your laptop is affected by this by running "ls /sys/class/backlight" if this shows only 1 entry and that entry is named "intel_backlight", "nouveau_bl", "amdgpu_bl0" or "radeon_bl0" then your laptop might be affected.

Note this is quite normal on modern(ish) laptops, a second check is to boot with "acpi_backlight=video" added to the kernel commandline and then run "ls /sys/class/backlight" again, if you now additionally also have an "acpi_video0" entry then your laptop should work fine with 6.1, if you don't have an "acpi_video0" entry please first do "cat /proc/cmdline" and check that "acpi_backlight=video" is present there.

If you have e.g. only the "intel_backlight" entry and adding "acpi_backlight=video" does not cause an "acpi_video0" entry to appear then 6.1 will likely break backlight control!

If you have a laptop which is likely affected by this then please run the following commands:

  • ls /sys/class/backlight > ls-backlight.txt

  • sudo dmesg > dmesg.txt

  • sudo dmidecode > dmidecode.txt

  • sudo acpidump -o acpidump.txt

And send me an email about this at hdegoede@redhat.com with the 4 generated .txt files attached. If possible please also give an actual 6.1 kernel a try and see if that indeed breaks things. E.g. for Fedora you can find 6.1 kernel builds here and see here for some install instructions for these Fedora kernel builds.
me

Announcing the Fedora BIOS boot SIG

Now that FESCo has decided that Fedora will keep supporting BIOS booting, the people working on Fedora's bootloader stack will need help from the Fedora community to keep Fedora booting on systems which require Legacy BIOS to boot.

To help with this the Fedora BIOS boot SIG (special interest group) has been formed. The main goal of this SIG are to help the Fedora bootloader people by:

  1. Doing regular testing of nightly Fedora N + 1 composes on hardware
    which only supports BIOS booting

  2. Triaging and/or fixing BIOS boot related bugs.


A biosboot-sig@lists.fedoraproject.org mailinglist and bugzilla account has been created, which will be used to discuss testing result and as assignee / Cc for bootloader bugzillas which are related to BIOS booting.

If you are interested in helping with Fedora BIOS boot support please:

  1. Subscribe to the email-list

  2. Add yourself to the Members section of the SIG's wiki page

me

PSA: The 5.17 kernel will require some initrd generator changes for kms drivers

Starting with kernel 5.17 the kernel supports the builtin privacy screens built into the LCD panel of some new laptop models.

This means that the drm drivers will now return -EPROBE_DEFER from their probe() method on models with a builtin privacy screen when the privacy screen provider driver has not been loaded yet.

To avoid any regressions distors should modify their initrd generation tools to include privacy screen provider drivers in the initrd (at least on systems with a privacy screen), before 5.17 kernels start showing up in their repos.

If this change is not made, then users using a graphical bootsplash (plymouth) will get an extra boot-delay of up to 8 seconds (DeviceTimeout in plymouthd.defaults) before plymouth will show and when using disk-encryption where the LUKS password is requested from the initrd, the system will fallback to text-mode after these 8 seconds.

I've written a patch with the necessary changes for dracut, which might be useful as an example for how to deal with this in other initrd generators, see: https://github.com/dracutdevs/dracut/pull/1666

I've also filed bugs for tracking this for Fedora, openSUSE, Arch, Debian and Ubuntu.
me

Unlocking the bootloader and disabling dm-verity on Android-X86 devices

For the hw-enablement for Bay- and Cherry-Trail devices which I do as a side project, sometimes it is useful to play with the Android which comes pre-installed on some of these devices.

Sometimes the Android-X86 boot-loader (kerneflinger) is locked and the standard "Developer-Options" -> "Enable OEM Unlock" -> "Run 'fastboot oem unlock'" sequence does not work (e.g. I got the unlock yes/no dialog, and could move between yes and no, but I could not actually confirm the choice).

Luckily there is an alternative, kernelflinger checks a "OEMLock" EFI variable to see if the device is locked or not. Like with some of my previous adventures changing hidden BIOS settings, this EFI variable is hidden from the OS as soon as the OS calls ExitBootServices, but we can use the same modified grub to change this EFI variable. After booting from an USB stick with the relevant grub binary installed as "EFI/BOOT/BOOTX64.EFI" or "BOOTIA32.EFI", entering the
following command on the grub cmdline will unlock the bootloader:

setup_var_cv OEMLock 0 1 1

Disabling dm-verity support is pretty easy on these devices because they can just boot a regular Linux distro from an USB drive. Note booting a regular Linux distro may cause the Android "system" partition to get auto-mounted after which dm-verity checks will fail! Once we have a regular Linux distro running step 1 is to find out which partition is the android_boot partition to do this as root run:

blkid /dev/mmcblk?p#

Replacing the ? for the mmcblk number for the internal eMMC and then for # is 1 to n, until one of the partitions is reported as having 'PARTLABEL="android_boot"', usually "mmcblk?p3" is the one you want, so you could try that first.

Now make an image of the partition by running e.g.:

dd if=/dev/mmcblk1p3" of=android_boot.img

And then copy the "android_boot.img" file to another computer. On this computer extract the file and then the initrd like this:

abootimg -x android_boot.img
mkdir initrd
cd initrd
zcat ../initrd.img | cpio -i


Now edit the fstab file and remove "verify" from the line for the system partition. after this update android_boot.img like this:

find . | cpio -o -H newc -R 0.0 | gzip -9 > ../initrd.img
cd ..
abootimg -u android_boot.img -r initrd.img


The easiest way to test the new image is using fastboot, boot the tablet into Android and connect it to the PC, then run:

adb reboot bootloader
fastboot boot android_boot.img


And then from an "adb shell" do "cat /fstab" verify that the "verify" option is gone now. After this you can (optionally) dd the new android_boot.img back to the android_boot partition to make the change permanent.

Note if Android is not booting you can force the bootloader to enter fastboot mode on the next boot by downloading this file and then under regular Linux running the following command as root:

cat LoaderEntryOneShot > /sys/firmware/efi/efivars/LoaderEntryOneShot-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f