Hans' hacking log

The firmware which drm/kms drivers need is becoming bigger and bigger and there is a push to move to generating a generic initramfs on distro's builders and signing the initramfs with the distro's keys for security reasons. When targetting desktops/laptops (as opposed to VMs) this means including firmware for all possible GPUs which leads to a very big initramfs.

This has made me think about dropping the GPU drivers from the initramfs  and instead make plymouth work well/better with simpledrm (on top of efifb). A while ago I discussed making this change for Fedora with the Red Hat graphics team spoiler: For now nothing is going to change.

Let me repeat that: For now there are no plans to implement this idea so if you believe you would be impacted by such a change: Nothing is going to change.

Still this is something worthwhile to explore further.

Advantages:

1. Smaller initramfs size:

* E.g. a host specific initramfs with amdgpu goes down from 40MB to 20MB
* No longer need to worry about Nvidia GSP firmware size in initrd
* This should also significantly shrink the initrd used in liveimages

2. Faster boot times:

* Loading + unpacking the initrd can take a surprising amount of time. E.g. on my old AMD64 embedded PC (with BobCat cores) the reduction of 40MB -> 20MB in initrd size shaves approx. 3 seconds of initrd load time + 0.6s seconds from the time it takes to unpack the initrd
*  Probing drm connectors can be slow and plymouth blocks the initrd -> rootfs transition while it is busy probing

3. Earlier showing of splash. By using simpledrm for the splash the splash can be shown earlier, avoiding the impression the machine is hanging during boot. An extreme example of this is my old AMD64 embedded PC, where the time to show the first frame of the splash goes down from 47 to 9 seconds.

4. One less thing to worry about when trying to create a uniform desktop pre-generated and signed initramfs (these would still need support for nvme + ahci and commonly used rootfs + lvm + luks).

Disadvantages:

Doing this will lead to user visible changes in the boot process:

1. Secondary monitors not lit up by the efifb will stay black during full-disk encryption password entry, since the GPU drivers will now only load after switching to the encrypted root. This includes any monitors connected to the non boot GPU in dual GPU setups.


2. With simpledrm plymouth does not get the physical size of the monitor, so plymouth will need to switch to using heuristics on the resolution instead of DPI info to decide whether or not to use hidpi (e.g. 2x size) rendering and even when switching to the real GPU driver plymouth needs to stay with its initial heuristics based decision to avoid the scaling changing when switching to the real driver which would lead to a big visual glitch / change halfway through the boot.

This may result in a different scaling factor for some setups, but I do not expect this really to be an issue.

3. On some (older) systems the efifb will not come up in native mode, but rather in 800x600 or 1024x768.

This will lead to a pretty significant discontinuity in the boot experience when switching from say 800x600 to 1920x1080 while plymouth was already showing the spinner at 800x600.

One possible workaround here is to add: 'video=efifb:auto' to the kernel commandline which will make the efistub switch to the highest available resolution before starting the kernel. But it seems that the native modes are simply not there on systems which come up at 800x600 / 1024x768 so this does not really help.

This does not actually break anything but it does look a bit ugly. So we will just need to document this as an unfortunate side-effect of the change and then we (and our users) will have to live with this (on affected hardware).

4. On systems where a full modeset is done the monitor going briefly black from the modeset will move from being just before plymouth starts to the switch from simpledrm drm to the real driver. So that is slightly worse. IMHO the answer here is to try and get fast modesets working on more systems.

5. On systems where the efifb comes up in the panel's native mode and a fast modeset can be done, the spinner will freeze for a (noticeable) fraction of a second as the switch to the real driver happens.

Preview:

To get an impression what this will look / feel like on your own systems, you can implement this right now on Fedora 40 with some manual configuration changes:

1. Create /etc/dracut.conf.d/omit-gpu-drivers.conf with:

omit_drivers+=" amdgpu radeon nouveau i915 "

And then run "sudo dracut -f" to regenerate your current initrd.

2. Add to kernel commandline: "plymouth.use-simpledrm"

3. Edit /etc/selinux/config, set SELINUX=permissive this is necessary because ATM plymouth has issues with accessing drm devices after the chroot from the initrd to the rootfs.

Note this all assumes EFI booting with efifb used to show the plymouth boot splash. For classic BIOS booting it is probably best to stick with having the GPU drivers inside the initramfs.

Many recent Intel laptops have replaced the standard UVC USB camera module with a raw MIPI camera-sensor connected to the IPU6 found in recent Intel laptop chips.

Both the hw interface of the ISP part of the IPU6 as well as the image processing algorithms used are considered a trade secret and so far the only Linux support for the IPU6 relies on an out of tree kernel driver with a proprietary userspace stack on top, which is currently available in rpmfusion.

Both Linaro and Red Hat have identified the missing ISP support for various ARM and X86 chips as a problem. Linaro has started a project to add a SoftwareISP component to libcamera to allow these cameras to work without needing proprietary software and Red Hat has joined Linaro in working on this.

FOSDEM talk

Bryan O'Donoghue (Linaro) and I are giving a talk about this at FOSDEM.

Fedora COPR repository

This work is at a point now where it is ready for wider testing. A Fedora COPR repository with a patched kernel and libcamera is now available for users to test, see the COPR page for install and test instructions.

This has been tested on the following devices:

• Lenovo ThinkPad X1 yoga gen 8 (should work on any ThinkPad with ov2740 sensor)
• Dell Latitude 9420 (ov01a1s sensor)
• HP Spectre x360 13.5 (2023 model, hi556 sensor)

Description of the stack

1. Kernel driver for the camera sensor, for the ov2740 used on current Lenovo designs (excluding MTL) I have landed all necessary kernel changes for this upstream.
2. Kernel support for the CSI receiver part of the IPU6 Intel is working on upstreaming this and has recently posted v3 of their patch series for this upstream and this is under active review.
3. A FOSS Software ISP stack inside libcamera to replace the missing IPU6 ISP (processing-system/psys) support. Work on this is under way. I've recently send out v2 of the patch-series for this.
4. Firefox pipewire camera support and support for the camera portal to get permission to access the camera. My colleague Jan Grulich has been working on this, see Jan's blogpost. Jan's work has landed in the just released Firefox 122.

There is an issue with the rpmfusion packaged IPU6 camera stack for Fedora is not working on many Dell laptop models after upgrading the kernel to a 6.5.y kernel.

This is caused by a new mainline ov0a10 sensor driver which takes precedence over the akmod ov0a10 driver but lacks VSC integration.

This can be worked around by running the following command:

sudo rm /lib/modules/$(uname -r)/kernel/drivers/media/i2c/ov01a10.ko.xz; sudo depmod -a

After the rm + depmod run:

sudo rmmod ov01a10; sudo modprobe ov01a10

Or reboot. After this your camera will hopefully work again.

I have submitted a pull-request to disable the mainline kernel's non working ov01a10 driver, so after the next Fedora kernel update this workaround should no longer be necessary.

I have just become aware that Fedora users using the packaged IPU6 camera stack are having an issue where the output from the camera is black. We have been updating the stack and the new version of ipu6-camera-bins has hit the stable updates repo, while the matching new version of ipu6-camera-hal is currently in the updates-testing repo.

This causes the versions of ipu6-camera-bins vs ipu6-camera-hal to get out of sync (unless you have updates-testing enabled) which leads to the camera output being all black

You can fix this issue by running the following command:

sudo dnf update --enablerepo=rpmfusion-nonfree-updates-testing 'ipu6-camera-*'

Sorry about the inconvenience, we'll make sure to push both packages at the same time for the next set of updates.

I have tagged all the new ipu6-camera-hal builds to be moved to the stable update repositories, so on the next rpmfusion updates push this should be resolved.

Installation

I am happy to announce that Intel's IPU6 camera stack has been packaged in rpmfusion and now can be installed under Fedora 37 and newer with a single `dnf install` command.

Note since this uses an out of tree kernel module build as unsigned akmod you need to disable secureboot for this to work; or alternatively sign the kmod with your own local key (instructions here).

First enable both the rpmfusion-free and rpmfusion-nonfree repositories, for instructions see https://rpmfusion.org/Configuration

The IPU6 support requires kernel >= 6.3.1 which is in updates-testing for now and v4l2loopback also needs to be updated to the latest version (in case you already have it installed):

sudo dnf update \
  --enablerepo=updates-testing \
  --enablerepo=rpmfusion-free-updates-testing \
  --enablerepo=rpmfusion-nonfree-updates-testing \
  'kernel*' '*v4l2loopback'

And now things are ready to install the IPU6 driver stack:

sudo dnf install \
  --enablerepo=updates-testing \
  --enablerepo=rpmfusion-free-updates-testing \
  --enablerepo=rpmfusion-nonfree-updates-testing \
  akmod-intel-ipu6

After this command reboot and you should be able to test your camera with https://mozilla.github.io/webrtc-landing/gum_test.html under firefox now.

This relies on Intel's partly closed-source hw-enablement for the IPU6, as such this known to not work on laptop models which are not covered by Intel's hw-enablement work. If your laptop has an option to come with Linux pre-installed and that SKU uses the IPU6 cameras then this should work. Currently known to work models are:

• Dell Latitude 9420 (ov01a1s sensor)


• Dell Precision 5470 (ov01a10 sensor)


• Dell XPS 13 Plus 9320 (ov01a10 sensor)


• Lenovo ThinkPad X1 Carbon Gen 10 (ov2740 sensor)


• Lenovo ThinkPad X1 Nano Gen 2 (ov2740 sensor)


• Lenovo ThinkPad X1 Yoga Gen 7 (ov2740 sensor)

If the IPU6 driver works for you on an unlisted model please drop mean email at so that the above list can be updated.

Description of the stack

The IPU6 camera stack consists of the following layers:

1. akmod-intel-ipu6 the IPU6 kernel drivers. These are currently out of tree. Work is ongoing on getting various IO-expander, sensor drivers and the CSI2 receiver patches upstream. This is a slow process though and currently there is no clear path to getting the actual ISP part of the IPU supported upstream.


2. ipu6-camera-bins this is a set of closed-source userspace libraries which the rest of the userspace stack builds on top of. There is a separate set of libraries for each IPU6 variant. Currently there are 2 sets, "ipu6" for Tiger Lake and "ipu6ep" for Alder Lake.


3. ipu6-camera-hal this is a library on top of the set of libraries in ipu6-camera-bins. This needs to be built separately for the "ipu6" and "ipu6ep" library sets from ipu6-camera-bins.


4. gstreamer1-plugins-icamerasrc a gstreamer plugin built on top of ipu6-camera-hal. This allows using the camera through gstreamer.


5. akmod-v4l2loopback + v4l2-relayd. Most apps don't use gstreamer for camera access and even those that do don't know they need to use the icamerasrc element. v4l2-relayd will monitor a v4l2loopback /dev/video0 node and automatically start a gstreamer pipeline to send camera images into the loopback when e.g. firefox opens the /dev/video0 node to capture video.

Packaging challenges and technical details

1. akmod-intel-ipu6: There were 2 challenges to overcome before the IPU6 kernel drivers could be packaged:
   
   
   1. The sensor drivers required patches to the main kernel package, specifically to the INT3472 driver which deals with providing GPIO, clk, regulator and LED resources to the sensor drivers. Patches have been written for both the main kernel, including some LED subsystem core additions, as well as patches to the IPU6 sensor drivers to bring them inline with mainline kernel conventions for GPIOs, clks and LEDs. All the necessary patches for this are upstream now, allowing the latest ipu6-drivers code to work with an unmodified mainline kernel.
   
   
   2. Until now the IPU6 drivers seem to have been used with a script which manually loads the modules in a specific order. Using automatic driver loading by udev exposed various probe-ordering issues. Requiring numerous patches (all upstreamed to Intel's github repo) to fix.
   
   


2. ipu6-camera-bins: Since there were 2 sets of libraries for different IPU6 versions, these are now installed in separate /usr/lib64/ipu6[ep] directories and the headers and pkgconfig files are also installed in 2 different variants.


3. ipu6-camera-hal: This needs to be built twice against the 2 different sets of ipu6-camera-bins libraries. Letting the user pick the right libcamhal.so build to install is not very user friendly, to avoid the user needing to manually chose:
   
   
   1. Both builds are installed in separate /usr/lib64/ipu6[ep] directories.
   
   
   2. The libcamhal.so under these directories is patched to include the directory it is installed in as RPATH, so that dynamic-linking against that libcamhal.so will automatically link against the right set of ipu6-camera-bins libraries.
   
   
   3. To make all this all work transparently the actual /usr/lib64/libcamhal.so is a symlink to /run/libcamhal.so and /run/libcamhal.so is set by udev rules to point to /usr/lib64/ipu6[ep]/libcamhal.so depending on the actual hw. The /run/libcamhal.so indirection is there so that things will also work with an immutable /usr .
   
   
   4. ipu6-camera-hal's udev rules also set a /run/v4l2-relayd config file symlink to configure the gstreamer pipeline use by v4l2-relayd to match the ipu6 vs ipu6ep capabilities.
   
   


4. akmod-v4l2loopback + v4l2-relayd: Getting this to work with Firefox was somewhat tricky, there were 2 issues which had to be solved:
   
   
   1. Firefox does not accept the NV12 image format generated by ipu6ep pipelines. To work around this a conversion to YUV420 has been added to the v4l2-relayd pipeline feeding into v4l2loopback. This workaround can be dropped once Firefox 114, which will have NV12 support, is released.
   
   
   2. Gstreamer sends v4l2-buffers with a wrong bytesused field value into v4l2loopback causing Firefox to reject the video frames. A patch has been written and merged upstream to make v4l2loopback fix up the bytesused value, fixing this.
   
   

Many thanks to my colleague Kate Hsuan for doing most of the packaging work for this.

And also a big thank you to the rpmfusion team for maintaining the rpmfusion repo and infrastructure which allows packaging software which does not meet Fedora's strict guidelines outside of the Fedora infra.

As mentioned in my previous blog post, I have written a new patch series for 6.2 to try to avoid having multiple entries in /sys/class/backlight for a single panel again.

This new series might cause regressions on a different set of even older laptop models then the one affected by the 6.1 backlight work. So I'm again looking for people willing to run a few quick tests.

To see if your laptop is possibly affected by the 6.2 change please run:

• ls /sys/class/backlight

if the output of this command contains both:

1. A GPU native backlight device, with intel/nv/nvidia/amd/radeon/psb/oaktrail in the name; and


2. A vendor backlight device, with either a model-series: eeepc, ideapad, thinkpad, etc; or a vendor-name: acer, asus, dell, toshiba, sony, etc. in the name

then 6.2 will cause a behavior change on your device, it will hide the vendor backlight device in preference of the native backlight device.

If your laptop shows only a native or only a vendor backlight device (possibly in combination with another type of backlight device such as acpi_video#), then your laptop will not be affected by the planned changes for 6.2.

Note that I expect only very old models to be affected, e.g. the Sony Vaio PCG-FRV3
from 2003 is known to be affected.

If your laptop has both native + vendor backlight devices, then please do 2 things:

1. Please run the following commands:
   
   
   1. ls /sys/class/backlight > ls-backlight.txt
   
   
   2. sudo dmesg > dmesg.txt
   
   
   3. sudo dmidecode > dmidecode.txt
   
   
   4. sudo acpidump -o acpidump.txt
   
   


2. Please test if the native backlight interface works, the example below assumes the native backlight is called "intel_backlight":
   
   
   1. cd /sys/class/backlight/intel_backlight
   
   
   2. cat max_brightness
   
   
   3. <the "cat max_brightness" will show the maximum brightness value supported>
   
   
   4. echo $max_brightness_value > brightness
   
   
   5. echo $half-of-max_brightness_value > brightness
   
   

After generating the 4 .txt files and running the native backlight tests, please send me an email about this at hdegoede@redhat.com with the results of the native backlight tests (panel brightness changes when echo-ing values to brightness or not?) and with the 4 generated .txt files attached.

If the native backlight interface works then things should keep working fine with 6.2 and typically you will get more fine-grained brightness control as an added bonus. Please send me an email with the test results even if the native backlight interface works.

I have received quite a few test reports in response to my previous blog post. Many thanks to everyone who has run the tests and send me their results!

These tests show that as a result of the current 6.1 changes quite a few laptop models will end up with an empty "/sys/class/backlight", breaking users ability to control their laptop panel's brightness.

I have submitted a patch-set for 6.1 upstream to fix this.

More detailed summary/analysis of the received test reports:

• Known Windows laptop models affected by this:
  
  
  • Acer Aspire 1640
  
  
  • HP Compaq nc6120
  
  
  • IBM ThinkPad X40
  
  
  • System76 Starling Star1
  
  


• Known MacBook models affected by this:
  
  
  • Apple MacBook 2.1
  
  
  • Apple MacBook 4.1
  
  
  • Apple MacBook Pro 7.1
  
  


• 30 unaffected models


• The Dell Inspiron N4010 has acpi_video support and acpi_osi_is_win8() returns false, so acpi_video_get_backlight_type() returns acpi_video, but acpi_video fails to register a backlight device due to a _BCM eval error. This model already has a DMI quirk for this, so it is unaffected.


• The following laptop models use vendor backlight control method, while also having a native backlight entry under /sys/class/backlight:
  
  
  • Asus EeePC 901 (native backlight confirmed to work better then vendor)
  
  
  • Dell Latitude D610 (native backlight confirmed to also work)
  
  
  • Sony Vaio PCG-FRV3 (native backlight control does not work, BIOS from 2003!)
  
  

The fixes for 6.1 restore the behavior where userspace can see multiple entries under "/sys/class/backlight" for a single panel and the kernel leaves figuring out which one actually works up to userspace. This is undesirable and having more then 1 backlight device for a single panel also blocks the new backlight userspace API work which I have planned.

This first round of testing has shown that native works well even on systems so old that they don't have acpi_video backlight control support.

So I have prepared a patch series to try again with 6.2 by making native be preferred over vendor, which should avoid the problems seen with the 6.1 changes before the fixes.

I have landed a large(ish) refactor of the ACPI/x86 backlight detection code in the kernel for 6.1. I have been very careful to try and not break things but there is a special group of laptops where the ability to control the backlight brightness may disappear because of this.

The most likely laptops to be hit by this are laptops which are either pretty old and or which are weird in some other way (e.g. flashed with coreboot, did not ship with Windows as factory os, ...). Note Chromebooks are affected by this too, but that special category has already been fixed.

You can check if your laptop is affected by this by running "ls /sys/class/backlight" if this shows only 1 entry and that entry is named "intel_backlight", "nouveau_bl", "amdgpu_bl0" or "radeon_bl0" then your laptop might be affected.

Note this is quite normal on modern(ish) laptops, a second check is to boot with "acpi_backlight=video" added to the kernel commandline and then run "ls /sys/class/backlight" again, if you now additionally also have an "acpi_video0" entry then your laptop should work fine with 6.1, if you don't have an "acpi_video0" entry please first do "cat /proc/cmdline" and check that "acpi_backlight=video" is present there.

If you have e.g. only the "intel_backlight" entry and adding "acpi_backlight=video" does not cause an "acpi_video0" entry to appear then 6.1 will likely break backlight control!

If you have a laptop which is likely affected by this then please run the following commands:

• ls /sys/class/backlight > ls-backlight.txt


• sudo dmesg > dmesg.txt


• sudo dmidecode > dmidecode.txt


• sudo acpidump -o acpidump.txt

And send me an email about this at hdegoede@redhat.com with the 4 generated .txt files attached. If possible please also give an actual 6.1 kernel a try and see if that indeed breaks things. E.g. for Fedora you can find 6.1 kernel builds here and see here for some install instructions for these Fedora kernel builds.

Now that FESCo has decided that Fedora will keep supporting BIOS booting, the people working on Fedora's bootloader stack will need help from the Fedora community to keep Fedora booting on systems which require Legacy BIOS to boot.

To help with this the Fedora BIOS boot SIG (special interest group) has been formed. The main goal of this SIG are to help the Fedora bootloader people by:

1. Doing regular testing of nightly Fedora N + 1 composes on hardware
   which only supports BIOS booting


2. Triaging and/or fixing BIOS boot related bugs.

A biosboot-sig@lists.fedoraproject.org mailinglist and bugzilla account has been created, which will be used to discuss testing result and as assignee / Cc for bootloader bugzillas which are related to BIOS booting.

If you are interested in helping with Fedora BIOS boot support please:

1. Subscribe to the email-list


2. Add yourself to the Members section of the SIG's wiki page

Starting with kernel 5.17 the kernel supports the builtin privacy screens built into the LCD panel of some new laptop models.

This means that the drm drivers will now return -EPROBE_DEFER from their probe() method on models with a builtin privacy screen when the privacy screen provider driver has not been loaded yet.

To avoid any regressions distors should modify their initrd generation tools to include privacy screen provider drivers in the initrd (at least on systems with a privacy screen), before 5.17 kernels start showing up in their repos.

If this change is not made, then users using a graphical bootsplash (plymouth) will get an extra boot-delay of up to 8 seconds (DeviceTimeout in plymouthd.defaults) before plymouth will show and when using disk-encryption where the LUKS password is requested from the initrd, the system will fallback to text-mode after these 8 seconds.

I've written a patch with the necessary changes for dracut, which might be useful as an example for how to deal with this in other initrd generators, see: https://github.com/dracutdevs/dracut/pull/1666

I've also filed bugs for tracking this for Fedora, openSUSE, Arch, Debian and Ubuntu.