Booting a custom kernel/OS on the Raspberry Pi 3 B (64-bit)

On Wednesday afternoon, we were having a discussion of various 64-bit ARM SoCs on the Adélie Linux chat. I mentioned that the only one I had was a PINE64, which did not support graphical display on the version of the Linux kernel we ship. Michael, one of our contributors, suggested I should buy a Raspberry Pi 3 for testing graphical software (such as KDE and Firefox). I was surprised to learn that Target carries the RPi3 and they had a few in stock locally. Off I went, stopping in for a 128 GB µSD card as well.

Then began the fun: I had a RPi 2B some years ago, but it was lost in the move. I only have experience booting these things using NOOBS, and I wanted to use NOOBS anyway because that would allow me an easy way to have multiple system images and choose which one to boot. This will let me perform 32-bit ARM testing on the RPi3 later on, when we support 32-bit ARM a bit better. Therefore the first order of business was to figure out how NOOBS’ boot menu works, so that I could write a custom configuration with the partition layout how I desired:

  • Shared /home, 40 GB XFS
  • 64-bit /boot volume, 1 GB FAT32
  • 64-bit root (/) volume, 39 GB ext4
  • 32-bit /boot volume, 1 GB FAT32
  • 32-bit root (/) volume, 39 GB ext4

Since NOOBS erases the SD card on first boot, I booted NOOBS once on the Pi before continuing. Then I went to work in fdisk on my Talos:


Disk /dev/sdb: 119.3 GiB, 128043712512 bytes, 250085376 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb41dea51

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sdb1            8192    137215    129024    63M  e W95 FAT16 (LBA)
/dev/sdb2          137216 250085375 249948160 119.2G  5 Extended
/dev/sdb5          139264    204797     65534    32M 83 Linux
/dev/sdb6          206848  81995775  81788928    39G 83 Linux
/dev/sdb7        81997824  84092927   2095104  1023M  c W95 FAT32 (LBA)
/dev/sdb8        84094976 167981055  83886080    40G 83 Linux
/dev/sdb9       167983104 170078207   2095104  1023M  c W95 FAT32 (LBA)
/dev/sdb10      170080256 250085375  80005120  38.2G 83 Linux

Through trial and error, with a fairly good reference and reading the source code of NOOBS, I was able to divine the layout of my desired installed_os.json file. Once NOOBS is booted for the first time, it makes a small “settings” partition; this was /dev/mmcblk0p5 on the Pi and /dev/sdb5 in the SD card reader of my workstation.


[
        {
                "description": "A friendly, easy-to-use desktop environment that uses little resources",
                "folder": "/mnt/os/Adelie64",
                "icon": "/settings/os/adelie.png",
                "name": "Adélie Linux (64-bit)",
                "partitions" : [ "/dev/mmcblk0p7", "/dev/mmcblk0p8" ],
                "release_date": "2019-02-21"
        },
        {
                "description": "A friendly, easy-to-use desktop environment that uses little resources",
                "folder": "/mnt/os/Adelie32",
                "icon": "/settings/os/adelie.png",
                "name": "Adélie Linux (32-bit)",
                "partitions" : [ "/dev/mmcblk0p9", "/dev/mmcblk0p10" ],
                "release_date": "2019-02-21"
        }
]

I placed this file in the root of the settings partition (/dev/sdb5 in this example) and downloaded the header image from our Web site to /os/adelie.png on the same. Now the structure looked like:


gwyn /mnt/PiStuff # tree
.
├── cache
│   ├── data7
[ lots of irrelevant files removed ]
├── installed_os.json
├── lost+found
├── noobs.conf
├── os
│   └── adelie.png
└── wpa_supplicant.conf

I then unpacked the Adélie Linux aarch64 root FS image to the root partition (/dev/sdb8 in this example). I downloaded the needed firmware files for /boot from the official Raspberry Pi Foundation GitHub repository, and put them in the boot partition (/dev/sdb7 in this example). Then came a full day of trial and error with the kernel.

Kernel config

Our Adélie easy-kernel package boots great on the PINE64 and a few other SoCs, but it needed small adjustments to be comfortable booting on the Raspberry Pi 3 B. In addition to the configuration knobs noted by the Raspberry Pi Foundation, we added CONFIG_FB_SSD1307=m and CONFIG_FB_UDL=m for eventual support of the Pi’s touchscreen interface. (I have not yet tested that and do not know if this is sufficient or not.)

Next was spending four hours trying to understand why the system booted properly but had no display on-screen. dmesg contained:


[   11.423042] vc4-drm soc:gpu: failed to bind 3f902000.hdmi (ops vc4_hdmi_ops [vc4]): -517
[   11.423147] vc4-drm soc:gpu: master bind failed: -517

As it turns out, this was because I was using DTB files from a different kernel build. It’s incredibly important to use the DTB files that ship with the kernel you are testing with! After installing the correct DTB files to the SD card and booting again, a different message was output:


[   14.347564] vc4_hdmi 3f902000.hdmi: vc4-hdmi-hifi  3f902000.hdmi mapping ok
[   14.348456] vc4-drm soc:gpu: bound 3f902000.hdmi (ops vc4_hdmi_ops [vc4])
[   14.348689] vc4-drm soc:gpu: bound 3f806000.vec (ops vc4_vec_ops [vc4])
[   14.349175] vc4-drm soc:gpu: bound 3f400000.hvs (ops vc4_hvs_ops [vc4])
[   14.349459] vc4-drm soc:gpu: bound 3f206000.pixelvalve (ops vc4_crtc_ops [vc4])
[   14.349687] vc4-drm soc:gpu: bound 3f207000.pixelvalve (ops vc4_crtc_ops [vc4])
[   14.349903] vc4-drm soc:gpu: bound 3f807000.pixelvalve (ops vc4_crtc_ops [vc4])
[   14.349968] vc4-drm soc:gpu: failed to allocate buffer with size 16777216
[   14.349995] vc4-drm soc:gpu: failed to allocate buffer with size 16777216
[   14.350080] [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from CMA:
[   14.350092] vc4_v3d 3fc00000.v3d: Failed to allocate memory for tile binning: -12. You may need to enable CMA or give it more memory.
[   14.350195] vc4-drm soc:gpu: failed to bind 3fc00000.v3d (ops vc4_v3d_ops [vc4]): -12
[   14.386347] vc4-drm soc:gpu: master bind failed: -12
[   14.386385] vc4-drm: probe of soc:gpu failed with error -12

This was correct; the solution was adding cma=256M@256M to the kernel command line. The final command line that I use is: root=/dev/mmcblk0p8 rootwait ro cma=256M@256M.

I rebuilt Mesa with VC4 support; our 32-bit ARM package had it already, but we did not enable it on 64-bit ARM. Then I installed KDE and X.Org: blackjack ~ # apk add kde x11 mesa-dri-vc4. Behold!

Adélie Linux running KDE Plasma 5 on a Raspberry Pi 3 in 64-bit mode

Please consider supporting our continued adventures bringing quality Libre Software to more platforms! You can find ways to help out on the Adélie Linux Contribution page, or directly on Patreon, PayPal, or Ko-fi. I hope this article was able to help you learn something new about your Pi.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s