Booting a custom kernel/OS on the Raspberry Pi 3 B (64-bit)

On Wednesday afternoon, we were having a discussion of various 64-bit ARM SoCs on the Adélie Linux chat. I mentioned that the only one I had was a PINE64, which did not support graphical display on the version of the Linux kernel we ship. Michael, one of our contributors, suggested I should buy a Raspberry Pi 3 for testing graphical software (such as KDE and Firefox). I was surprised to learn that Target carries the RPi3 and they had a few in stock locally. Off I went, stopping in for a 128 GB µSD card as well.

Then began the fun: I had a RPi 2B some years ago, but it was lost in the move. I only have experience booting these things using NOOBS, and I wanted to use NOOBS anyway because that would allow me an easy way to have multiple system images and choose which one to boot. This will let me perform 32-bit ARM testing on the RPi3 later on, when we support 32-bit ARM a bit better. Therefore the first order of business was to figure out how NOOBS’ boot menu works, so that I could write a custom configuration with the partition layout how I desired:

  • Shared /home, 40 GB XFS
  • 64-bit /boot volume, 1 GB FAT32
  • 64-bit root (/) volume, 39 GB ext4
  • 32-bit /boot volume, 1 GB FAT32
  • 32-bit root (/) volume, 39 GB ext4

Since NOOBS erases the SD card on first boot, I booted NOOBS once on the Pi before continuing. Then I went to work in fdisk on my Talos:


Disk /dev/sdb: 119.3 GiB, 128043712512 bytes, 250085376 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb41dea51

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sdb1            8192    137215    129024    63M  e W95 FAT16 (LBA)
/dev/sdb2          137216 250085375 249948160 119.2G  5 Extended
/dev/sdb5          139264    204797     65534    32M 83 Linux
/dev/sdb6          206848  81995775  81788928    39G 83 Linux
/dev/sdb7        81997824  84092927   2095104  1023M  c W95 FAT32 (LBA)
/dev/sdb8        84094976 167981055  83886080    40G 83 Linux
/dev/sdb9       167983104 170078207   2095104  1023M  c W95 FAT32 (LBA)
/dev/sdb10      170080256 250085375  80005120  38.2G 83 Linux

Through trial and error, with a fairly good reference and reading the source code of NOOBS, I was able to divine the layout of my desired installed_os.json file. Once NOOBS is booted for the first time, it makes a small “settings” partition; this was /dev/mmcblk0p5 on the Pi and /dev/sdb5 in the SD card reader of my workstation.


[
        {
                "description": "A friendly, easy-to-use desktop environment that uses little resources",
                "folder": "/mnt/os/Adelie64",
                "icon": "/settings/os/adelie.png",
                "name": "Adélie Linux (64-bit)",
                "partitions" : [ "/dev/mmcblk0p7", "/dev/mmcblk0p8" ],
                "release_date": "2019-02-21"
        },
        {
                "description": "A friendly, easy-to-use desktop environment that uses little resources",
                "folder": "/mnt/os/Adelie32",
                "icon": "/settings/os/adelie.png",
                "name": "Adélie Linux (32-bit)",
                "partitions" : [ "/dev/mmcblk0p9", "/dev/mmcblk0p10" ],
                "release_date": "2019-02-21"
        }
]

I placed this file in the root of the settings partition (/dev/sdb5 in this example) and downloaded the header image from our Web site to /os/adelie.png on the same. Now the structure looked like:


gwyn /mnt/PiStuff # tree
.
├── cache
│   ├── data7
[ lots of irrelevant files removed ]
├── installed_os.json
├── lost+found
├── noobs.conf
├── os
│   └── adelie.png
└── wpa_supplicant.conf

I then unpacked the Adélie Linux aarch64 root FS image to the root partition (/dev/sdb8 in this example). I downloaded the needed firmware files for /boot from the official Raspberry Pi Foundation GitHub repository, and put them in the boot partition (/dev/sdb7 in this example). Then came a full day of trial and error with the kernel.

Kernel config

Our Adélie easy-kernel package boots great on the PINE64 and a few other SoCs, but it needed small adjustments to be comfortable booting on the Raspberry Pi 3 B. In addition to the configuration knobs noted by the Raspberry Pi Foundation, we added CONFIG_FB_SSD1307=m and CONFIG_FB_UDL=m for eventual support of the Pi’s touchscreen interface. (I have not yet tested that and do not know if this is sufficient or not.)

Next was spending four hours trying to understand why the system booted properly but had no display on-screen. dmesg contained:


[   11.423042] vc4-drm soc:gpu: failed to bind 3f902000.hdmi (ops vc4_hdmi_ops [vc4]): -517
[   11.423147] vc4-drm soc:gpu: master bind failed: -517

As it turns out, this was because I was using DTB files from a different kernel build. It’s incredibly important to use the DTB files that ship with the kernel you are testing with! After installing the correct DTB files to the SD card and booting again, a different message was output:


[   14.347564] vc4_hdmi 3f902000.hdmi: vc4-hdmi-hifi  3f902000.hdmi mapping ok
[   14.348456] vc4-drm soc:gpu: bound 3f902000.hdmi (ops vc4_hdmi_ops [vc4])
[   14.348689] vc4-drm soc:gpu: bound 3f806000.vec (ops vc4_vec_ops [vc4])
[   14.349175] vc4-drm soc:gpu: bound 3f400000.hvs (ops vc4_hvs_ops [vc4])
[   14.349459] vc4-drm soc:gpu: bound 3f206000.pixelvalve (ops vc4_crtc_ops [vc4])
[   14.349687] vc4-drm soc:gpu: bound 3f207000.pixelvalve (ops vc4_crtc_ops [vc4])
[   14.349903] vc4-drm soc:gpu: bound 3f807000.pixelvalve (ops vc4_crtc_ops [vc4])
[   14.349968] vc4-drm soc:gpu: failed to allocate buffer with size 16777216
[   14.349995] vc4-drm soc:gpu: failed to allocate buffer with size 16777216
[   14.350080] [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from CMA:
[   14.350092] vc4_v3d 3fc00000.v3d: Failed to allocate memory for tile binning: -12. You may need to enable CMA or give it more memory.
[   14.350195] vc4-drm soc:gpu: failed to bind 3fc00000.v3d (ops vc4_v3d_ops [vc4]): -12
[   14.386347] vc4-drm soc:gpu: master bind failed: -12
[   14.386385] vc4-drm: probe of soc:gpu failed with error -12

This was correct; the solution was adding cma=256M@256M to the kernel command line. The final command line that I use is: root=/dev/mmcblk0p8 rootwait ro cma=256M@256M.

I rebuilt Mesa with VC4 support; our 32-bit ARM package had it already, but we did not enable it on 64-bit ARM. Then I installed KDE and X.Org: blackjack ~ # apk add kde x11 mesa-dri-vc4. Behold!

Adélie Linux running KDE Plasma 5 on a Raspberry Pi 3 in 64-bit mode

Please consider supporting our continued adventures bringing quality Libre Software to more platforms! You can find ways to help out on the Adélie Linux Contribution page, or directly on Patreon, PayPal, or Ko-fi. I hope this article was able to help you learn something new about your Pi.

YouTube. 1080p. Big-endian PowerPC + Firefox.

H.264:

[Nightmare Moon grinning about the same way I was when I saw this]
The Moon Rises, playing in 1080p on big-endian Firefox with correct colours.

VP8/VP9:

[ Tractors ]
VP8 and VP9 WebM demos, playing in big-endian Firefox

I additionally shot a short 4 second video clip of the Talos in action.

I’ve filed a bug with Mozilla to upstream this work. If you do have a bmo account, consider Voting for this issue. (Don’t spam the bug tracker with +1 comments; it won’t help.)

Please also consider supporting my progress on Patreon, PayPal, or Ko-fi. I hope you’re as excited about this as I am!

A day in the life of a PowerPC user

By extremely popular request, I’m going to blog my entire computing experience from yesterday (Saturday, the 24th of November, 2018). All of the events written here are real, and actually happened yesterday on my Talos II running Adélie Linux. Names have not been changed to protect the innocent.

When I walk in to my office, the first thing I do is open Konsole and switch to WeeChat, which runs in tmux. I read the backlog for the Adélie Linux and Talos Workstation channels, and view highlights I have from anywhere else. Then, I open Pidgin and catch up on what people have been saying in XMPP conversations (if any). By this time, I’m ready to put on my music for the day, so I switch to Audacious and load up a CD in the Blu-Ray drive connected to my Talos via USB 3.0.

After that, it’s on to Thunderbird 52 ESR, where I sift through my email. Occasionally I’ll copy a few interesting tidbits into KWrite, a simple text editor, to share with others later. (Sometimes I will think of new blog article ideas and will write them in KWrite as well, but that didn’t happen Saturday).

Then it’s on to Firefox, where I check the Adélie BTS, GitLab, and GitHub for any notifications. Depending on how many emails, messages, and notifications I’ve had, it may be time to change CDs — weekends are usually lighter, so I’m still listening to the same Pet Shop Boys album when I finish with that.

I caught up with a few friends on XMPP (via Pidgin) and Mastodon (via Firefox). After having some pleasant conversation, I left my office at this point to help my grandmother. When I came back, I went back to WeeChat and we discussed documentation layouts. Then I went into Firefox and edited the Adélie MediaWiki with status updates on the 1.0-BETA2 roadmap.

After that, I used OpenSSH in Konsole to connect to the new x86_64 builder and set it up with the proper configuration to start building x86_64 packages again. A very boring half hour later I went and played a game of Spider Solitaire using KPat on my Talos while the builder ran rsync for the package cache and cloned necessary Git repositories.

Now were a few fixes to our Web site, which I accomplished using a mixture of vim and Kate on the Talos. Then I used TigerVNC to connect to a remote VM, our European mirror, to inspect its status for a potential hardware upgrade.

KDE packaging is up next. We still had a few packages left from the KDE Applications Suite, so I brought up KDE’s home page in Firefox and copied the URLs into Konsole for newapkbuild. I packaged a few more KDE Applications using my Talos, and once I was satisfied they were working correctly, I signed my commits with GPG and then pushed them via Git so that they could be built on our builders.

Then I wrote more documentation in Kate and talked with some people in WeeChat, until I was satisfied with the progress made. Then I opened VLC and watched a few videos from YouTube before heading off to bed.


Wasn’t that the single-most boring blog post you ever read, just as I said it would be? I don’t know why so many people wanted me to write this out …

‘Twas the night before Thanksgiving…

…and the main Adélie Linux Web serving box went down, in a strange way.

Network access to all the KVM VMs running on our primary dedicated server suddenly dropped, and new connections were refused. Connecting to the host via BMC, I was greeted with possibly the oddest machine identification I’ve ever seen:

Adélie Linux 4.14.76-mc11-easy-p8/ppc64 on chloe
Sat Aug 6 3160 10:08:02 (hvc0; 0 users)

Upon login, it became even weirder:

chloe ~ # uptime
10:07:54 up 24855 days, 3:14, 1 user, load averages: 1.10, 0.25, 0.08

Note how time went backwards! It actually seemed to be stuck in a loop going from about 10:06 AM to 10:11 AM UTC on the 6th of August, 3160 AD. (We have a rack-mounted time machine?) It turns out that there is an issue with the 4.14 branch running KVM on 64-bit PowerPC. Upgrading the box to kernel 4.19.3 appears to have solved the issue.