OK3588 6.1.118 Buildroot PREEMPT_RT Real-Time Patch Application and Performance Testing

by yir65681 in Circuits > Microcontrollers

9 Views, 0 Favorites, 0 Comments

OK3588 6.1.118 Buildroot PREEMPT_RT Real-Time Patch Application and Performance Testing

19d95717-4b80-4cd2-9e0b-20c7ce06bc9f.png

In industrial control, motion control, machine vision, edge data acquisition, and other application scenarios, the system must not only run but also operate on time. In scenarios that require high determinism in response, standard Linux often struggles to meet strict real-time requirements related to scheduling latency and interrupt response. As a result, enhancing real-time performance on Linux platforms has become a key focus for many developers.

This article provides a comprehensive overview of the acquisition, porting, compilation adaptation, and real-time verification of the PREEMPT_RT real-time patch on the OK3588 Buildroot platform, along with actual testing processes. Real-time performance before and after applying the patch is compared using cyclictest. The results indicate that the maximum scheduling latency of the system is significantly improved in both idle and stress scenarios.

How to Obtain the Real-Time Patch for the OK3588 Platform?

For the OK3588 platform, the SDK provides the PREEMPT_RT patch corresponding to the relevant version.

ls docs/rk3588/Patches/Real-Time-Performance/PREEMPT_RT/kernel-6.1/kernel-6.1.118/
0001-patch-6.1.99-rt36-on-rockckip-base-5c295c763974.patch
0002-sched-isolation-remove-HK_FLAG_TICK-for-nohz_full-fo.patch
0003-mm-Kconfig-remove-selection-of-MIGRATION-for-CMA-to-.patch
0004-ARM-configs-add-rockchip_rt.config-for-PREEMPT_RT.patch
0005-arm64-configs-optimize-latency-for-PREEMPT_RT.patch
0006-phy-rockchip-inno-usb2-Fix-DEBUG_LOCKS_WARN_ON-in-ch.patch

The path includes several patch files related to real-time performance, such as the foundational RT patch, scheduler isolation optimizations, memory configuration adjustments, additional real-time kernel configurations, and USB-related fixes, among others.

It is important to note that applying PREEMPT_RT is not a simple task of "apply one patch and you're done." Since the preemption points of the real-time patch are distributed throughout nearly the entire kernel, chip manufacturers often need to adjust some underlying drivers accordingly. Therefore, from an engineering perspective, it is more reliable and easier to implement patch sets that have already been curated by chip manufacturers or platform providers, rather than using generic upstream RT patches directly.

How to Port the Real-Time Patch to the Buildroot Kernel?

According to the workflow outlined in the documentation, porting the real-time patch for the OK3588 6.1.118

Batch Application of Patches in the Kernel Directory

First, navigate to the kernel directory and use the patch command to import the real-time patch files provided in the SDK.

for i in ../docs/rk3588/Patches/Real-Time-Performance/\
PREEMPT_RT/kernel-6.1/kernel-6.1.118/*;\
do patch -p1 -i $i; done

Supplementing and Saving the Corresponding Kernel Configuration Items

After entering the patch, you also need to add the corresponding CONFIG item.

# Must be executed, otherwise there will be problems with the compilation environment.
export ARCH=arm64
make menuconfig
make savedefconfig
# You need to select the defconfig corresponding to the SDK
cp defconfig arch/arm64/configs/OK3588-C-linux_defconfig

Operation on the menu is performed by pressing the ''/'' key to search for the PREEMPT option, then pressing 1 to enter it.

Linux kernel menuconfig terminal interface showing the search results for the PREEMPT option during OK3588 Development Board Buildroot compilation

Navigate further into the submenu, select the highest real-time priority level, and proceed to compile.

Linux kernel menuconfig submenu displaying the selection of the Fully Preemptible Kernel (RT) priority level for the PREEMPT_RT patch configuration

Return to the Top-level SDK Directory and Perform a Full Compilation

After configuration is complete, return to the top-level SDK directory and execute ./build.sh to compile the entire system.

tips:
Encountering errors during compilation is common, as different SDKs incorporate various drivers that may not be adapted to the modifications introduced by the real-time patch.

Console terminal output displaying Buildroot compilation error logs caused by driver incompatibilities after applying the PREEMPT_RT real-time patch on the OK3588 Dev Kit

For instance, by examining the changes made to specific functions within the real-time patch, clues about the issue can often be identified.

cd docs/rk3588/Patches/Real-Time-Performance/PREEMPT_RT/kernel-6.1/kernel-6.1.118/
grep -nr "u64_stats_fetch_begin"
cd -

Console terminal output showing the grep command results identifying the replacement of the u64_stats_fetch_begin_irq function in the PREEMPT_RT patch source code

As observed in the real-time patch, the function `u64_stats_fetch_begin_irq` has been completely replaced by `u64_stats_fetch_begin`. By referencing this modification from the real-time patch and making the corresponding adjustments to the driver, the compilation was successful, allowing the firmware to be flashed for testing.

Real-Time Performance Testing

Download the compressed package and place it on the development board. Execute the command `tar -xvf rt-tests-2.2.tar -C /` to extract the contents into the root directory. The `cyclictest` tool will then be available in `/usr/bin/`.

Testing Without the Real-Time Patch Applied

cyclictest -c 0 -m -t 8 -p 99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.00 0.02 0.00 1/208 3200
T: 0 ( 2749) P:99 I:1000 C: 94901 Min: 2 Act: 3 Avg: 2 Max: 43
T: 1 ( 2750) P:99 I:1500 C: 63265 Min: 2 Act: 2 Avg: 2 Max: 31
T: 2 ( 2751) P:99 I:2000 C: 47446 Min: 2 Act: 3 Avg: 2 Max: 36
T: 3 ( 2752) P:99 I:2500 C: 37956 Min: 2 Act: 3 Avg: 3 Max: 39
T: 4 ( 2753) P:99 I:3000 C: 31628 Min: 2 Act: 3 Avg: 2 Max: 42
T: 5 ( 2754) P:99 I:3500 C: 27109 Min: 2 Act: 2 Avg: 2 Max: 16
T: 6 ( 2755) P:99 I:4000 C: 23720 Min: 2 Act: 2 Avg: 2 Max: 69
T: 7 ( 2756) P:99 I:4500 C: 21084 Min: 2 Act: 2 Avg: 2 Max: 28

Testing With the Real-Time Patch Applied

No-load Test

root@OK3588-C-buildroot:/# cyclictest -c 0 -m -t 8 -p 99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 2.70 0.93 0.34 1/373 2203
T: 0 ( 1961) P:99 I:1000 C: 46702 Min: 0 Act: 0 Avg: 0 Max: 4
T: 1 ( 1963) P:99 I:1500 C: 31133 Min: 0 Act: 1 Avg: 0 Max: 6
T: 2 ( 1964) P:99 I:2000 C: 23348 Min: 0 Act: 1 Avg: 0 Max: 3
T: 3 ( 1965) P:99 I:2500 C: 18677 Min: 0 Act: 1 Avg: 0 Max: 3
T: 4 ( 1966) P:99 I:3000 C: 15563 Min: 0 Act: 0 Avg: 0 Max: 1
T: 5 ( 1967) P:99 I:3500 C: 13339 Min: 0 Act: 0 Avg: 0 Max: 1
T: 6 ( 1968) P:99 I:4000 C: 11670 Min: 0 Act: 0 Avg: 0 Max: 1
T: 7 ( 1969) P:99 I:4500 C: 10373 Min: 0 Act: 0 Avg: 0 Max: 1

Test Performance Mode

echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
echo performance > /sys/devices/system/cpu/cpufreq/policy4/scaling_governor
echo performance > /sys/devices/system/cpu/cpufreq/policy6/scaling_governor
cyclictest -c 0 -m -t 8 -p 99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 2.03 2.76 2.40 2/345 11678
T: 0 (11652) P:99 I:1000 C: 3968 Min: 0 Act: 1 Avg: 0 Max: 2
T: 1 (11653) P:99 I:1500 C: 2641 Min: 0 Act: 0 Avg: 0 Max: 6
T: 2 (11654) P:99 I:2000 C: 1977 Min: 0 Act: 1 Avg: 0 Max: 2
T: 3 (11655) P:99 I:2500 C: 1581 Min: 0 Act: 1 Avg: 0 Max: 2
T: 4 (11656) P:99 I:3000 C: 1315 Min: 0 Act: 0 Avg: 0 Max: 1
T: 5 (11657) P:99 I:3500 C: 1127 Min: 0 Act: 0 Avg: 0 Max: 1
T: 6 (11658) P:99 I:4000 C: 986 Min: 0 Act: 0 Avg: 0 Max: 1
T: 7 (11659) P:99 I:4500 C: 875 Min: 0 Act: 0 Avg: 0 Max: 1

Pressure Test

stress-ng -c 8 --io 2 --vm 1 --vm-bytes 1024M --timeout 1000000s &
cyclictest -c 0 -m -t 8 -p 99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 9.48 3.99 2.84 11/365 13327
T: 0 (13232) P:99 I:1000 C: 18287 Min: 0 Act: 1 Avg: 1 Max: 7
T: 1 (13233) P:99 I:1500 C: 12192 Min: 0 Act: 1 Avg: 1 Max: 8
T: 2 (13234) P:99 I:2000 C: 9142 Min: 0 Act: 1 Avg: 1 Max: 3
T: 3 (13235) P:99 I:2500 C: 7310 Min: 0 Act: 1 Avg: 1 Max: 4
T: 4 (13236) P:99 I:3000 C: 6088 Min: 0 Act: 0 Avg: 0 Max: 6
T: 5 (13237) P:99 I:3500 C: 5219 Min: 0 Act: 0 Avg: 0 Max: 3
T: 6 (13238) P:99 I:4000 C: 4565 Min: 0 Act: 1 Avg: 0 Max: 3
T: 7 (13239) P:99 I:4500 C: 4056 Min: 0 Act: 0 Avg: 0 Max: 4