multiprocessing - ARM: Start/Wakeup/Bringup the other CPU cores/APs and pass execution start address?

Question

I've been banging my head with this for the last 3-4 days and I can't find a DECENT explanatory documentation (from ARM or unofficial) to help me. I've got an ODROID-XU board (big.LITTLE 2 x Cortex-A15 + 2 x Cortex-A7) board and I'm trying to understand a bit more about the ARM architecture. In my "experimenting" code I've now arrived at the stage where I want to WAKE UP THE OTHER CORES FROM THEIR WFI (wait-for-interrupt) state.

The missing information I'm still trying to find is:

1. When getting the base address of the memory-mapped GIC I understand that I need to read CBAR; But no piece of documentation explains how the bits in CBAR (the 2 PERIPHBASE values) should be arranged to get to the final GIC base address

2. When sending an SGI through the GICD_SGIR register, what interrupt ID between 0 and 15 should I choose? Does it matter?

3. When sending an SGI through the GICD_SGIR register, how can I tell the other cores WHERE TO START EXECUTION FROM?

4. How does the fact that my code is loaded by the U-BOOT bootloader affect this context?

The Cortex-A Series Programmer's Guide v3.0 (found here: link) states the following in section 22.5.2 (SMP boot in Linux, page 271):

While the primary core is booting, the secondary cores will be held in a standby state, using the WFI instruction. It (the primary core) will provide a startup address to the secondary cores and wake them using an Inter-Processor Interrupt(IPI), meaning an SGI signalled through the GIC

How does Linux do that? The documentation-S don't give any other details regarding "It will provide a startup address to the secondary cores".

My frustration is growing and I'd be very grateful for answers. Thank you very much in advance!

EXTRA DETAILS

Documentation I use:

• ARMv7-A&R Architecture Reference Manual
• Cortex-A15 TRM (Technical Reference Manual)
• Cortex-A15 MPCore TRM
• Cortex-A Series Programmer's Guide v3.0
• GICv2 Architecture Specification

What I've done by now:

• UBOOT loads me at 0x40008000; I've set-up Translation Tables (TTBs), written TTBR0 and TTBCR accordingly and mapped 0x40008000 to 0x8000_0000 (2GB), so I also enabled the MMU
• Set-up exception handlers of my own
• I've got Printf functionality over the serial (UART2 on ODROID-XU)

All the above seems to work properly.

What I'm trying to do now:

• Get the GIC base address => at the moment I read CBAR and I simply AND (&) its value with 0xFFFF8000 and use this as the GIC base address, although I'm almost sure this ain't right
• Enable the GIC distributor (at offset 0x1000 from GIC base address?), by writting GICD_CTLR with the value 0x1
• Construct an SGI with the following params: Group = 0, ID = 0, TargetListFilter = "All CPUs Except Me" and send it (write it) through the GICD_SGIR GIC register
• Since I haven't passed any execution start address for the other cores, nothing happens after all this

....UPDATE....

I've started looking at the Linux kernel and QEMU source codes in search for an answer. Here's what I found out (please correct me if I'm wrong):

• When powering up the board ALL THE CORES start executing from the reset vector
• A software (firmware) component executes WFI on the secondary cores and some other code that will act as a protocol between these secondary cores and the primary core, when the latter wants to wake them up again
• For example, the protocol used on the EnergyCore ECX-1000 (Highbank) board is as follows:

**(1)** the secondary cores enter WFI and when

**(2)** the primary core sends an SGI to wake them up

**(3)** they check if the value at address (0x40 + 0x10 * coreid) is non-null;

**(4)** if it is non-null, they use it as an address to jump to (execute a BX)

**(5)** otherwise, they re-enter standby state, by re-executing WFI

**(6)** So, if I had an EnergyCore ECX-1000 board, I should write (0x40 + 0x10 * coreid) with the address I want each of the cores to jump to and send an SGI

Questions:

• 1. What is the software component that does this? Is it the BL1 binary I've written on the SD Card, or is it U-BOOT?
• 2. From what I understand, this software protocol differs from board to board. Is it so, or does it only depend on the underlying processor?
• 3. Where can I find information about this protocol for a pick-one ARM board? - can I find it on the official ARM website or on the board webpage?

Answer 1

Ok, I'm back baby. Here are the conclusions:

• The software component that puts the CPUs to sleep is the bootloader (in my case U-Boot)
• Linux somehow knows how the bootloader does this (hardcoded in the Linux kernel for each board) and knows how to wake them up again

For my ODROID-XU board the sources describing this process are UBOOT ODROID-v2012.07 and the linux kernel found here: LINUX ODROIDXU-3.4.y (it would have been better if I looked into kernel version from the branch odroid-3.12.y since the former doesn't start all of the 8 processors, just 4 of them but the latter does).

Anyway, here's the source code I've come up with, I'll post the relevant source files from the above source code trees that helped me writing this code afterwards:

typedef unsigned int DWORD;
typedef unsigned char BOOLEAN;
#define FAILURE (0)
#define SUCCESS (1)
#define NR_EXTRA_CPUS (3) // actually 7, but this kernel version can't wake them up all -> check kernel version 3.12 if you need this

// Hardcoded in the kernel and in U-Boot; here I've put the physical addresses for ease
// In my code (and in the linux kernel) these addresses are actually virtual
// (thus the 'VA' part in S5P_VA_...); note: mapped with memory type DEVICE
#define S5P_VA_CHIPID (0x10000000)
#define S5P_VA_SYSRAM_NS (0x02073000)
#define S5P_VA_PMU (0x10040000)
#define EXYNOS_SWRESET ((DWORD) S5P_VA_PMU + 0x0400)
// Other hardcoded values
#define EXYNOS5410_REV_1_0 (0x10)
#define EXYNOS_CORE_LOCAL_PWR_EN (0x3)

BOOLEAN BootAllSecondaryCPUs(void* CPUExecutionAddress){

// 1. Get bootBase (the address where we need to write the address where the woken CPUs will jump to)
//    and powerBase (we also need to power up the cpus before waking them up (?))
DWORD bootBase, powerBase, powerOffset, clusterID;

asm volatile ("mrc p15, 0, %0, c0, c0, 5" : "=r" (clusterID));
clusterID = (clusterID >> 8);
powerOffset = 0;
if( (*(DWORD*)S5P_VA_CHIPID & 0xFF) < EXYNOS5410_REV_1_0 )
{
    if( (clusterID & 0x1) == 0 ) powerOffset = 4;
}
else if( (clusterID & 0x1) != 0 ) powerOffset = 4;

bootBase = S5P_VA_SYSRAM_NS + 0x1C;
powerBase = (S5P_VA_PMU + 0x2000) + (powerOffset * 0x80);

// 2. Power up each CPU, write bootBase and send a SEV (they are in WFE [wait-for-event] standby state)
for (i = 1; i <= NR_EXTRA_CPUS; i++)
{
    // 2.1 Power up this CPU
    powerBase += 0x80;
    DWORD powerStatus = *(DWORD*)( (DWORD) powerBase + 0x4);

    if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) == 0)
    {
        *(DWORD*) powerBase = EXYNOS_CORE_LOCAL_PWR_EN;
        for (i = 0; i < 10; i++) // 10 millis timeout
        {
            powerStatus = *(DWORD*)((DWORD) powerBase + 0x4);
            if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) == EXYNOS_CORE_LOCAL_PWR_EN)
                break;
            DelayMilliseconds(1); // not implemented here, if you need this, post a comment request 
        }
        if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) != EXYNOS_CORE_LOCAL_PWR_EN)
            return FAILURE;
    }
    if ( (clusterID & 0x0F) != 0 )
    {
        if ( *(DWORD*)(S5P_VA_PMU + 0x0908) == 0 )
        do { DelayMicroseconds(10); } // not implemented here, if you need this, post a comment request
        while (*(DWORD*)(S5P_VA_PMU + 0x0908) == 0);
        *(DWORD*) EXYNOS_SWRESET = (DWORD)(((1 << 20) | (1 << 8)) << i);
    }

    // 2.2 Write bootBase and execute a SEV to finally wake up the CPUs
    asm volatile ("dmb" : : : "memory");
    *(DWORD*) bootBase = (DWORD) CPUExecutionAddress;
    asm volatile ("isb");
    asm volatile ("
   dsb
   sev
   nop
");
}
return SUCCESS;
}

This successfully wakes 3 of 7 of the secondary CPUs.

And now for that short list of relevant source files in u-boot and the linux kernel:

• UBOOT: lowlevel_init.S - notice lines 363-369, how the secondary CPUs wait in a WFE for the value at _hotplug_addr to be non-zeroed and to jump to it; _hotplug_addr is actually bootBase in the above code; also lines 282-285 tell us that _hotplug_addr is to be relocated at CONFIG_PHY_IRAM_NS_BASE + _hotplug_addr - nscode_base (_hotplug_addr - nscode_base is 0x1C and CONFIG_PHY_IRAM_NS_BASE is 0x02073000, thus the above hardcodings in the linux kernel)

• LINUX KERNEL: generic - smp.c (look at function __cpu_up), platform specific (odroid-xu): platsmp.c (function boot_secondary, called by generic __cpu_up; also look at platform_smp_prepare_cpus [at the bottom] => that's the function that actually sets the boot base and power base values)