crt0.s suggestion

The various invocations of crt0.s should probably build proper stack frames rather than just branching to main so debuggers can know when they reach the end of a stack trace for a PD. Cosmetic, non-breaking, and unimportant, just a 'would be nice to have some day' for those of us with OCD.

What is involved with proper stack frames? My understanding was that all you needed was the link register setup (for AArch64), but that's just set to 0 at entrypoint. This seems to work fine in GDB for us. Although I do recall warnings about something related but I didn't see what could be done to solve this.

I should have said this doesn't happen all the time only in the right circumstances. Currently, crt0.s for aarch64 looks like this:

.global _start;
.type _start, %function;
_start:
    b main

Sometimes the GDB stack trace heuristic asks the question "who called _start?" but because _start has no call frame here it will get resolved as '???' (as I said it doesn't break anything, it's purely cosmetic). But that's why generally you will see crt0.s coded as:

.global _start;
.type _start, %function;
_start:
    mov x29, #0
    mov x30, #0
    b   main

I'm confused, though. The link register and frame pointer already start at 0, so when GDB walks the stack it sees zero. Why would this change it?

Ah, you are right and I hadn't thought about the fact the kernel is zeroing the registers when I wrote this. I further thought a NULL value return address is what causes gdb to stop the unwind but that's really (and I mean REALLY) dated information. Instead, hitting the symbol 'main' is what causes it to stop unwinding now but ONLY if that symbol is defined in the primary executable at startup. We load all the PD symbol tables via 'add-symbol-table' command so gdb just keeps trying to unwind for the PDs, stopping one frame up from main because it detects identical stack frames. The only way to fix it is to use the .cfi_startproc/.cfi_endproc/.cfi_undefined instructions to crt0.s and it's more trouble than it's worth.

Isn't that easily fixed by doing bl instead of b? If there is a valid stack at this point there is no reason not to do bl.

(This is also what I thought you meant originally.)

You guys actually do that for the x86 version, it does a call for main. The arm and risc versions both use branches. I would agree with you the bl is the better way but our problem stems from the way gdb works internally and is not a call frame issue. To fix the problem we are seeing requires a bit more, e.g., something like:

.global _start;
.type _start, %function;
.cfi_startproc
.cfi_undefined x30
_start:
    bl main
1:  b 1b
.cfi_endproc

P.S. I would also argue a more correct way is not to rely on the kernel providing zero values and crt0.s should be loading explicitly the values that are 'correct' for the machine it is running on.

Have you tried it? Because I think it will fix the problem of main not showing in the backtrace. _start having '???' is perfectly fine if there is a main before it.

P.S. I would also argue a more correct way is not to rely on the kernel providing zero values and crt0.s should be loading explicitly the values that are 'correct' for the machine it is running on.

It is the user space which launches the new process that determines the values, and that code cooperates with the code in _start. There is some ABI that is being followed, not relying on a random small part of it seems silly.

Have you tried it? Because I think it will fix the problem of main not showing in the backtrace. _start having '???' is perfectly fine if there is a main before it.

Main does show in the backtrace with branch as is, e.g.,

(gdb) pd break sensor::notified
Breakpoint 1 at 0x6200408: file sensor.c, line 33.
[pd break] sensor::notified at 0x6200408
(gdb) cont
Continuing.
[Switching to Thread 4]

Thread 4 "sensor" hit Breakpoint 1, notified (ch=0) at sensor.c:33
33	{

[sensor] Breakpoint 1 at notified (sensor.c:33)
(gdb) bt
#0  notified (ch=0) at sensor.c:33
#1  0x0000000006200734 in handler_loop () at src/main.c:194
#2  main () at src/main.c:220
#3  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

The issue is that gdb does not understand it needs to stop unwinding at main so it actually continues and tries to show the previous frame to main and reports a potential problem. Again, this is completely cosmetic and acceptable as is and I never would have brought it up except with our vscode extension that uses cppdebug, it actually causes it a bigger problem that sometimes causes a crash (but mostly not). We will just fix the extension to be more tolerant.

P.S. I would also argue a more correct way is not to rely on the kernel providing zero values and crt0.s should be loading explicitly the values that are 'correct' for the machine it is running on.

It is the user space which launches the new process that determines the values, and that code cooperates with the code in _start. There is some ABI that is being followed, not relying on a random small part of it seems silly.

Ah right, I forgot there was code in between.

Have you tried it? Because I think it will fix the problem of main not showing in the backtrace. _start having '???' is perfectly fine if there is a main before it.

Main does show in the backtrace with branch as is, e.g.,

What does it show with bl?

Have you tried it? Because I think it will fix the problem of main not showing in the backtrace. _start having '???' is perfectly fine if there is a main before it.

Main does show in the backtrace with branch as is, e.g.,

What does it show with bl?

Can I ask what you are thinking here? I'm obviously missing something. I haven't done it but I would anticipate it gets a real, correctly-named _start frame which, again, I would certainly agree is better and more accurate than the current b instruction alone. But with no .cfi_undefined, gdb will still try using x30/LR to look back another frame and the same problem occurs. So I guess my question would be, if you are going to make a change anyway (b -> bl) why not go the whole nine yards and also produce the DWARF CFI?

I expect an output similar to:

(gdb) bt
#0  notified (ch=0) at sensor.c:33
#1  0x0000000006200734 in handler_loop () at src/main.c:194
#2  main () at src/main.c:220
#3 _start () at crt0.s...
#4  0x0000000000000000 in ?? () (maybe)

Notably, no corrupt stack warning.

The reason I expect the warning to be gone is because I think GCC assumed that main was called like a function, while it wasn't. So the warning was correct to say that the "stack" may be corrupted (if you consider LR as part of the stack). My expectation is that GCC is smart enough to know that _start isn't a normal function but the program entry point and to stop the backtrace there without warning.

However, I'm not sure about any of this, but was curious what GDB would show for you.

So I guess my question would be, if you are going to make a change anyway (b -> bl) why not go the whole nine yards and also produce the DWARF CFI?

Well, in this case having a _start that does nothing is silly, it could just have made main the entry point directly. I don't know why Microkit does it like this. As for the dwarf CFI, it would be good to know whether it's actually needed, don't you think? Also, I don't think your proposed solution is long-term robust, what you want is a way to tell the compiler to stop unwinding when it reaches the entry point. But that info is there already.

FWIW: the same thing happens for seL4 debugging as well. I've looked into this before and I have a hard time finding a definitive reference on how this is supposed to work.

GDB is supposed to have an option that doesn't unwind past main and that is supposed to be turned off but that doesn't seem to apply here. (of course it would be nice to be correct if it did want to)

(gdb) pd break sensor::notified
Breakpoint 1 at 0x6200408: file sensor.c, line 33.
[pd break] sensor::notified at 0x6200408
(gdb) cont
Continuing.
[Switching to Thread 4]

Thread 4 "sensor" hit Breakpoint 1, notified (ch=0) at sensor.c:33
33	{

[sensor] Breakpoint 1 at notified > (sensor.c:33)
(gdb) bt
#0  notified (ch=0) at sensor.c:33
#1  0x0000000006200734 in handler_loop () at src/main.c:194
#2  main () at src/main.c:220
#3  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

BTW, I'm intrigued. Is this a custom GDB integration that somehow knows about seL4 threads? Or is this a gdb server running as a microkit PD? (using the PR we have in Microkit that and work surrounding it?). I'm intrigued by this pd break command you've added, and that it knows about inferiors.

(or is this a modification to openOCD or the like?)

GDB is supposed to have an option that doesn't unwind past main and that is supposed to be turned off but that doesn't seem to apply here.

There's set backtrace past-entry off and set backtrace past-main off.

BTW, I'm intrigued. Is this a custom GDB integration that somehow knows about seL4 threads? Or is this a gdb server running as a microkit PD? (using the PR we have in Microkit that and work surrounding it?). I'm intrigued by this pd break command you've added, and that it knows about inferiors.

(or is this a modification to openOCD or the like?)

I was assuming it was openOCD with custom gdb scripts.

But considering #550, my guess is that they combined all binaries together for easier debugging and the info of main and entry points is lost maybe.

openOCD is much too slow of a development path for what we want to do. This is part of a larger suite of tools we've built/are building for PD development.

How much I can say right now is limited but I will confirm it is a gdb RSP PD, using gdb (and gdb scripts) and vscode as the host tools. It's extremely likely that everything will eventually be released as open source.

Nice. Is this based on some of our (TS's) libgdb work or something from scratch?

I expect an output similar to:
(gdb) bt
#0  notified (ch=0) at sensor.c:33
#1  0x0000000006200734 in handler_loop () at src/main.c:194
#2  main () at src/main.c:220
#3 _start () at crt0.s...
#4  0x0000000000000000 in ?? () (maybe)
Notably, no corrupt stack warning.

The reason I expect the warning to be gone is because I think GCC assumed that main was called like a function, while it wasn't. So the warning was correct to say that the "stack" may be corrupted (if you consider LR as part of the stack). My expectation is that GCC is smart enough to know that _start isn't a normal function but the program entry point and to stop the backtrace there without warning.

However, I'm not sure about any of this, but was curious what GDB would show for you.

Well, we don't have to make a modification to see this since the x86 version already does it. Here is the parallel output for the x86 version:

_text () at src/x86_64/crt0.s:10
10	    call    main

[pinger] stopped at _text (crt0.s:10)
(gdb) pd break sensor::notified
Breakpoint 1 at 0x620035b: file sensor.c, line 33.
[pd break] sensor::notified at 0x620035b
(gdb) c
Continuing.
[Switching to Thread 4]

Thread 4 "sensor" hit Breakpoint 1, notified (ch=0) at sensor.c:33
33	{

[sensor] Breakpoint 1 at notified (sensor.c:33)
(gdb) bt
#0  notified (ch=0) at sensor.c:33
#1  0x0000000006200763 in handler_loop () at src/main.c:194
#2  main () at src/main.c:220
#3  0x0000000006200005 in _text () at src/x86_64/crt0.s:10
Backtrace stopped: Cannot access memory at address 0x7fffffffd000

So I guess my question would be, if you are going to make a change anyway (b -> bl) why not go the whole nine yards and also produce the DWARF CFI?

Well, in this case having a _start that does nothing is silly, it could just have made main the entry point directly. I don't know why Microkit does it like this. As for the dwarf CFI, it would be good to know whether it's actually needed, don't you think? Also, I don't think your proposed solution is long-term robust, what you want is a way to tell the compiler to stop unwinding when it reaches the entry point. But that info is there already.

There are a few things I'd say in response. The first is that solving the problem for gdb is not solving the problem. Supporting DWARF CFI supports any debugger that adheres the DWARF standard (basically all of them) and is the reason it even exists. Second is that it is in virtually all the code emitted by the compiler so why the reluctance to add it to one little bit of code that calls all the other code? Third is that I don't know enough about microkit internals to know how to set this up in crt0.s so it would work without DWARF CFI so the solution I proposed was just one that is simple, I know will work for what the issue is, will require very little modification on a per-architecture basis, and doesn't require me to know more than I already do. This is why I didn't actually propose a solution in the first place. It was to encourage you all to do whatever you think is best with your vaster microkit internals knowledge if you wanted to do anything at all. Finally, using _start is a nearly universal pattern. Kernel puts what it wants to communicate on the stack, _start sets up any requirements for run time and does whatever it wants, including calling the main code of whatever language is being used. I haven't a clue as to where microkit development plans to go, but if it is into supporting other languages, I would think it is a pattern you might want to keep.

Nice. Is this based on some of our (TS's) libgdb work or something from scratch?

From scratch.

I'm not personally opposed to adding necessary directives to the startup sequence to make this work.

My concern is that I don't understand why these directives make the debugger work here; is there some standard prose that says debuggers should behave like this (why does marking _start as a function make it stop the stack unwinding)? Otherwise this feels like something that works just for GDB.

Well, we don't have to make a modification to see this since the x86 version already does it.

x86 is missing ".type _start, %function;" though. There's ENTRY(_start); in the linker script of course, but I don't know whether that's enough for GDB (especially if you don't feed it the original ELF file).

I'm not personally opposed to adding necessary directives to the startup sequence to make this work.

My concern is that I don't understand why these directives make the debugger work here; is there some standard prose that says debuggers should behave like this (why does marking _start as a function make it stop the stack unwinding)? Otherwise this feels like something that works just for GDB.

Marking _start as function doesn't have anything at all to do with the stack unwinding. It serves a different purpose (but should still be there). The stack unwinding is guided by the .cfi instructions. If the -g flag is present when the assembler is invoked, then these .cfi instructions (and all the other debug related instructions in the source assembly file) are processed by the assembler and put into the debug related sections of the generated elf obj. [If -g is not specified they are ignored.] Collectively these debug sections and their specific formats are defined by the DWARF standard (https://dwarfstd.org/doc/Debugging%20using%20DWARF-2012.pdf).

Crucial to the discussion here: if a debugger adheres to the DWARF standard and the debug sections exist in the obj file, they are considered authoritative and will contain everything necessary for the debugger to do the unwinding perfectly every time, even if you, for example, selected the -fomit-frame-pointer flag. If they don't exist then gdb will attempt to use it's own examination of the code, registers, and stacks to do the unwind.

So, again to be clear, my suggestion above will only work when the obj is built using the -g flag, so it's not a panacea. But on the other hand, there is no harm in having the .cfi instructions either as they are simply ignored if -g is not specified.

Well, we don't have to make a modification to see this since the x86 version already does it.

x86 is missing ".type _start, %function;" though. There's ENTRY(_start); in the linker script of course, but I don't know whether that's enough for GDB (especially if you don't feed it the original ELF file).

[sensor] Breakpoint 1 at notified (sensor.c:33)
(gdb) bt
#0  notified (ch=0) at sensor.c:33
#1  0x0000000006200763 in handler_loop () at src/main.c:194
#2  main () at src/main.c:220
#3  0x0000000006200005 in _text () at src/x86_64/crt0.s:10
Backtrace stopped: Cannot access memory at address 0x7fffffffd000

The missing "type _start, %function;" in x86_64/crt0.s is the reason that frame 3 shows ..in _text () at... instead of ...in _start () at.... Haven't tested it but 95% certain.

All very well, and no objection to adding CFI hints, but the DWARF standard doesn't give a well defined way of saying that there is no way to return from a function. Looking around, it does seem to be the convention to use .cfi_undefined on the return address register, for archs that have one. So with that in mind, your solution of using .cfi_startproc + .cfi_undefined x30 seems acceptable.

But it's not anything obvious if you look at DWARF, It's just what works for current compilers. Note that using .cfi_undefined on unexpected registers has crashed debuggers in the past, so it's not obviously safe either. And the original problem was that the debugger couldn't continue the backtrace and it complained about that. Marking LR as undefined has the same effects, so it is not obvious that compilers won't give a similar complaint in the future.

E.g. there is also .cfi_return_column, which probably inserts a DW_AT_return_address, if that could be specified to "undefined" directly, then that would be more direct than fiddling with arch-specific registers or return address stack locations. (The link between assembler CFI directives and the corresponding DWARF info is also a bit unclear sometimes.)

As part of our efforts to move to 2.3.0, we added this change to x86 and aarch64 during the microkit build, and it works as expected, e.g.,

_start () at src/aarch64/crt0.s:22
22	    bl main

[pinger] stopped at _start (crt0.s:22)
(gdb) pd br sensor::notified
Breakpoint 1 at 0x620040c: file sensor.c, line 33.
[pd break] sensor::notified at 0x620040c
(gdb) c
Continuing.
[Switching to Thread 4]

Thread 4 "sensor" hit Breakpoint 1, notified (ch=0) at sensor.c:33
33	{

[sensor] Breakpoint 1 at notified (sensor.c:33)
(gdb) bt
#0  notified (ch=0) at sensor.c:33
#1  0x0000000006200763 in handler_loop () at src/main.c:194
#2  main () at src/main.c:220
#3  0x0000000006200005 in _start () at src/x86_64/crt0.s:21
(gdb) up
#1  0x0000000006200744 in handler_loop () at src/main.c:194
194	                    notified(idx);
(gdb) 
#2  main () at src/main.c:220
220	    handler_loop();
(gdb) 
#3  0x0000000006200004 in _start () at src/aarch64/crt0.s:22
22	    bl main
(gdb) 
❌️ Initial frame selected; you cannot go up.
(gdb) down
#2  main () at src/main.c:220
220	    handler_loop();
(gdb) 
#1  0x0000000006200744 in handler_loop () at src/main.c:194
194	                    notified(idx);
(gdb) 
#0  notified (ch=0) at sensor.c:33
33	{
(gdb) 
❌️ Bottom (innermost) frame selected; you cannot go down.

with similar results on x86.

The one thing we changed from the suggestion I made above was to force a fault in the case of an accidental return from main(). The idea of an infinite loop silently eating up time slices was just a bit too much to bear. E.g.,

.global _start;
.type _start, %function;
_start:
    .cfi_startproc
    .cfi_undefined x30
    bl main
    /* main() should never return; if it does, trap with an illegal instruction */
    udf #0
    .cfi_endproc

Navigation Menu

Repository navigation

crt0.s suggestion #554

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

crt0.s suggestion #554

Description

Activity

midnightveil commented on Jul 17, 2026

joep commented on Jul 17, 2026

midnightveil commented on Jul 17, 2026

joep commented on Jul 17, 2026

Indanz commented on Jul 17, 2026

joep commented on Jul 17, 2026

joep commented on Jul 17, 2026

Indanz commented on Jul 17, 2026

Indanz commented on Jul 17, 2026

joep commented on Jul 17, 2026

joep commented on Jul 17, 2026

Indanz commented on Jul 17, 2026

joep commented on Jul 17, 2026

Indanz commented on Jul 17, 2026

midnightveil commented on Jul 18, 2026

midnightveil commented on Jul 18, 2026

Indanz commented on Jul 18, 2026

Indanz commented on Jul 18, 2026

joep commented on Jul 18, 2026

midnightveil commented on Jul 18, 2026

joep commented on Jul 18, 2026

joep commented on Jul 18, 2026

midnightveil commented on Jul 20, 2026

Indanz commented on Jul 20, 2026

joep commented on Jul 21, 2026

joep commented on Jul 21, 2026

Indanz commented on Jul 21, 2026

joep commented on Jul 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions