Discussion:
Segmented stack debug hack
(too old to reply)
Daniel Morsing
2013-11-09 14:01:02 UTC
Permalink
Hi,

I'm working on a hack that allows the DWARF3 call frame information to
describe segmented stacks. The goal is to have debuggers and profilers
be able to unwind over the stack segments, giving better stack traces
for tools that aren't Go aware. Doing post-mortem analysis on core
dumps is one of the problems I want to address.

This hack has the unfortunate properties of being large enough that I
can't land it for go1.2 and will be made irrelevant once Keiths
contiguous stack work lands for go1.3. I'm making this available as a
patch mostly for critique and for adventurous users to apply in the
window between 1.2 and 1.3.

Patch available at https://codereview.appspot.com/23930043 .

I have this working on amd64 with some machine specific code, Don't
even think about running this with 386 or arm yet. It's not going to
work. I have tested it with gdb and it works some of the time, but
there are still some problems.

gdb complains when a calculated stack pointer is less than the current
stack frame and stops the backtrace. It does this based on what it
knows about the amd64 ABI which doesn't include segmented stacks. I
think I can fix this by putting an augmentation on the CIE, but I'm
not sure which one. Any help from DWARF3 savvy developers is
appreciated.

Somewhat related to this, I took a look at the linux kernel perf tool
while researching tools that might benefit from this hack. Perf has
previously been unable to decode any of the debug symbols in Go
binaries. Because of the way perf works, it's impossible for it to
traverse a stack link, but I did manage to set it up so that it can
decode symbols from the instruction pointer and traverse the top stack
segment.

Apply this https://lkml.org/lkml/2013/9/26/267 and rebuild perf. My
DWARF patch is not needed here. Call graph recording can be turned on
with --call-graph=dwarf, but produces rather large perf dumps.

Implementation detail:
This hack was achieved by changing the fixed offset information in the
FDE entries to be DWARF expressions.The bytecode is roughly equivalent
to the following pseudocode:

spip = $sp + cfa - ptrsize
if *spip != runtime.lessstack {
return $sp + cfa
}
return *(spip + argsize + stktop_gobuf_sp_off)

For functions marked as nosplit, the old stack traversal method is used.

Regards,
Daniel Morsing
--
---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Ian Lance Taylor
2013-11-09 14:59:29 UTC
Permalink
Post by Daniel Morsing
gdb complains when a calculated stack pointer is less than the current
stack frame and stops the backtrace. It does this based on what it
knows about the amd64 ABI which doesn't include segmented stacks. I
think I can fix this by putting an augmentation on the CIE, but I'm
not sure which one. Any help from DWARF3 savvy developers is
appreciated.
The idea of an augmentation has been mentioned but as far as I know it
has never been implemented. Unfortunately the current code in gdb
looks for the specific symbol __morestack. Set get_prev_frame_1 in
gdb/frame.c.

Ian
--
---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Daniel Morsing
2013-11-09 17:24:41 UTC
Permalink
Post by Ian Lance Taylor
Post by Daniel Morsing
gdb complains when a calculated stack pointer is less than the current
stack frame and stops the backtrace. It does this based on what it
knows about the amd64 ABI which doesn't include segmented stacks. I
think I can fix this by putting an augmentation on the CIE, but I'm
not sure which one. Any help from DWARF3 savvy developers is
appreciated.
The idea of an augmentation has been mentioned but as far as I know it
has never been implemented. Unfortunately the current code in gdb
looks for the specific symbol __morestack. Set get_prev_frame_1 in
gdb/frame.c.
Ian
I'm thinking about appropriating an existing augmentation that would
allow for arbitrary stack gymnastics. Only one I can find is the
signal handler one, and that causes gdb to come to a halt whenever a
function is called.
--
---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Daniel Morsing
2013-11-09 20:39:27 UTC
Permalink
Post by Daniel Morsing
gdb complains when a calculated stack pointer is less than the current
Anyone feeling more intrepid than usual can try applying this patch to
their version of gdb. Note that this turns off a sanity check, so if
you have heap corruption you might end up in an infinite loop.

However, It is really nice to get full stack traces on Go binaries.
--
---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Dmitry Vyukov
2013-11-14 10:02:11 UTC
Permalink
Post by Daniel Morsing
Hi,
I'm working on a hack that allows the DWARF3 call frame information to
describe segmented stacks. The goal is to have debuggers and profilers
be able to unwind over the stack segments, giving better stack traces
for tools that aren't Go aware. Doing post-mortem analysis on core
dumps is one of the problems I want to address.
This hack has the unfortunate properties of being large enough that I
can't land it for go1.2 and will be made irrelevant once Keiths
contiguous stack work lands for go1.3. I'm making this available as a
patch mostly for critique and for adventurous users to apply in the
window between 1.2 and 1.3.
Patch available at https://codereview.appspot.com/23930043 .
I have this working on amd64 with some machine specific code, Don't
even think about running this with 386 or arm yet. It's not going to
work. I have tested it with gdb and it works some of the time, but
there are still some problems.
gdb complains when a calculated stack pointer is less than the current
stack frame and stops the backtrace. It does this based on what it
knows about the amd64 ABI which doesn't include segmented stacks. I
think I can fix this by putting an augmentation on the CIE, but I'm
not sure which one. Any help from DWARF3 savvy developers is
appreciated.
Somewhat related to this, I took a look at the linux kernel perf tool
while researching tools that might benefit from this hack. Perf has
previously been unable to decode any of the debug symbols in Go
binaries. Because of the way perf works, it's impossible for it to
traverse a stack link, but I did manage to set it up so that it can
decode symbols from the instruction pointer and traverse the top stack
segment.
Apply this https://lkml.org/lkml/2013/9/26/267 and rebuild perf. My
DWARF patch is not needed here. Call graph recording can be turned on
with --call-graph=dwarf, but produces rather large perf dumps.
This hack was achieved by changing the fixed offset information in the
FDE entries to be DWARF expressions.The bytecode is roughly equivalent
spip = $sp + cfa - ptrsize
if *spip != runtime.lessstack {
return $sp + cfa
}
return *(spip + argsize + stktop_gobuf_sp_off)
For functions marked as nosplit, the old stack traversal method is used.
Are you sure perf will work with continuous stacks? That would be
awesome. I thought that perf relies on frame pointers, which Go does
not have...
--
---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Daniel Morsing
2013-11-14 10:15:33 UTC
Permalink
Post by Dmitry Vyukov
Are you sure perf will work with continuous stacks? That would be
awesome. I thought that perf relies on frame pointers, which Go does
not have...
Newer versions of perf can use DWARF3 for figuring out the symbols for
return addresses on the stack. Instead of traversing via the FP, perf
will instead grab a snapshot of the memory above the stack pointer and
translate it to symbols using the unwind info when you run report on
it. Unfortunately, it'll still be limited by the amount of memory you
snapshot per sample and the dumps are large enough to quickly fill up
a HDD.
--
---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.
Continue reading on narkive:
Loading...