Subject: union filesystem [was Re: debugging kernel core dumps ?]

From: der Mouse <[email-masked]>

> Having compile a NetBD-current/sparc kernel with the DEBUG and
> DIAGNOSTIC options in, how do I then debug the core dumps ?

Good question.  I would have used adb -k, but adb doesn't exist under
NetBSD/sparc (which is a pain for more reasons than just kernel
coredumps!).

> gdb won't recognise vmcore,

Did you try gdb -k?  Just a guess by analogy to adb -k....

> the FAQ (3.2.4) mentions "device-pseudo ddb" (adding this just makes
> config.new barf) and CTL-ALT-ESC does nothing.

Add "options DDB" (which is probably what the "pseudo-device ddb" is
trying to provoke).  Then when you L1-A or get a panic, it breaks to a
debugger.

Unfortunately it's quite rudimentary.  The disassembler is nonexistent
and breakpoints simply do not work as far as I can tell, for example.
You also have to jump through a few hoops to get symbols in SPARC ddb.
Send me mail and I'll regale you with the full story.

> The man page for ddb(4) says nothing useful...I'm lost, where should
> I be looking ?

> If I add "options KGDB", I presume I need to add the next line
> defining KGDBDEV and KGDBRATE, correct ?

I think so, but I don't have kgdb working, and the SPARC port people
tell me it doesn't work.

> If so, does this limit debugging to using an active kernel and being
> at the console ? (yuck!)

What little I understand of kgdb indicates that you need another
machine, connected to a serial port on the machine being debugged.

> Also, who is currently working on the union file system ?  I'm after
> a way to debug these:  [...]

I'd like to know who's doing the union fs too, but for a different
reason - I want to know why I can't seem to layer the union fs on top
of the fdesc fs.  I do

	mount -t fdesc fdesc /fdesc
	mount -t union -o -b /fdesc /dev

and everything works...except that under some circumstances I get a
panic in namei().  In particular, this happens repeatably if I boot
single-user, do those two mounts, and then "mount -u /".  One stack
frame is missing from the traceback; it appears namei is trying to look
up /dev/sd0a and panicking.  I've tried various things with no success,
largely because ddb is crippled at best (eg, when I set a breakpoint at
_ffs_mount I get an illegal instruction trap from an address which
corresponds to the beginning of some other _ffs_* routine); I'm about
to scatter printfs through ffs_mount and/or namei in the hope of
getting some idea what's going wrong.  (Yes, I know about "mount -t
fdesc -o union fdesc /dev"; that is not what I want, for various
reasons.)

Once, I got "panic: vref used where vget needed" (or perhaps I have
vref and vget switched) when trying to layer union over fdesc this way.
I was unable to repeat it (the message is from memory), but it may
point someone in a useful direction.

> Also, can anyone tell me what this means ?

> Sep  4 06:43:54 netbsd /netbsd: note: lost 32 pages in translation

It has something to do with setting up memory based on the data
structures set up by the ROM monitor, I think.  I noticed it too, and
went looking (grep through the kernel source)...and it looked both not
very serious and mostly unavoidable, so I paid it no more mind.

					der Mouse

			    [email-masked]

---

From: Dave Burgess <[email-masked]>

> Did you try gdb -k?  Just a guess by analogy to adb -k....

If this works, I will add it to the FAQ.  If not, then what would the
correct incantation to make it go be?

> Send me mail and I'll regale you with the full story.

I will be sure to update this in the FAQ to prevent this aggravating
anyone else.

> machine, connected to a serial port on the machine being debugged.

I would like to add a one-liner in the kernel 'mucking-about' section
about the debugger; unfortunately, I'm not sure where to look for a good
description of using the one in NetBSD.  Pointers to the pointers would
be good enough to at least let me put that into the FAQ.

---

From: der Mouse <[email-masked]>

> If this works, I will add it to the FAQ.

I have since tried it, and it works fine.  ("gdb -k netbsd.0 vmcore.0",
for example.)

> about the debugger; unfortunately, I'm not sure where to look for a
> good description of using the one in NetBSD.  Pointers to the
> pointers would be good enough to at least let me put that into the
> FAQ.

I've found DDB to be so crippled it's faster/easier to tell it "c", let
it take a kernel coredump, reboot and savecore, and use gdb -k on the
coredump.  Everything a run-time debugger can give me that a
post-mortem debugger can't is dependent on working breakpoints, which
as far as I can tell do not exist in SPARC DDB.  (I haven't
investigated what's wrong with them.)

					der Mouse

---

From: der Mouse <[email-masked]>

> The way you're expected to mount fdesc is:

> mount -t fdesc -o union fdesc /dev

That is unsatisfactory, largely because /dev then becomes effectively
read-only - one has to unmount the fdesc layer to do anything, like
creating new device special files.

Hmm.  If I use nullfs to clone /dev somewhere else, _before_ I mount
fdesc -o union, perhaps that would let me modify /dev underneath the
union fdesc layer...and it would fix problems caused by root doing
something like "command producing lots of output > /dev/nulll" and the
like.  I'll have to try that - maybe this is a feature rather than a
bug.

But still, it seems to me that what I tried _ought_ to work.  (I
actually wanted -t union -o -b mounts for another purpose, and it
occurred to me to use it for /dev as well.)

> Regarding union fs in general, it's somewhat unreliable, but should
> be fixed when we (I?) import a new version, some time after the
> release.  (The changes are just too pervasive to do now.)

Oh, okay.  I'll wait, then.

					der Mouse

---

From: der Mouse <[email-masked]>

der Mouse