# $Source: /usr/local/kcs/sys.DAVIS_800/debug/RCS/ReadMe,v $
# $Revision: 1.5.98.1 $	$Author: root $
# $State: Exp $   	$Locker:  $
# $Date: 95/07/20 10:21:47 $

		      Symbolic Crashdump Analysis
				  With
		       The 11th Hour Kernel Tools

	      Getting The Stuff You Care About Supported:
			    A "How-To" Guide

Introduction

  11th Hour Kernel Tools is a collection of symbolic kernel crashdump
  debugging tools designed to supplement adb and replace QDB and
  analyze.  The debugger referred to in this document is called "q4" and
  will be/was the first of the 11th Hour tools to be rolled off the
  assembly line.

  Release 10.0 is the first release to support and be supported by the
  11th Hour tools.  All PA-RISC-based HP-UX systems running 10.0 or
  later are supported.  Some 10.0 kernels created prior to release are
  supported as well.  It all depends on how early the kernel was
  created.  (Kernels created following the instructions in this document
  are supported.  Those that don't are not.)

Background

  Breaking with the past, these tools needn't be built from a specific
  set of kernel headers, so they don't suffer from the restriction of
  only working properly on the specific kernel for which they were
  compiled.

  Instead, a description of the kernel data structures of interest is
  compiled and linked into the kernel and examined at crash analysis
  time by the tools.  This information is stored as standard HP symbolic
  debug info.

  Because symbolic debug is still somewhat incompatible with the
  optimizer and can also be quite large, the normal kernel source files
  aren't compiled with the "-g" option.  Instead a special set of files
  containing only header #include's are compiled with symbolic debug
  turned on.  This gives the debugger the information it needs without
  sacrificing code optimization and without adding too much baggage to
  the kernel.  (This baggage takes up space in the vmunix file, but is
  not loaded into memory.)

What You Need

  Both analyze and QDB work by having intimate knowledge of the workings
  of the kernel compiled into them.  Since the 11th Hour tools don't do
  this, the information must be somewhere else, and that somewhere is
  the kernel, specifically, the vmunix file.

  A consequence of this is that analyze and QDB can be modified to debug
  new data structures even after the kernel sources have been frozen and
  shipped.  The downside is that analyze and QDB aren't portable.  They
  have to be built for a specific kernel.

  An advantage of the 11th Hour tools is that they are portable so one
  version will cover all 10.0 kernels (and probably later version as
  well).  The cost is that someone (read:  you) has to tell the tools
  which data types you care about.  Since this info is compiled and
  linked into the kernel, the tools can't be modified after kernel
  code-freeze or release to add new data types.

  MORAL:  Take the time to add the types you care about, or *might* care
	  about while you still can.

What You Do

  1.  There is a new directory in the kernel source tree.  It is called
      "debug".  It contains only this file (called ReadMe) and a
      collection of ".c" files.  Each functional area of the kernel
      should have one or more ".c" files here.  For example,
      "proc_man.c" for process management, "virt_mem.c" for virtual
      memory, etc....

      Make sure there is a file here for your functional part of the
      kernel (see knewfile(KCS)).

  2.  Add to conf_conv/files a line describing your new file.  For
      example:

	debug/proc_man.c	hp-ux debug
	debug/my_subsys.c	hp-ux debug +ansi

      Note the new keyword "debug".  This is required and is reserved for
      these files only.  The entry must go into libhp-ux.a.  You will
      find these files grouped together toward the beginning of "files".  
      For the convenience of all concerned, please keep them together.

  3.  Each of these files should be made up of #include's and, at your
      discretion, comments.  Don't put kernel code in any of them.  It
      will only make the vmunix file bigger.  It will not help the
      tools.

      Include in your file any and all kernel header files that define
      data types you might want to look at in a crashdump.  If you
      aren't sure whether or not you want to add a certain file, add it
      anyway.  (Remember that once the kernel is frozen you won't be
      able to add support for any types you missed.)

      Once a header file is included in one of these functional areas
      (".c" files) it may be included in others, but it isn't necessary.
      Thus, once the process management project has included proc.h, you
      don't need to add it to your file to be able to debug proc's.
      Conversely, multiple include are okay (though they do make the
      debug area bigger without any offsetting benefit).

  4.  Structure and union fields whose contents are #define'd constants
      or bit masks can't be displayed symbolically.  (#define'd
      constants don't end up in the symbolic debug info because cpp eats
      them before the C compiler gets to see them.)

      If you want the tools to be able to display these fields
      symbolically, you have to change them to use enumerated types.

      For example, the process status field in the proc structure was
      (and may still be) declared like this:

	struct proc {
		...
		char	p_stat;	/* current process state (see below) */
		...
	};
	...
	#define	SSLEEP	1 /* awaiting an event */
	#define	SWAIT	2 /* (abandoned state) */
	#define	SRUN	3 /* running */
	#define	SIDL	4 /* intermediate state in process creation */
	#define	SZOMB	5 /* intermediate state in process termination */
	#define	SSTOP	6 /* process being traced */

      Changing it to this will make it possible to see p_stat in
      symbolic form:

	typedef char enum proc_state {
		SSLEEP = 1,	/* awaiting an event */
		SWAIT = 2,	/* (abandoned state) */
		SRUN = 3,	/* running */
		SIDL = 4,	/* intermediate state in process creation */
		SZOMB = 5,	/* intermediate state in process termination */
		SSTOP = 6	/* process being traced */
	} proc_state_t;
	...
	struct proc {
		...
		proc_state_t p_stat;	/* current process state (see above) */
		...
	};

      Notice three important things:  (1) enumerants can be given values
      *you* choose, (2) you can create enumerated types in non-word
      sizes and, (3) the changes described here are made to the
      declarations only -- you don't have to go over the kernel changing
      all the references to these value.

      With these features, it should be possible to convert nearly any
      set of #define's to an enumerated type without having to make
      changes that will affect backward compatibility.

      The only time this gets more complicated is when the #define'd
      constants have been customer-visible, where the customer may have
      written code like this:

	#ifdef SIGDIL
	...
	#endif

      This won't work if SIGDIL is changed to an enumerant because cpp
      handles #ifdef's and doesn't speak C.  Here's one way to get
      around this one:

	#define	SIGDIL	32		/* same as before ... */
	...
	typedef char enum signum {
	...
		sigdil = SIGDIL,
	...
	} signum_t;
	...
	struct proc {
		...
		signum_t p_cursig;	/* number of current pending signal */
		...
	};
