kolab-cyrus-imapd/debian/README.Debian.debug

Cyrus IMAP for Debian, debugging procedures
$Id: README.Debian.debug 229 2005-12-08 23:26:29Z astronut $
-------------------------------------------

For more information, please consult http://asg.web.cmu.edu/cyrus/imapd/.

Cyrus has various levels of debugging aid, which can and should be used to
offer more information about any problems you are facing with Cyrus.

First, edit /etc/default/cyrus2.2, and set CYRUS_VERBOSE to a number higher
than zero.  The higher the number, more debug information is provided. Numbers
above 30 will cause Cyrus services to pause for 15s before executing (so that
you can do something to it, such as attach strace or a debugger to the
process).

You can, and should use strace and ltrace to gather more information about what
was happening to Cyrus when it malfunctioned.  straces are useful when
networking or signal problems appear to be the issue, and ltraces can give
hints on what the problem might be.

If a Cyrus service is crashing and cyrmaster logs that the service is being
killed by a signal, please use the debugging hooks to provide a back-trace
using gdb (see below).  Back-traces are extremely useful when locating where
Cyrus is dying, and why.

Debugging information is sent to syslogd, using the DEBUG priority, facilities
MAIL and DAEMON.

You can also try to set MALLOC_CHECK_=2 in the environment, so that malloc()
will cause Cyrus to dump core if it detects any sort of corruption.


Telemetry logs
--------------

Cyrus will happily log all communications between the Cyrus store closed-box and
the outside world.  These logs are sometimes vital to understand exactly what
is happening and to reproduce bugs.

To enable telemetry logging, create a directory under /var/lib/cyrus/log with
the same name as the username for which you want the communication sessions to
be logged.  Cyrus will log all imap, pop3, sieve and lmtp talks authenticated
as that user (including proxied connections).  Make sure the directory is owned
by user cyrus.

Watch out for sensitive information such as passwords when you submit the
telemetry logs to a public bug-tracking system or mailinglist.


Recompiling Cyrus with debugging information
--------------------------------------------

In order to produce useful back-traces, or to interactively debug Cyrus,
you must rebuild the package with debugging information.  It is quite
easy to do so:

1. Install all source dependencies to build the package (needs root):
   apt-get install build-essential fakeroot
   apt-get build-dep cyrus-imapd-2.2

2. Download and rebuild Cyrus with debug information:
   apt-get source cyrus-imapd-2.2
   cd cyrus-imapd-2.2*
   DEB_BUILD_OPTIONS=debug,noopt,nostrip dpkg-buildpackage -uc -us -rfakeroot

3. Install the Cyrus packages with debug information (needs root):
   cd ..
   dpkg -i *deb     (or something like that)

Now Cyrus should be working fine, using binaries with full debug information
for gdb.  For interactive debugging, you may want to make sure there are no
optimizations, in which case you should use "DEB_BUILD_OPTIONS=noopt,nostrip
dpkg-buildpackage -uc -us -rfakeroot".

Warning: the next time you run apt-get update, apt will probably download the
non-debugging version of the Cyrus debs, and install them over the debugging
packages.

To install the non-debugging, optimized version of Cyrus over the debugging
one, issue "apt-get --reinstall install (package)" commands for all the Cyrus
packages you want replaced.


Attaching debuggers to Cyrus, and getting traces
------------------------------------------------

You can tell Cyrus services to run a debugging command just before they
start doing real work.  This can be used to run strace, ltrace and gdb
or ddd (for interactive debugging and back-tracing) quite easily.

Set the shell command to be run in /etc/imapd.conf, option debug_command.
Then, add the command line switch "-D" to the Cyrus services you want to
run the debug_command in /etc/cyrus.conf, and restart cyrmaster using
/etc/init.d/cyrus2.2 restart.

The debugging command must be given as a single line in the configuration file.

To get a back-trace using gdb:
debug_command: /usr/bin/gdb -batch -cd=/tmp -x /usr/lib/cyrus/getbacktrace.gdb /usr/lib/cyrus/bin/%s %d >/tmp/gdb-backtrace.cyrus.%1$s.%2$d <&- 2>&1 &

The above will produce a back-trace of every service run with -D that segfaults
in the files /tmp/gdb-backtrace.cyrus.*;  /usr/lib/cyrus/getbacktrace.gdb
simply has the sequence of commands for gdb: c (to continue running the
service), bt (to get the back-trace if the program didn't exit normally), quit
(to quit gdb).

For strace, you can use:
debug_command: /usr/bin/strace -tt -o /tmp/strace.cyrus.%s.%d -p %2$d <&- 2>&1 &

Which will produce straces in /tmp/strace.cyrus.*

For ltrace, you can use:
debug_command: /usr/bin/ltrace -tt -n 2 -o /tmp/ltrace.cyrus.%s.%d -p %2$d <&- 2>&1 &

Which will produce ltraces in /tmp/ltrace.cyrus.*

Be warned that sensitive information such as passwords may be disclosed in the
strace and ltrace output, so mangle them before sending such traces to public
bug-tracking systems or mailing lists.

 -- Henrique de Moraes Holschuh <hmh@debian.org>
1	Cyrus IMAP for Debian, debugging procedures
2	$Id: README.Debian.debug 229 2005-12-08 23:26:29Z astronut $
3	-------------------------------------------
4
5	For more information, please consult http://asg.web.cmu.edu/cyrus/imapd/.
6
7	Cyrus has various levels of debugging aid, which can and should be used to
8	offer more information about any problems you are facing with Cyrus.
9
10	First, edit /etc/default/cyrus2.2, and set CYRUS_VERBOSE to a number higher
11	than zero. The higher the number, more debug information is provided. Numbers
12	above 30 will cause Cyrus services to pause for 15s before executing (so that
13	you can do something to it, such as attach strace or a debugger to the
14	process).
15
16	You can, and should use strace and ltrace to gather more information about what
17	was happening to Cyrus when it malfunctioned. straces are useful when
18	networking or signal problems appear to be the issue, and ltraces can give
19	hints on what the problem might be.
20
21	If a Cyrus service is crashing and cyrmaster logs that the service is being
22	killed by a signal, please use the debugging hooks to provide a back-trace
23	using gdb (see below). Back-traces are extremely useful when locating where
24	Cyrus is dying, and why.
25
26	Debugging information is sent to syslogd, using the DEBUG priority, facilities
27	MAIL and DAEMON.
28
29	You can also try to set MALLOC_CHECK_=2 in the environment, so that malloc()
30	will cause Cyrus to dump core if it detects any sort of corruption.
31
32
33	Telemetry logs
34	--------------
35
36	Cyrus will happily log all communications between the Cyrus store closed-box and
37	the outside world. These logs are sometimes vital to understand exactly what
38	is happening and to reproduce bugs.
39
40	To enable telemetry logging, create a directory under /var/lib/cyrus/log with
41	the same name as the username for which you want the communication sessions to
42	be logged. Cyrus will log all imap, pop3, sieve and lmtp talks authenticated
43	as that user (including proxied connections). Make sure the directory is owned
44	by user cyrus.
45
46	Watch out for sensitive information such as passwords when you submit the
47	telemetry logs to a public bug-tracking system or mailinglist.
48
49
50	Recompiling Cyrus with debugging information
51	--------------------------------------------
52
53	In order to produce useful back-traces, or to interactively debug Cyrus,
54	you must rebuild the package with debugging information. It is quite
55	easy to do so:
56
57	1. Install all source dependencies to build the package (needs root):
58	apt-get install build-essential fakeroot
59	apt-get build-dep cyrus-imapd-2.2
60
61	2. Download and rebuild Cyrus with debug information:
62	apt-get source cyrus-imapd-2.2
63	cd cyrus-imapd-2.2*
64	DEB_BUILD_OPTIONS=debug,noopt,nostrip dpkg-buildpackage -uc -us -rfakeroot
65
66	3. Install the Cyrus packages with debug information (needs root):
67	cd ..
68	dpkg -i *deb (or something like that)
69
70	Now Cyrus should be working fine, using binaries with full debug information
71	for gdb. For interactive debugging, you may want to make sure there are no
72	optimizations, in which case you should use "DEB_BUILD_OPTIONS=noopt,nostrip
73	dpkg-buildpackage -uc -us -rfakeroot".
74
75	Warning: the next time you run apt-get update, apt will probably download the
76	non-debugging version of the Cyrus debs, and install them over the debugging
77	packages.
78
79	To install the non-debugging, optimized version of Cyrus over the debugging
80	one, issue "apt-get --reinstall install (package)" commands for all the Cyrus
81	packages you want replaced.
82
83
84	Attaching debuggers to Cyrus, and getting traces
85	------------------------------------------------
86
87	You can tell Cyrus services to run a debugging command just before they
88	start doing real work. This can be used to run strace, ltrace and gdb
89	or ddd (for interactive debugging and back-tracing) quite easily.
90
91	Set the shell command to be run in /etc/imapd.conf, option debug_command.
92	Then, add the command line switch "-D" to the Cyrus services you want to
93	run the debug_command in /etc/cyrus.conf, and restart cyrmaster using
94	/etc/init.d/cyrus2.2 restart.
95
96	The debugging command must be given as a single line in the configuration file.
97
98	To get a back-trace using gdb:
99	debug_command: /usr/bin/gdb -batch -cd=/tmp -x /usr/lib/cyrus/getbacktrace.gdb /usr/lib/cyrus/bin/%s %d >/tmp/gdb-backtrace.cyrus.%1$s.%2$d <&- 2>&1 &
100
101	The above will produce a back-trace of every service run with -D that segfaults
102	in the files /tmp/gdb-backtrace.cyrus.*; /usr/lib/cyrus/getbacktrace.gdb
103	simply has the sequence of commands for gdb: c (to continue running the
104	service), bt (to get the back-trace if the program didn't exit normally), quit
105	(to quit gdb).
106
107	For strace, you can use:
108	debug_command: /usr/bin/strace -tt -o /tmp/strace.cyrus.%s.%d -p %2$d <&- 2>&1 &
109
110	Which will produce straces in /tmp/strace.cyrus.*
111
112	For ltrace, you can use:
113	debug_command: /usr/bin/ltrace -tt -n 2 -o /tmp/ltrace.cyrus.%s.%d -p %2$d <&- 2>&1 &
114
115	Which will produce ltraces in /tmp/ltrace.cyrus.*
116
117	Be warned that sensitive information such as passwords may be disclosed in the
118	strace and ltrace output, so mangle them before sending such traces to public
119	bug-tracking systems or mailing lists.
120
121	-- Henrique de Moraes Holschuh <hmh@debian.org>