XS4ALL Diablo patches

I'm running the diablo NNTP software on our newszilla usenet cluster at XS4ALL.

We're using the diablo-5-CUR-20090530-00 version from the development snapshots.

Debian init.d script

We run diablo on Debian/Linux systems, and we store the history file on ramfs (note, not tmpfs, as that can get swapped out, if that happens performance is worse than any other disk-based filesystem. Ramfs cannot get swapped out.). This is the /etc/init.d/diablo file that sets it all up:

. Ofcourse there's a matching /etc/init.d/dreaderd file, which sources a /etc/default/dreaderd file for config settings: (version init-20110628)

New set of /news/adm scripts

The sample administration script that ship with diablo are meant to be customized for your local setup. Unfortunately they were written in C shell. I did a rewrite in standard /bin/sh, which in most cases you do not have to customize at all. They automatically detect if dreaderd or diablo are running, and run only if needed. Just call the scripts from news' crontab and you're all set. (version adm-20110520)

libradius-linux patch

We use libradius-linux on our Linux systems to get the Diablo radius support to work. It's an (old, but working) port of the FreeBSD radius library. You're supposed to link it with the libmd MD5 library, but I like to keep dependencies to a minimum so I patched it to link with libcrypto from openssl instead (which is almost certainly included with your linux distro). (version libradius-linux-20040827)

Diablo patches

Finally, these are the patches we use on top of diablo-5-CUR-20090530-00:
(version patches-20110713)

Required fixes

01_diablo-CUR-64bit-fixes.patch Fixes remaining 64 bit issues - mostly debug printf's
02_diablo-CUR-large-artno-fixes.patch Fixes dgrpctrl and dreaderover output for long article numbers.
03_diablo-CUR-ipv46-dns.patch Don't do unnecessary DNS PTR queries
04_diablo-CUR-dreaderd-msg.patch If SendMsg() is used without filedescriptor-passing, do not attach a zero-length control message.
05_diablo-CUR-zlib-lfs.patch zlib.h doesn't want to compile if _FILE_OFFSET_BITS == 64 but_LARGEFILE64_SOURCE isn't defined, work around that
06_diablo-CUR-glibc-strptime.patch Add strptime() prototype - not standard in glibc headers
07_diablo-CUR-dreaderd-linux-db185.patch Linux needs a compatibility include if the db 1.85 API is used
08_diablo-CUR-listen-v4-v6.patch On current (e)glibc versions getaddrinfo() returns the wildcard IPv4 address before the wildcard IPv6 address, which means that we don't get a shared IPv6/IPv4 socket anymore. It never worked on *BSD anyways (where IPV6_V6ONLY is the default). Fix it up correctly.
09_diablo-CUR-zalloc-fixes.patch o Fix a memory leak in nzalloc(). When adding an extra mempool to the list of pools, the original mempool was lost subsequently never freed by freePool(). o Fix a wrong assumption in zallocStrTrim2(): you cannot shorten a string from zallocStr by inserting a '\0', because strlen() is used by zfreeStr() to find out the number of bytes to free.
10_diablo-CUR-dreaderd-groupmap-memleak.patch ReadAccessCache() forgot to set accessmap.GroupMapSize, so that xunmap() failed on it - it never got unmapped.
11_diablo-CUR-dreader-sr_cconn-null.patch When a spoolserver dies, don't requeue pending requests for clients that have already disconnected. This makes dreaderd forks much more longlived, so do not apply this without the memory leak fixes (zalloc-fixes and dreaderd-groupmap-memleak)
12_diablo-CUR-dreaderd-fast-connlost.patch When the remote side closes the connection, write() returns a write error (which we mostly ignore) but after that select() never tells us that the socket is writable. There is a check for this condition in the reader.c mainloop but it might take long before it's noticed. Explicitly check for write() errors and kill the session if so.

Required enhancements

20_diablo-CUR-large-fdset.patch Increase the number of filedescriptors we can select() on with glibc
21_diablo-CUR-dreaderd-limit-pipelining.patch Dreaderd allows pipeling with a large limit (one read buffer, 64K bytes). If a client sends "BODY \r\nBODY \r\n" with hundreds of articles, and the connection to the client is slower than the connection to the backend (which it always is), dreaderd will download and buffer all of those articles in memory. I've seen dreaderd swell up to gigabytes because of this. This patch makes dreaderd process the next command from the input buffer only after the output buffer gets emptied.

Optional enhancements

30_diablo-CUR-diablo-allocstrat-weighted.patch New allocstrat: "weighted", to spread the articles over different spools relative to the spool size
31_diablo-CUR-dreaderd-prealloc.patch Use disk preallocation for the overview database, prevents fragmentation. The implementation is just done for Linux, but can easily be added for other OS's if they offer this feature.
32_diablo-CUR-nostatic.patch Don't compile diablo and dnewslink statically .. it's 2010
33_diablo-CUR-ipv6-enable.patch Enable IPv6 by default .. it's 2010

XS4ALL local enhancements

40_diablo-CUR-dreaderd-friendly-maxconn-msgs.patch Change messages like "Your host connection limit reached" into "Too many connections from your host (max %d)", etc, because they were confusing users ("I thought there was no download limit!").
41_diablo-CUR-dreaderd-rad-readerdef.patch Make it possible to return dreader.access attributes in a radius reply
42_diablo-CUR-xs4all-config.patch XS4ALL/Newszilla compile settings

Experimental patches

50_diablo-CUR-zalloc-debug.patch Keep a shadow administration of mempools and memnodes, and dump usage to a file in /news/run on SIGUSR2. Helped a lot with debugging, but has a lot of overhead when enabled (compile-time setting).

For diablo stuff I'm currently mikevs [at] xs4all.net.