Is there any memory leak in unbound?

Tao_Ma · September 8, 2009, 6:30am

Hi guys,I got a problem.

I installed unbound 1.3.3 and found when the unbound is running,it use more and more memory..
I even found it occupied 96% of memory and then was killed by the system.When it was killed,the following log was found in /var/log/messages(“Dnsserv” is the hostname):
Sep 8 00:21:02 Dnsserv Out of memory: kill process 23999 (unbound) score 125734 or a child
Sep 8 00:21:02 Dnsserv Killed process 23999 (unbound)

I only specify some normal items in unbound.conf and it was shown bellow:
server:
verbosity: 1
statistics-interval: 1
num-threads: 1
interface: 0.0.0.0
interface-automatic: no
port: 53
msg-cache-size: 4m
rrset-cache-size: 4m
do-ip4: yes
do-ip6: no
do-udp: yes
access-control: 10.0.0.0/8 allow
access-control: 127.0.0.0/8 allow
statistics-interval: 10
#chroot: “/var/unbound”
username: “unbound”
directory: “/var/unbound”
logfile: “”
use-syslog: no
log-time-ascii: yes
pidfile: “/var/run/unbound.pid”
root-hints: “/var/unbound/named.cache”
hide-version: no
version: “hiahia”

local-zone: “mydns123.com.” static
local-data: “mydns123.com. SOA ns.mydns123.com tj.mydns123.com. 1 28800 7200 604800 86400”
local-data: “mydns123.com. IN NS ns.mydns123.com.”
local-data: “mydns123.com. IN NS ns2.mydns123.com.”
local-data: “ns.mydns123.com. IN A 192.1.1.1”
local-data: “ns1.mydns123.com. IN A 192.1.1.2”
local-data: “www.mydns123.com. IN A 192.1.1.3”
local-data: “ftp.mydns123.com. IN A 192.1.1.4”
local-data: “wa.mydns123.com. IN A 192.1.1.5”
local-data: “pp.mydns123.com. IN CNAME wa.mydns123.com.”

local-zone:“1.1.192.in-addr.arpa” static
local-data: “mydns123.com. SOA ns.mydns123.com tj.mydns123.com. 1 28800 7200 604800 86400”
local-data: “mydns123.com. IN NS ns.mydns123.com.”
local-data: “mydns123.com. IN NS ns2.mydns123.com.”
local-data:“1.1.1.192.in-addr.arpa IN PTR ns.mydns123.com.”
local-data:“2.1.1.192.in-addr.arpa IN PTR ns1.mydns123.com.”
local-data:“3.1.1.192.in-addr.arpa IN PTR www.mydns123.com.”
local-data:“4.1.1.192.in-addr.arpa IN PTR ftp.mydns123.com.”
local-data:“5.1.1.192.in-addr.arpa IN PTR was.mydns123.com.”

Is there any mistake in my configuration?Or is it a bug of unbound?

|

PaulWouters · September 8, 2009, 3:00pm

Is this an embedded device with little memory? eg 16mb or 32mb?

The OOM killer might be killing the biggest memory consumer, which
not neccessarilly means that process is the one leaking memory. unbound might be running stable at just over 8mb, and some other
process might be slowly allocating more memory over time, but still
end up using a smaller memory footprint, for which unbound takes the
blame (and the shot)

Paul

Tao_Ma · September 9, 2009, 7:01am

--- Paul Wouters <paul@xelerance.com>Ð´µÀ:

> I installed unbound 1.3.3 and found when the
unbound is running,it use
> more and more memory..
> I even found it occupied 96% of memory and then
was killed by the

> Sep 8 00:21:02 Dnsserv Out of memory: kill process
23999 (unbound) score

> I only specify some normal items in unbound.conf
and it was shown bellow:
> server:

> msg-cache-size: 4m
> rrset-cache-size: 4m

Is this an embedded device with little memory? eg
16mb or 32mb?

The OOM killer might be killing the biggest memory
consumer, which
not neccessarilly means that process is the one
leaking memory.
unbound might be running stable at just over 8mb,
and some other
process might be slowly allocating more memory over
time, but still
end up using a smaller memory footprint, for which
unbound takes the
blame (and the shot)

Paul

Hi Paul,thanks for your reply.
Sorry to send this mail again because I forgot to cc
to unbound-users@unbound.net in the last mail.

It's my fault to forget to tell you my server's
hardware and software enviroment in the first mail.
The server has 2G memory and the CPU is Intel(R)
Celeron(R) M processor 1.50GHz.I didn't specify any
other options except "--prefix=/usr/local/unboud" in
configure,then installed unbound by "make&&make
install".

Here is the shared library used by unboud:
linux-gate.so.1 => (0xffffe000)
libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8
(0xb7f45000)
libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8
(0xb7e0f000)
libpthread.so.0 => /lib/libpthread.so.0 (0xb7df8000)
libc.so.6 => /lib/libc.so.6 (0xb7cd0000)
libdl.so.2 => /lib/libdl.so.2 (0xb7ccc000)
/lib/ld-linux.so.2 (0xb7f8d000)

When processing lots of DNS requests(like about 50,000
per second),unbound was found consume more and more
memory.I found it by reading the output of command
"top".Unobund wouldn't stop consuming more memory
until killed by OOM killer.
In my opinion,the memory consumed by unbound should be
stable when the count of requests per
second(say,3,000) is stable,is it right?

Any hints for me to find the reason of the problem?
Thanks.

Wouter · September 10, 2009, 2:25pm

Hi Tao Ma,

The memory statistics that unbound makes can help?

It prints them into the log file (depending on verbosity
level and --enable-debug to configure) like this:
debug: cache memory msg=1620127 rrset=2964258 infra=738306 val=128832

Also the memory statistics are printed when you enable extended
statistics and do unbound-control stats, like this:
mem.cache.rrset=2980222
mem.cache.message=1633031
mem.mod.iterator=16460
mem.mod.validator=130126

Best regards,
Wouter

Tao_Ma · September 11, 2009, 3:50am

Hi Wouter,thanks very much for your reply.

I tried the method you mentioned,here is the info I got from the log:
debug: cache memory msg=16528 rrset=33040 infra=672 val=24888
This info keeps the same from the start of the unbound(I have only about 5 records in unbound).

I also found the value of mem.total.sbrk printed by “unbound-control stats” keeps growing.The item “mem.total.sbrk” is an estimate of the heap size of unbound in bytes,it is close to the memory used by unbound.

I have a script keeps recording mem.total.sbrk in a file,and found when mem.total.sbrk grew to 2017873920,unbound was killed by system(My server has a 2G memory).

Unbound runs in a Gentoo Linux with 2.6.23 kernel,does the problem relate to the OS?

Thanks & Best regards,
Tao Ma

— 09年9月10日，周四, W.C.A. Wijngaards wouter@NLnetLabs.nl 写道：

> 发件人: W.C.A. Wijngaards wouter@NLnetLabs.nl
> 主题: Re: [Unbound-users] 回复： Re: Is there any memory leak in unbound?
> 收件人: “Tao Ma” matao_reg@yahoo.com.cn
> 抄送: “Paul Wouters” paul@xelerance.com, unbound-users@unbound.net
> 日期: 2009年9月10日,周四,下午10:25
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Tao Ma,
>
> The memory statistics that unbound makes can help?
>
> It prints them into the log file (depending on verbosity
> level and --enable-debug to configure) like this:
> debug: cache memory msg=1620127 rrset=2964258 infra=738306 val=128832
>
> Also the memory statistics are printed when you enable extended
> statistics and do unbound-control stats, like this:
> mem.cache.rrset=2980222
> mem.cache.message=1633031
> mem.mod.iterator=16460
> mem.mod.validator=130126
>
> Best regards,
> Wouter

|

Tao_Ma · September 11, 2009, 8:06am

What's more,I found that the memory occupied by unbound would not be freed when all requets finished,that means,unbound only consume memory but never free them. What does unbound hold the memory for?It quite like that a memory leak happens... Anyone else has been in the same trouble?

Best Regards,
Tao Ma

— 09年9月11日，周五, Tao Ma <matao_reg@yahoo.com.cn> 写道：

发件人: Tao Ma <matao_reg@yahoo.com.cn>
主题: Re: [Unbound-users] Is there any memory leak in unbound?
收件人: “W.C.A. Wijngaards” <wouter@nlnetlabs.nl>
抄送: unbound-users@unbound.net
日期: 2009年9月11日,周五,上午11:50

Hi Wouter,thanks very much for your reply.

I tried the method you mentioned,here is the info I got from the log:
debug: cache memory msg=16528 rrset=33040 infra=672 val=24888
This info keeps the same from the start of the unbound(I have only about 5 records in unbound).

I also found the value of mem.total.sbrk printed by “unbound-control stats” keeps growing.The item “mem.total.sbrk” is an estimate of the heap size of unbound in bytes,it is close to the memory used by unbound.

I have a script keeps recording mem.total.sbrk in a file,and found when mem.total.sbrk grew to 2017873920,unbound was killed by system(My server has a 2G memory).

Unbound runs in a Gentoo Linux with 2.6.23 kernel,does the problem relate to the OS?

Thanks & Best regards,
Tao Ma

— 09年9月10日，周四, W.C.A. Wijngaards <wouter@NLnetLabs.nl> 写道：

> 发件人: W.C.A. Wijngaards <wouter@NLnetLabs.nl>
> 主题: Re: [Unbound-users] 回复： Re: Is there any memory leak in unbound?
> 收件人: “Tao Ma” <matao_reg@yahoo.com.cn>
> 抄送: “Paul Wouters” <paul@xelerance.com>, unbound-users@unbound.net
> 日期: 2009年9月10日,周四,下午10:25
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Tao Ma,
>
> The memory statistics that unbound makes can help?
>
> It prints them into the log file (depending on verbosity
> level and --enable-debug to configure) like this:
> debug: cache memory msg=1620127 rrset=2964258 infra=738306 val=128832
>
> Also the memory statistics are printed when you enable extended
> statistics and do unbound-control stats, like this:
> mem.cache.rrset=2980222
> mem.cache.message=1633031
> mem.mod.iterator=16460
> mem.mod.validator=130126
>
> Best regards,
> Wouter

好玩贺卡等你发，邮箱贺卡全新上线！

-----下面为附件内容-----

Unbound-users mailing list
Unbound-users@unbound.net
http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users

Artis_Caune · September 11, 2009, 11:02am

Hi,

do you reload unbound while it's running ?

Tao_Ma · September 11, 2009, 11:37am

Hi,I didn’t reload unbound while it’s running,I always start it and then send requests to the server,then it will consume more and more memory.

— 09年9月11日，周五, Artis Caune artis.caune@gmail.com 写道：

> 发件人: Artis Caune artis.caune@gmail.com
> 主题: Re: [Unbound-users] Is there any memory leak in unbound?
> 收件人: “Tao Ma” matao_reg@yahoo.com.cn
> 抄送: wouter@nlnetlabs.nl, unbound-users@unbound.net
> 日期: 2009年9月11日,周五,下午7:02
>
> Hi,
>
> do you reload unbound while it’s running ?
>
> –
> Artis Caune
>
> 聽聽Everything should be made as simple as possible, but not simpler.

|

Wouter · September 15, 2009, 9:26am

Hi Tao Ma,

debug: cache memory msg=16528 rrset=33040 infra=672 val=24888

The values from your log file look very reasonable. No problems here.
The sbrk keeps growing?

Ok, it is possible to enable even more memory debug, configure with
- --enable-alloc-checks and compile again. Unbound prints:
info: Memory conditions: %u front=%u back=%u mesh=%u msg=%u rrset=%u
infra=%u iter=%u val=%u alloccache=%u globalalloccache=%u me=%u
But with values.

It also prints a log line for *every* alloc call. So, you can see where
in the code it allocates all that memory. This will make the
server very slow. If you send me the log file (off-list to
wouter@nlnetlabs.nl , then I can parse it with testcode/memstats.c;
or you can do that yourself: make memstats ; ./memstats < logfile ).

To answer your question, unbound keeps some memory around, up to a
maximum, because free-ing something that you need later again is slow.
This should top out at 40Mb for the standard config.

Gentoo? Is it 64bit? Does it use a 'special' memory allocator in libc?

Best regards,
Wouter

Wouter · September 15, 2009, 9:29am

Hi Tao Ma,

Just realized something else, the sbrk estimate is a dirty hack.
Gentoo may report a different sbrk() value, and the dirty hack
does not work. The sbrk() value keeps growing but maybe it is
not true that unbound's memory keeps growing. Does the 'top' or
'ps' memory number also grow?

Best regards,
Wouter

Tao_Ma · September 17, 2009, 9:47am

Hi Wouter,I’m very glad to tell you,I patched the diff file you gave me and now unbound works well on the server,thanks very much!

I’m sorry that I didn’t tell you clearly about the compile problem.When we found we couldn’t compile the code with “–enable-alloc-checks”,we added another option “–enable-static-exe” and then finished compiling smoothly,that’s all.

Thanks again for your warmhearted help.

Best Regards,
Tao Ma

— 09年9月16日，周三, W.C.A. Wijngaards wouter@nlnetlabs.nl 写道：

> 发件人: W.C.A. Wijngaards wouter@nlnetlabs.nl
> 主题: Re: [Unbound-users] Is there any memory leak in unbound?
> 收件人: “Tao Ma” matao_reg@yahoo.com.cn
> 日期: 2009年9月16日,周三,下午8:47
>
> Hi Tao Ma,
>
> Does this solve your memory leak problem?
>
> Index: libunbound/libworker.c
> ===================================================================
> — libunbound/libworker.c (revision 1831)
> +++ libunbound/libworker.c (working copy)
> @@ -514,6 +514,7 @@
> ldns_buffer_write_u16_at(w->back->udp_buff, 2, qflags);
> if(local_zones_answer(ctx->local_zones, &qinfo, &edns,
> w->back->udp_buff, w->env->scratch)) {
> + regional_free_all(w->env->scratch);
> libworker_fillup_fg(q, LDNS_RCODE_NOERROR,
> w->back->udp_buff, sec_status_insecure);
> libworker_delete(w);
> @@ -630,6 +631,7 @@
> ldns_buffer_write_u16_at(w->back->udp_buff, 2, qflags);
> if(local_zones_answer(w->ctx->local_zones, &qinfo, &edns,
> w->back->udp_buff, w->env->scratch)) {
> + regional_free_all(w->env->scratch);
> q->msg_security = sec_status_insecure;
> add_bg_result(w, q, w->back->udp_buff, UB_NOERROR);
> free(qinfo.qname);
> Index: daemon/worker.c
> ===================================================================
> — daemon/worker.c (revision 1831)
> +++ daemon/worker.c (working copy)
> @@ -806,6 +806,7 @@
> }
> if(local_zones_answer(worker->daemon->local_zones, &qinfo, &edns,
> c->buffer, worker->scratchpad)) {
> + regional_free_all(worker->scratchpad);
> if(ldns_buffer_limit(c->buffer) == 0) {
> comm_point_drop_reply(repinfo);
> return 0;
>
> Sorry about the compile problem with unbound-host and alloc-checks, I’ll
> look at it later.
>
> Best regards,
> Wouter
>
> On 09/16/2009 10:05 AM, Tao Ma wrote:
> > Hi Wouter,thanks for your reply.
> >
> > We use a 32-bit Gentoo and I don’t think there’s a special memory
> > allocator in libc.
> > I can always find the memory consumed by unbound keep growing from ouput
> > of ‘ps’ and ‘top’,the result of sbrk estimate seems to be in accord with
> > this.
> >
> > Following your suggestion,we configure with --enable-alloc-checks and
> > compile unbound again,now we can find the “Memory conditions” value keep
> > growing in the log.
> >
> > We use memstats to parse the log,but found nothing but the following ouput:
> > Reading /var/unbound/log_test of size 38767531
> > 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20%
> > 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38%
> > 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56%
> > 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74%
> > 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92%
> > 93% 94% 95% 96% 97% 98% 99% 100% done
> > ------------
> > 0 / 0 total in 0 code lines
> >
> > We got a complete record in the log from the start of unbound to the
> > exiting of unbound,and the log file has been attatched to the mail,maybe
> > it will help you to analyse.
> >
> > Let me know if you find any mistakes in my operation.
> >
> > By the way,when we configure unbound-1.3.3 with “./configure
> > --prefix=/usr/local/unbound --enable-alloc-checks” and compile,we got
> > the following error:
> > build/smallapp/.libs/unbound-host.o: In function print_rd': > > /home/mt/tools/unbound-1.3.3/smallapp/unbound-host.c:233: undefined > > reference to unbound_stat_malloc_log’
> > /home/mt/tools/unbound-1.3.3/smallapp/unbound-host.c:255: undefined
> > reference to unbound_stat_free_log' > > build/smallapp/.libs/unbound-host.o: In function pretty_rdata’:
> > /home/mt/tools/unbound-1.3.3/smallapp/unbound-host.c:247: undefined
> > reference to unbound_stat_free_log' > > build/smallapp/.libs/unbound-host.o: In function lookup’:
> > /home/mt/tools/unbound-1.3.3/smallapp/unbound-host.c:384: undefined
> > reference to unbound_stat_free_log' > > collect2: ld returned 1 exit status > > make: *** [unbound-host] Error 1 > > > > We did some modification to the code and finished compiling.Is there > > anything wrong in the code? > > > > Thanks & Best regards, > > Tao Ma > > > > > > > > --- *09年9月15日，周二, W.C.A. Wijngaards /<wouter@NLnetLabs.nl>/* 写道： > > > > > > 发件人: W..C.A. Wijngaards <wouter@NLnetLabs.nl> > > 主题: Re: [Unbound-users] Is there any memory leak in unbound? > > 收件人: "W.C.A. Wijngaards" <wouter@nlnetlabs.nl> > > 抄送: "Tao Ma" <matao_reg@yahoo.com.cn>, unbound-users@unbound.net > > 日期: 2009年9月15日,周二,下午5:29 > > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > Hi Tao Ma, > > > > Just realized something else, the sbrk estimate is a dirty hack. > > Gentoo may report a different sbrk() value, and the dirty hack > > does not work. The sbrk() value keeps growing but maybe it is > > not true that unbound's memory keeps growing. Does the 'top' or > > 'ps' memory number also grow? > > > > Best regards, > > Wouter > > > > On 09/15/2009 11:26 AM, W.C.A. Wijngaards wrote: > > > Hi Tao Ma, > > > > > >> debug: cache memory msg=16528 rrset=33040 infra=672 val=24888 > > > > > > The values from your log file look very reasonable. No problems here. > > > The sbrk keeps growing? > > > > > > Ok, it is possible to enable even more memory debug, configure with > > > --enable-alloc-checks and compile again. Unbound prints: > > > info: Memory conditions: %u front=%u back=%u mesh=%u msg=%u rrset=%u > > > infra=%u iter=%u val=%u alloccache=%u globalalloccache=%u me=%u > > > But with values. > > > > > > It also prints a log line for *every* alloc call. So, you can see > > where > > > in the code it allocates all that memory. This will make the > > > server very slow. If you send me the log file (off-list to > > > wouter@nlnetlabs.nl </mc/compose?to=wouter@nlnetlabs.nl> , then I > > can parse it with testcode/memstats.c; > > > or you can do that yourself: make memstats ; ./memstats < logfile ). > > > > > > To answer your question, unbound keeps some memory around, up to a > > > maximum, because free-ing something that you need later again is > > slow. > > > This should top out at 40Mb for the standard config. > > > > > > Gentoo? Is it 64bit? Does it use a 'special' memory allocator in > > libc? > > > > > > Best regards, > > > Wouter > > > > > > On 09/11/2009 10:06 AM, Tao Ma wrote: > > >> What's more,I found that the memory occupied by unbound would not be > > >> freed when all requets finished,that means,unbound only consume > > memory > > >> but never free them. > > >> What does unbound hold the memory for?It quite like that a > > memory leak > > >> happens... > > >> Anyone else has been in the same trouble? > > > > > > > > >> Best Regards, > > >> Tao Ma > > > > > > > > >> --- *09t911氓h聰, Tao Ma /<matao_reg@yahoo.com.cn > > </mc/compose?to=matao_reg@yahoo.com.cn>>/* 聶S > > > > > > > > >> 脩枚潞: Tao Ma <matao_reg@yahoo.com.cn > > </mc/compose?to=matao_reg@yahoo.com.cn>> > > >> ;聵: Re: [Unbound-users] Is there any memory leak in unbound? > > >> 6枚潞: "W.C.A. Wijngaards" <wouter@nlnetlabs.nl > > </mc/compose?to=wouter@nlnetlabs.nl>> > > >> 聞: unbound-users@unbound.net > > </mc/compose?to=unbound-users@unbound.net> > > >> 氓: 2009t911氓,h聰, > > > H11:50 > > > > > >> Hi Wouter,thanks very much for your reply. > > > > > >> I tried the method you mentioned,here is the info I got from the > > log: > > >> debug: cache memory msg=16528 rrset=33040 infra=672 val=24888 > > >> This info keeps the same from the start of the unbound(I have only > > >> about 5 records in unbound). > > > > > >> I also found the value of mem..total.sbrk printed by > > "unbound-control > > >> stats" keeps growing.The item "mem.total.sbrk" is an estimate of the > > >> heap size of unbound in bytes,it is close to the memory used by > > unbound. > > > > > >> I have a script keeps recording mem.total.sbrk in a file,and found > > >> when mem.total.sbrk grew to 2017873920,unbound was killed by > > >> system(My server has a 2G memory). > > > > > >> Unbound runs in a Gentoo Linux with 2.6.23 kernel,does the problem > > >> relate to the OS? > > > > > >> Thanks & Best regards, > > >> Tao Ma > > > > > > > > > > > >> --- *09t910氓h脹, W.C.A. Wijngaards /<wouter@NLnetLabs.nl > > </mc/compose?to=wouter@NLnetLabs.nl>>/* > > >> 聶S > > > > > > > > >> 脩枚潞: W.C.A. Wijngaards <wouter@NLnetLabs.nl > > </mc/compose?to=wouter@NLnetLabs.nl>> > > >> ;聵: Re: [Unbound-users] 脼 Re: Is there any memory leak in > > >> unbound? > > >> 6枚潞: "Tao Ma" <matao_reg@yahoo.com.cn > > </mc/compose?to=matao_reg@yahoo.com.cn>> > > >> 聞: "Paul Wouters" <paul@xelerance.com > > </mc/compose?to=paul@xelerance.com>>, unbound-users@unbound.net > > </mc/compose?to=unbound-users@unbound.net> > > >> 氓: 2009t910氓,h脹,H10:25 > > > > > >> -----BEGIN PGP SIGNED MESSAGE----- > > >> Hash: SHA1 > > > > > >> Hi Tao Ma, > > > > > >> The memory statistics that unbound makes can help? > > > > > >> It prints them into the log file (depending on verbosity > > >> level and --enable-debug to configure) like this: > > >> debug: cache memory msg=1620127 rrset=2964258 infra=738306 > > >> val=128832 > > > > > >> Also the memory statistics are printed when you enable extended > > >> statistics and do unbound-control stats, like this: > > >> mem.cache.rrset=2980222 > > >> mem.cache.message=1633031 > > >> mem.mod.iterator=16460 > > >> mem.mod.validator=130126 > > > > > >> Best regards, > > >> Wouter > > > > > > > > >> > > ------------------------------------------------------------------------ > > >> }漏:aI脩庐卤:ah掳
> > > 驴
> > >>
> > <http://cn.rd.yahoo.com/mail_cn/tagline/card/*http://card.mail.cn.yahoo.com/>
> > >
> > >> -----b:D枚聟鹿-----
> > >
> > >> _______________________________________________
> > >> Unbound-users mailing list
> > >> Unbound-users@unbound.net
> > /mc/compose?to=Unbound-users@unbound.net
> > </mc/compose?to=Unbound-users@unbound.net
> > /mc/compose?to=Unbound-users@unbound.net>
> > >> http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
> > >
> > >
> > >>
> > ------------------------------------------------------------------------
> > >> }漏:aI`脩庐卤:ah掳
> > > 驴
> > >>
> > <http://cn.rd.yahoo.com/mail_cn/tagline/card/*http://card.mail.cn.yahoo.com/>
> > >
> > >
> > _______________________________________________
> > Unbound-users mailing list
> > Unbound-users@unbound.net /mc/compose?to=Unbound-users@unbound.net
> > http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.9 (GNU/Linux)
> > Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
> >
> > iEYEARECAAYFAkqvXmkACgkQkDLqNwOhpPisoACeKatx2Pl4/B7k5MtJ86KTPBaR
> > 2WkAn0sd6bW9UDjCbdQ4pOLHW2WiFOg+
> > =NrkI
> > -----END PGP SIGNATURE-----
> >
> >
> > ------------------------------------------------------------------------
> > 好玩贺卡等你发，邮箱贺卡全新上线！
> > <http://cn.rd.yahoo.com/mail_cn/tagline/card/*http://card.mail.cn.yahoo.com/>
> >

|