Issue with pythonmodule

Tom_Hendrikx · July 21, 2011, 11:39pm

Hi,

When trying to add a python script to unbound, I receive some errors
(tested with both 1.4.10 and 1.4.12):

Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named unboundmodule
[1311289662] unbound-checkconf[7398:0] error: pythonmod: cannot
initialize core module: unboundmodule.py
[1311289662] unbound-checkconf[7398:0] fatal error: bad config for
python module

Testing by hand gives a similar output:

$ sudo /usr/sbin/unbound -dv -c /etc/unbound/unbound.conf
[1311289789] unbound[7421:0] notice: Start of unbound 1.4.12.
[1311289789] unbound[7421:0] warning: increased limit(open files) from
1024 to 4152
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named unboundmodule

After some debugging I realised that unbound is unable to find the
unboundmodule.py file that is placed in
/usr/lib64/python2.7/site-packages. I was able to temporarily resolve
this by adding a symlink to the python file in /etc/unbound/, but this
is of course a hack.

Since the file is placed in /usr/lib64/python2.7/site-packages/ by the
Makefile, it looks as if there is no special adaption done by the distro
package. This is why I am asking my question here first

Any ideas as to why this is not working out of the box?

PaulWouters · July 22, 2011, 3:07am

Thanks for reporting this.

I can confirm this problem is present in the fedora/rhel/centos package.
Though for me, on x86_64 it is placed in:

/usr/lib/python2.7/site-packages/unboundmodule.py

I can confirm linking it in /etc/unbound resolves this.

I changed the %configure from using --with-pythonmodule to use:

--with-pythonmodule=%{python_sitearch}

which expands to:

--with-pythonmodule=/usr/lib64/python2.7/site-packages

but it seems unbound still places the files in the wrong location when
running "make install" and places the sitearch files in the sitelib
directory

Makefile.inc shows:

$(INSTALL) -c -m 644 pythonmod/unboundmodule.py $(DESTDIR)$(PYTHON_SITE_PKG)/unboundmodule.py

PYTHON_SITE_PKG is set to:

python -c "import distutils.sysconfig; print distutils.sysconfig.get_python_lib(0,0);"
/usr/lib/python2.7/site-packages

However, the unbound daemon does not seem to look there......

Note that /usr/lib/python2.7/site-packages/_unbound.so.2.12.0 also seems to be in the 32 instead of
the 64 bit location....

A grep shows me :

[paul@bofh unbound-1.4.12rc1]$ grep get_python_lib *
acx_python.m4: print '-L' + get_python_lib(0,1), \
acx_python.m4: '-L' + os.path.dirname(get_python_lib(0,1)), \
acx_python.m4: print distutils.sysconfig.get_python_lib(0,0);"`
configure: print '-L' + get_python_lib(0,1), \
configure: '-L' + os.path.dirname(get_python_lib(0,1)), \
configure: print distutils.sysconfig.get_python_lib(0,0);

The first argument to get_python_lib is a boolean for plat_specific=
If I change that to 1, then it seems the files are installed in their
right locations:

[root@bofh unbound]# rpm -ql unbound-python
/usr/lib64/python2.7/site-packages/_unbound.so
/usr/lib64/python2.7/site-packages/_unbound.so.2
/usr/lib64/python2.7/site-packages/_unbound.so.2.12.0
/usr/lib64/python2.7/site-packages/unbound.py
/usr/lib64/python2.7/site-packages/unbound.pyc
/usr/lib64/python2.7/site-packages/unbound.pyo
/usr/lib64/python2.7/site-packages/unboundmodule.py
/usr/lib64/python2.7/site-packages/unboundmodule.pyc
/usr/lib64/python2.7/site-packages/unboundmodule.pyo

but it seems the python module still fails to initialise:

Jul 21 22:59:47 libunbound[846:0] notice: init module 1: python
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named unboundmodule
Jul 21 22:59:47 libunbound[846:0] error: pythonmod: cannot initialize core module: unboundmodule.py
Jul 21 22:59:47 libunbound[846:0] error: module init for module python failed
resolve error: initialization failure

The search path in strace of unbound-host shows:

[root@bofh unbound]# grep unboundmodule out
stat("/usr/lib64/python2.7/unboundmodule", 0x7fff16839290) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/unboundmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/unboundmodulemodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/unboundmodule.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/unboundmodule.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/plat-linux2/unboundmodule", 0x7fff16839290) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/unboundmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/unboundmodulemodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/unboundmodule.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/unboundmodule.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/lib-dynload/unboundmodule", 0x7fff16839290) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/unboundmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/unboundmodulemodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/unboundmodule.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/unboundmodule.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("./unboundmodule", 0x7fff16839290) = -1 ENOENT (No such file or directory)
open("./unboundmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./unboundmodulemodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./unboundmodule.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./unboundmodule.pyc", O_RDONLY) = -1 ENOENT (No such file or di

and then finally /etc/unbound/

I'm not sure yet why it is not searching in /usr/lib64/python2.7/site-packages/ first

I also think that the value for --with-pythonmodule seems to be lost during make install

Paul

Tom_Hendrikx · July 22, 2011, 11:10am

ImportError: No module named unboundmodule

After some debugging I realised that unbound is unable to find the
unboundmodule.py file that is placed in
/usr/lib64/python2.7/site-packages. I was able to temporarily resolve
this by adding a symlink to the python file in /etc/unbound/, but this
is of course a hack.

Since the file is placed in /usr/lib64/python2.7/site-packages/ by the
Makefile, it looks as if there is no special adaption done by the distro
package. This is why I am asking my question here first

Thanks for reporting this.

I can confirm this problem is present in the fedora/rhel/centos package.
Though for me, on x86_64 it is placed in:

/usr/lib/python2.7/site-packages/unboundmodule.py

I can confirm linking it in /etc/unbound resolves this.

I changed the %configure from using --with-pythonmodule to use:

     --with-pythonmodule=%{python_sitearch}

which expands to:

    --with-pythonmodule=/usr/lib64/python2.7/site-packages

but it seems unbound still places the files in the wrong location when
running "make install" and places the sitearch files in the sitelib
directory

Makefile.inc shows:

        $(INSTALL) -c -m 644 pythonmod/unboundmodule.py
$(DESTDIR)$(PYTHON_SITE_PKG)/unboundmodule.py

PYTHON_SITE_PKG is set to:

python -c "import distutils.sysconfig; print
distutils.sysconfig.get_python_lib(0,0);"
/usr/lib/python2.7/site-packages

However, the unbound daemon does not seem to look there......

Note that /usr/lib/python2.7/site-packages/_unbound.so.2.12.0 also seems
to be in the 32 instead of
the 64 bit location....

I'm no expert on this, but AFAIK, the /usr/lib/ contents depends on the
arch of the host (i.e. on x86_64 it will contain 64-bit libraries).On
gentoo x86_64, there are /usr/lib32 and /usr/lib64 locations, with
/usr/lib being a symlink to the latter.

Unbound searching in /usr/lib/ would work for me, and seems reasonable.

A grep shows me :

[paul@bofh unbound-1.4.12rc1]$ grep get_python_lib *
acx_python.m4: print '-L' + get_python_lib(0,1), \
acx_python.m4: '-L' +
os.path.dirname(get_python_lib(0,1)), \
acx_python.m4: print
distutils.sysconfig.get_python_lib(0,0);"`
configure: print '-L' + get_python_lib(0,1), \
configure: '-L' +
os.path.dirname(get_python_lib(0,1)), \
configure: print
distutils.sysconfig.get_python_lib(0,0);

The first argument to get_python_lib is a boolean for plat_specific=
If I change that to 1, then it seems the files are installed in their
right locations:

[root@bofh unbound]# rpm -ql unbound-python
/usr/lib64/python2.7/site-packages/_unbound.so
/usr/lib64/python2.7/site-packages/_unbound.so.2
/usr/lib64/python2.7/site-packages/_unbound.so.2.12.0
/usr/lib64/python2.7/site-packages/unbound.py
/usr/lib64/python2.7/site-packages/unbound.pyc
/usr/lib64/python2.7/site-packages/unbound.pyo
/usr/lib64/python2.7/site-packages/unboundmodule.py
/usr/lib64/python2.7/site-packages/unboundmodule.pyc
/usr/lib64/python2.7/site-packages/unboundmodule.pyo

but it seems the python module still fails to initialise:

Jul 21 22:59:47 libunbound[846:0] notice: init module 1: python
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named unboundmodule
Jul 21 22:59:47 libunbound[846:0] error: pythonmod: cannot initialize
core module: unboundmodule.py
Jul 21 22:59:47 libunbound[846:0] error: module init for module python
failed
resolve error: initialization failure

The search path in strace of unbound-host shows:

[root@bofh unbound]# grep unboundmodule out
stat("/usr/lib64/python2.7/unboundmodule", 0x7fff16839290) = -1 ENOENT
(No such file or directory)
open("/usr/lib64/python2.7/unboundmodule.so", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/usr/lib64/python2.7/unboundmodulemodule.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/lib64/python2.7/unboundmodule.py", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/usr/lib64/python2.7/unboundmodule.pyc", O_RDONLY) = -1 ENOENT (No
such file or directory)
stat("/usr/lib64/python2.7/plat-linux2/unboundmodule", 0x7fff16839290) =
-1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/unboundmodule.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/unboundmodulemodule.so",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/unboundmodule.py", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/unboundmodule.pyc", O_RDONLY) =
-1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/lib-dynload/unboundmodule", 0x7fff16839290) =
-1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/unboundmodule.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/unboundmodulemodule.so",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/unboundmodule.py", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/unboundmodule.pyc", O_RDONLY) =
-1 ENOENT (No such file or directory)
stat("./unboundmodule", 0x7fff16839290) = -1 ENOENT (No such file or
directory)
open("./unboundmodule.so", O_RDONLY) = -1 ENOENT (No such file or
directory)
open("./unboundmodulemodule.so", O_RDONLY) = -1 ENOENT (No such file or
directory)
open("./unboundmodule.py", O_RDONLY) = -1 ENOENT (No such file or
directory)
open("./unboundmodule.pyc", O_RDONLY) = -1 ENOENT (No such file or di

and then finally /etc/unbound/

I'm not sure yet why it is not searching in
/usr/lib64/python2.7/site-packages/ first

I also think that the value for --with-pythonmodule seems to be lost
during make install

I agree this is the problem: unbound never checks the site-packages/
directory. I did the same tests here, even with passing the correct path
to ./configure using --with-pythonmodule, but unbound never goes there.

Passing the path to --with-pythonmodule is undocumented (as far as
./configure --help concerns), but since make install seems to figure out
the (correct?) install location in site-packages/, it seems to me that
passing in the path to ./configure is not really necessary. But it would
be logical that the rest of the unbound package would use this path too
when needed, which is currently not the case.

Tom_Hendrikx · August 8, 2011, 7:39pm

I did some more research on this and I think I found the culprit.
Attached is a patch against unbound 1.4.12 that adds the site-packages
dir to the search path when initialising the python module.

With this patch, things Work For Me (tm)

(attachments)

pythonmod_check_site-packages_dir.patch (549 Bytes)

PaulWouters · August 8, 2011, 10:02pm

I can confirm this fixes the pythonmod loading! Thanks! I've built it
for fedora rawhide, and over the next few days should trickly up in all
branches.

Wouter: Please apply patch to git^Wsvn

We're still seeing installs in the wrong dir though:

[paul@bofh unbound]$ rpm -ql unbound-python
/usr/lib/python2.7/site-packages/_unbound.so
/usr/lib/python2.7/site-packages/_unbound.so.2
/usr/lib/python2.7/site-packages/_unbound.so.2.12.0
/usr/lib/python2.7/site-packages/unbound.py
/usr/lib/python2.7/site-packages/unbound.pyc
/usr/lib/python2.7/site-packages/unbound.pyo
/usr/lib/python2.7/site-packages/unboundmodule.py
/usr/lib/python2.7/site-packages/unboundmodule.pyc
/usr/lib/python2.7/site-packages/unboundmodule.pyo

That should be /usr/lib64/ not /usr/lib/

Paul

Wouter · August 9, 2011, 7:45am

Hi,

I did some more research on this and I think I found the culprit.
Attached is a patch against unbound 1.4.12 that adds the site-packages
dir to the search path when initialising the python module.

With this patch, things Work For Me (tm)

I can confirm this fixes the pythonmod loading! Thanks! I've built it
for fedora rawhide, and over the next few days should trickly up in all
branches.

Wouter: Please apply patch to git^Wsvn

okay. (yes git is fashionable, but svn works, and featureset of git is
not necessary, this is not the linux kernel, this is the only patch in
months ). The patch has been committed, thanks Tom!

We're still seeing installs in the wrong dir though:

[paul@bofh unbound]$ rpm -ql unbound-python
/usr/lib/python2.7/site-packages/_unbound.so
/usr/lib/python2.7/site-packages/_unbound.so.2
/usr/lib/python2.7/site-packages/_unbound.so.2.12.0
/usr/lib/python2.7/site-packages/unbound.py
/usr/lib/python2.7/site-packages/unbound.pyc
/usr/lib/python2.7/site-packages/unbound.pyo
/usr/lib/python2.7/site-packages/unboundmodule.py
/usr/lib/python2.7/site-packages/unboundmodule.pyc
/usr/lib/python2.7/site-packages/unboundmodule.pyo

That should be /usr/lib64/ not /usr/lib/

Paul

in acx_python.m4 it does this:

Tom_Hendrikx · August 9, 2011, 9:42am

It seems to me that the path that is appended in my patch should be the
same as the one used in the Makefile. Probably you should update the
patch when it is clear what the correct path would be, depending on the
input of Paul.

On my system, the Makefile test returns
'/usr/lib/python2.7/dist-packages', which works because the distro
symlinks /usr/lib to /usr/lib64 on 64-bit systems.

Python docs state that distutils.sysconfig.get_python_lib(1,0) would
return the platform dependant path. I'm curious to see whether this
includes /usr/lib64 on Paul's system...

PaulWouters · August 9, 2011, 2:25pm

import distutils.sysconfig
distutils.sysconfig.get_python_lib(1,0)
'/usr/lib64/python2.7/site-packages'

Paul

Wouter · August 10, 2011, 6:06am

Hi Paul, Tom,