Python’s module search path

Upgrading software on an Ubuntu 12.04 machine, I hit a snag. A daemon wouldn’t start. The error log showed the following:

from util.pystone import pystones
ImportError: No module named util.pystone

Reading a little, pystone is used in benchmarking and comes with Python 2.7. Yep, it’s there.

$ ls /usr/lib/python2.7/test
__init__.py __init__.pyc pystone.py pystone.pyc regrtest.py regrtest.pyc test_support.py test_support.pyc

Strange. Firing up python and importing this from the python prompt failed.

I tried only specifying this path to see if it would import:

$ python
Python 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path=['/usr/lib/python2.7']
>>> print sys.path
['/usr/lib/python2.7']
>>> import test.pystone
>>>

At this point I suspected something in Python’s module search path was fucking me.

A little more messing around I found a good one.

$ python
Python 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.sort()
>>> import test.pystone
>>>

and

$ python
Python 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.reverse()
>>> import test.pystone
>>>

Okay something is matching in the search path before it get’s to /usr/lib/python2.7 which is about half way through the sys.path. So, I did what anybody in my shoes would do and removed them one by one.

>>> sys.path.reverse()
>>> print sys.path.pop()
/usr/lib/pymodules/python2.7
>>> sys.path.reverse()
>>> import test.pystone Traceback (most recent call last):
File "", line 1, in
ImportError: No module named pystone

Until I got to an empty list.

>>> print sys.path.pop()
/usr/lib/python2.7
>>> print sys.path
[]
>>> sys.path.append('/usr/lib/python2.7')
>>> print sys.path
['/usr/lib/python2.7']
>>> import test.pystone
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named pystone

wat
Immediately, I assumed I was losing my mind. Of course, I should check that’s the case.
$ python
Python 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import test.pystone
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named pystone
>>> sys.path=['/usr/lib/python2.7']
>>> import test.pystone
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named pystone

Around this point I thought I was crazy and/or had a flaw in my testing process. Then, it dawned on me. If I call test.pystone before zeroing out the sys.path it WON’T find it ever.

Well, shit.

I wrote a BASH script to iterate over the list removing one at a time and testing the import:

#!/bin/bash
FUCK=" /usr/local/lib/python2.7/dist-packages/pyserial-2.6-py2.7.egg /usr/local/lib/python2.7/dist-packages/simplejson-3.3.0-py2.7-linux-x86_64.egg /usr/local/lib/python2.7/dist-packages/gdata-2.0.18-py2.7.egg /usr/local/lib/python2.7/dist-packages/todoist-0.0.1-py2.7.egg /usr/local/lib/python2.7/dist-packages/python_twitter-1.1-py2.7.egg /usr/local/lib/python2.7/dist-packages/requests_oauthlib-0.4.0-py2.7.egg /usr/local/lib/python2.7/dist-packages/oauthlib-0.6.0-py2.7.egg /usr/local/lib/python2.7/dist-packages/gmusicapi-3.1.1_dev-py2.7.egg /usr/local/lib/python2.7/dist-packages/appdirs-1.3.0-py2.7.egg /usr/local/lib/python2.7/dist-packages/mock-1.0.1-py2.7.egg /usr/local/lib/python2.7/dist-packages/oauth2client-1.2-py2.7.egg /usr/local/lib/python2.7/dist-packages/proboscis-1.2.6.0-py2.7.egg /usr/local/lib/python2.7/dist-packages/validictory-0.9.3-py2.7.egg /usr/local/lib/python2.7/dist-packages/httplib2-0.9-py2.7.egg /usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg /usr/local/lib/python2.7/dist-packages/walter-0.1-py2.7.egg /usr/local/lib/python2.7/dist-packages/geeknote-0.2a-py2.7.egg /usr/local/lib/python2.7/dist-packages/thrift-0.9.1-py2.7-linux-x86_64.egg /usr/local/lib/python2.7/dist-packages/markdown2-2.3.0-py2.7.egg /usr/local/lib/python2.7/dist-packages/html2text-2014.9.25-py2.7.egg /usr/local/lib/python2.7/dist-packages/oauth2-1.5.211-py2.7.egg /usr/lib/python2.7 /usr/lib/python2.7/plat-linux2 /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages/PIL /usr/lib/python2.7/dist-packages/gst-0.10 /usr/lib/python2.7/dist-packages/gtk-2.0 /usr/lib/pymodules/python2.7 /usr/lib/python2.7/dist-packages/ubuntu-sso-client"

for ITEM in $FUCK
do
echo $ITEM
python -c “import sys; sys.path.remove(‘$ITEM’); import test.pystone”
done

This did it. I found the package: html2text-2014.9.25-py2.7.egg

$ cd /usr/local/lib/python2.7/dist-packages/html2text-2014.9.25-py2.7.egg
$ ls
EGG-INFO html2text test

It has a ‘test’ directory and was earlier in the search path.

pip uninstall html2text

And away we go! What a lovely way to spend an evening…

dirty python network graphite hacks

I’ve been pushing historical temperature data for Vancouver into a local graphite instance using python. While pushing the data, I received this error:
socket.error: [Errno 99] Cannot assign requested address
This was strange, as I had been using the address in the loop many, many, MANY times before hitting this error. A little more sleuthing uncovered a ton of connections in the TIME_WAIT state. Around 26,000 open sockets was where I couldn’t open any more.

This little gem, was the fix:
sock.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER,struct.pack('ii', 1, 0))
sock.close()

Normally, after cleanly closing the connecting in python, the system will still hold the connection open for a minute or two in the TIME_WAIT state. This was causing port exhaustion. The above lines told python to send a RST (reset) which will immediately drop the connection.

Graphite complains about the reset, but does log the data.
Connection to the other side was lost in a non-clean fashion.
Perhaps, not the most elegant solution, but it served my purpose.