Something they don’t tell you about PyMongo 3.0 and Multiprocessing.

EDIT: This post turned into a bug report over at the mongo python driver wiki, where it was confirmed to be a bug, and not a feature. Ultimately, the issue hasn’t been resolved yet, but version 3.0.4 will now throw a warning, preventing this issue from failing silently. Thanks to A. Jesse Jiryu Davis for suggesting I file it as a bug, and Anna Herlihy for the patch!

I had an interesting bug in a piece of software that I’ve been working on, that involves some heavy multithreading.  Running 18 processes simultaneously, of which at least 9 of them require some form of database interaction with MongoDB, is really not all that complicated… but I hit something that tossed in a wrench and confused me for 2 days.  What was it, you might ask?

Well, it looked like this:

 File "something.py", line 177, in flush
  b.execute()
File "/Users/afejes/sandboxes/pipeline4/lib/python2.7/site-packages/pymongo/bulk.py", line 582, in execute
  return self.__bulk.execute(write_concern)
File "/Users/afejes/sandboxes/pipeline4/lib/python2.7/site-packages/pymongo/bulk.py", line 430, in execute
  with client._socket_for_writes() as sock_info:
File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
  return self.gen.next()
File "/Users/afejes/sandboxes/pipeline4/lib/python2.7/site-packages/pymongo/mongo_client.py", line 663, in _get_socket
  server = self._get_topology().select_server(selector)
File "/Users/afejes/sandboxes/pipeline4/lib/python2.7/site-packages/pymongo/topology.py", line 121, in select_server
address))
File "/Users/afejes/sandboxes/pipeline4/lib/python2.7/site-packages/pymongo/topology.py", line 97, in select_servers
  self._error_message(selector))
ServerSelectionTimeoutError: No servers found yet

Basically, the new pymongo drivers (3.0.x) have changed their initialization, so that they no longer actually create the connection pool when you initialize them.  You say:

mongo = MongoClient()

and they go off and do a non-blocking initialization of everything pymongo needs to start the server. All is good.

However, if you’re doing multiprocessing, the temptation is to allow each of your threads to launch a new instance of the MongoClient. Indeed, I’ve done that before with 2.8.x series of pymongo, and it worked well. However, in this case, pymongo 3.0.2 REALLY doesn’t like it, and you’ll get the “No Servers found yet” error when you try to retrieve results from your database. Oddly enough, it’s especially hard to figure out because pymongo has one more hidden surprise for you: serverSelectionTimeoutMS.

You probably have never heard of this parameter, but it’s kinda important, now. It goes on your initialization of the MongoWapper:

self.mongo = MongoClient(mongo_url, mongo_port, serverSelectionTimeoutMS=500) 

If you don’t put it there, the default value is 30 seconds… Which means your application sits there, waiting to see if the mongo database will connect for 30 seconds, once it realizes that the database is missing. When it finally does fail, you’ll get the error above… 30 seconds after your database went down. That’s cool… except when the issue is actually not related to the database going down.

In my case, the issue was not that the database went down, but that each thread should not be initializing a new instance of MongoClient! The only solution: have the parent thread create one instance of MongoClient, and then pass that as a parameter to the processes. Tada! – the error disappears, and your program starts to run, instead of failing and waiting 30 seconds to tell you.

5 thoughts on “Something they don’t tell you about PyMongo 3.0 and Multiprocessing.

  1. Hi. PyMongo is developed by human beings just like you, who sincerely want to know when people encounter problems with our code. Would you do us the courtesy of opening a ticket at jira.mongodb.org, in the “PYTHON” project, and tell us more about what’s broken? It sounds like an unintentional change or bug in PyMongo 3.

    Thanks,
    Jesse

  2. Workaround for this issue. (pymongo 3.0.3)
    Pass “connect=False” in MongoClient object initialisation

    MongoClient(uri, connect=False)

    Or simply wait for few seconds before creating instance of MongoClient in the child process (like time.sleep(2)).

    • It’s true, but none of that was known when I wrote the article. If you want more information, there is some discussion of the issue on the bug report.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.