POSKeyError Debugging and Fixing Cookbook

By Calvin Hendryx-Parker

January 18, 2019

Very rarely things can go sideways with your ZODB backing your application. We just got the opportunity to fix up an issue that happened due to cross ZODB mount point cut-and-paste. If you do this as a cut and not a copy, you will eventually end up packing the source mount point’s ZODB and the object that was cut will now no longer exist in that database. The only reference to it was pasted into a different mount point that is unaware that it is now missing.

If you find yourself in this position, you can fix it with a little digging at the Zope debug prompt.

There is a very nice tool called zc.zodbgc that can aid in this process, but it was tricky to get setup on this older Zope instance. There is documentation and if you can get this to work, it is a faster way to get the same thing done.

If that won’t work for you, you can still walk the ZODB structure and when you attempt to access one of these broken objects, you will get a POSKeyError and you will now have some clues as to which objects are broken and what type of object it was in a previous life.

Example Code to find the broken objects

With this snippet from the Plone Chix page, you can walk the database and report out broken objects for fixing afterwards.

from ZODB.POSException import POSKeyError
def error_finder(folder, exception=POSKeyError, stop_on_first=None):
    """ start at the given folderish object.
    If stop_on_first is true, quit after one exception;
    otherwise, keep going through the whole tree."""
    for id, next_item in folder.objectItems():
        try:
            next_item.getId()
        except exception:
            print `exception`, "in folder",
            print  '/'.join(folder.getPhysicalPath()),
            print "at id:", id
            if stop_on_first:
                raise "done"   # hack to break out of recursion
        else:
            # no error, recurse if it's objectManagerish
            if hasattr(next_item.aq_base, 'objectItems'):
                error_finder(next_item, exception, stop_on_first)

error_finder(app.Engr, stop_on_first=False)

You can now run this from the command line like this:

$ bin/instance run find_bad_oids.py

Code to fix and delete the broken objects in the ZODB

Now that you have your object(s) to fix. You can make a quick and dirty script to run against your databases. Make sure to test on your copy of your data first. I make no guarantee that this won’t eat your dog.

from ZODB.utils import p64
import cStringIO
import cPickle
import transaction

# This part is probably not needed unless you wanted to actually recreate
# the objects that were there vs deleting them. SimpleItem would probably
# work just as well.
from Products.HTMLDocument.HTMLDocument import HTMLDocument
from Products.PageTemplates.ZopePageTemplate import ZopePageTemplate

# A sequence of OID and Content Types to fix
# The walking script should have output the id of the broken item and
# the content type of the item if you wanted to fix it back to a previous state
bad_oids = [
    (0x6e85eb,HTMLDocument('homepage_feature_links')),
    (0x6e85ed, ZopePageTemplate('copy_of_index_html'))
]

for o, data in bad_oids:
    oid = p64(o)

    # This can be simplified using the code snippets from the zc.zodbgc docs
    s = app._p_jar.db()._storage
    file = cStringIO.StringIO()
    p = cPickle.Pickler(file, 1)
    p.dump((data.__class__, None))
    p.dump(data.__getstate__())
    t = transaction.Transaction()
    t.description = 'Fix POSKeyError'
    s.tpc_begin(t)
    s.store(oid,None,file.getvalue(),'',t)
    s.tpc_vote(t)
    s.tpc_finish(t)

# We just needed to get rid of the objects so we can now delete them the 
# standard Zope way.
folder = app.unrestrictedTraverse('/Engr/AboutUs/News/Spotlights')
folder.manage_delObjects([data.id for o,data in bad_oids])
transaction.commit()

You can now run this from the command line like this:

$ bin/instance run kill_bad_oids.py

Resources

https://plonechix.blogspot.com/2009/12/definitive-guide-to-poskeyerror.html — The definitely incomplete guide
https://www.mail-archive.com/zodb-dev@zope.org/msg02175.html — The theory of what to do, Chris’ post has the real howto
https://mail.zope.org/pipermail/zodb-dev/2008-February/011606.html — Post from Chris Withers with how to replace your bad object
http://www.zodb.org/en/latest/articles/multi-zodb-gc.html
https://github.com/zopefoundation/zc.zodbdgc/tree/0.6.1 — Older version for Zope 2

Thanks for filling out the form! A Six Feet Up representative will be in contact with you soon.

Blog