[OPEN-ILS-DEV] Trouble saving changes to org units

[Justin Hopkins]

I've been stumped by this problem since late last week. Although I've had some good help in IRC, I've still not figured out the cause.  Hopefully someone here can help.

When I try to make a change to an org unit in the staff client (Admin -> Server Administration -> Organizational Units), my changes are not saved. From the point of view of the staff client nothing happens at all. There is no change to the UI at all - it just fails silently. The logs look like this when I hit the 'Save' button:

==> gateway.log <==
osrf_http_translator 2012-06-25 11:10:41 [ERR :9057:osrf_cache.c:63:134063968790573] Failed to cache key:value [0.73571041987615081340640626406]:[{"ip":"66.112.104.91","jid":"opensrf at private.localhost/open-ils.pcrud_drone_mig_1340637872.246171_1755","service":"open-ils.pcrud"}] - WRITE FAILURE
osrf_http_translator 2012-06-25 11:10:41 [INFO:9057:./osrf_http_translator.c:530:134063968790573] translator resulted in status 0
osrf_http_translator 2012-06-25 11:10:41 [ERR :9057:./osrf_http_translator.c:231:134063968790574] attempt to send directly to opensrf at private.localhost/open-ils.pcrud_drone_mig_1340637872.246171_1755 without a session
osrf_http_translator 2012-06-25 11:10:41 [INFO:9057:./osrf_http_translator.c:530:134063968790574] translator resulted in status 400

==> osrfsys.log <==
open-ils.pcrud 2012-06-25 11:10:47 [INFO:1755:osrf_prefork.c:460:134063968790573] No request was received in 6 seconds, exiting stateful session
open-ils.pcrud 2012-06-25 11:10:47 [INFO:1755:osrf_app_session.c:1017:134063968790573] [open-ils.pcrud] sent 182 bytes of data to opensrf at public.localhost/translator_mig_1340639687.624630_9057

This is happening both on my standalone migration server and on my production system. Production has a bare metal database server (that's running memcached), 6 "app servers" running apache and evergreen and a couple load balancers in front. The VM for the migration server was started as a clone from one of the production app servers (which were all clones of the first app server) so whatever this problem is it's affecting all of my systems. FWIW, I did apt-get purge ejabberd and reinstall for each server.

I'm running postgres9.1 and rel_2_2_0.

Thanks,
Justin Hopkins