Recovering a corrupt LDAP DB on OS X Server

This weekend a problem arose with the OS X Server back home. I initially noticed it when I couldn't gain SSH access to the machine.

Using verbose output the SSH connection would reach the status of "entering interactive session" but would proceed no further.

When a user attempted to log on locally to the machine, even though the password and username were correct it would still display a logon failed message.

Using the local administrative account I determined that slapd wasn't starting up.

Log output was as below:

com.apple.launchd[1] (org.openldap.slapd[27382]) Exited with exit code: 1

com.apple.launchd[1] (org.openldap.slapd) Throttling respawn: Will start in 10 seconds

Using the terminal, I tested the saved slapd configuration and received the output as below:

administrator$ sudo /usr/libexec/slapd -Tt

overlay_config(): warning, overlay “dynid” already in list

overlay_config(): warning, overlay “dynid” already in list

overlay_config(): warning, overlay “dynid” already in list

overlay_config(): warning, overlay “dynid” already in list

overlay_config(): warning, overlay “dynid” already in list

bdb(dc=stuff,dc=com): PANIC: fatal region error detected; run recovery

bdb_db_open: Database cannot be opened, err -30978. Restore from backup!

bdb(dc=stuff,dc=com): DB_ENV->lock_id_free interface requires an environment configured for the locking subsystem

backend_startup_one: bi_db_open failed! (-30978)

slap_startup failed (test would succeed using the -u switch)

So there we go, OpenLDAP had become corrupted!

With the help of Google I encountered a helpful tip! As in linux, there is a LDAP recovery tool called "db_recover". Our saviour!

Sadly I had no backup previously ( :-s ) so I just had to run the recovery command, test the config and hope for the best!

administrator$ sudo db_recover -h /var/db/openldap/openldap-data/

administrator$ sudo /usr/libexec/slapd -Tt

overlay_config(): warning, overlay “dynid” already in list

overlay_config(): warning, overlay “dynid” already in list

overlay_config(): warning, overlay “dynid” already in list

overlay_config(): warning, overlay “dynid” already in list

overlay_config(): warning, overlay “dynid” already in list

config file testing succeeded

This was followed by a reboot of the machine, a reloading of the server processes when it had booted back up and one further restart. All is well with the world once more!

Note :: 

All help came from the following URL's. I'm just posting this to gather them all together!

http://discussions.apple.com/thread.jspa?messageID=8221483

http://www.prestonlee.com/2009/07/08/recovering-a-corrupt-openldap-database-on-osx-server/