Dependent module libclntsh.a(shr.o) could not be loaded

I've found this error "Dependent module libclntsh.a(shr.o) could not be loaded" on many occasions, but usually only when the Oracle client or version in general had been changed. And often in conjuction with a faling "magic number". Magic being somewhat non-descriptive, when they only mean 32 or 64 bit.... But nevermind teasing the developers ;)

Usually, it's caused by the incorrect setting of the LIBPATH (for AIX), SHLIB_PATH (for HP-UX) or similar library path.
On occasion I've even had to relink my Oracle to get the libraries running correctly again.

So let me just set things straight:
On an SAP system, the user ORA<SID> needs to have the vaiable set to include <path to SYS/exe/run> and /oracle/<SID>/<version>/lib (and remember to differentiate between 32 and 64 bit libraries, if you're using both).
The user <SID>ADM needs the varibale to point to /oracle/client/<version>/instantclient (assuming you're on Oracle version 10 or newer and thus using the instantclient)

The other day, however, I did notice problems that neither spelling errors, incorreclty set variable or even a relink couldn't fix. And it took me a while to find the cause of the problem. The system had been unicode converted, oracle upgraded, had a new client installed, and even got an SAP enhancement package installed. So there were a lot of stuff that could have gone wrong. But for some reason a few (but not all) of the systems would throw the error "Dependent module libclntsh.a(shr.o) could not be loaded" when I tried running brconnect, brspace or other similar calls. Whenever the caller was the <SID>ADM it wouldnt work....

I struggled for a while, especially because the shr.o reference (rather than shr_64.o) would seem to indicate that a 32 bit module was being attempted loaded. And since I was running Oracle 11.2 64 bit, then ofcourse there'd be no 32 bit library available. But I followed the traces around, and finally I found the culprit. In the instantclient directory, there was no softlink named instantclient. There was only the instantclient_<version> directory, and the LIBPATH pointed directly to this. That meant that any test I would subject it to would pass, but for some reason there must be someting hardcoded in the SAP kernel that points to the instastantclient directory/softlink despite setting the variable LIBPATH.....

I quickly created a new softlink, and lo and behold. Everything was working like a charm again. On the upside I got to clean up all the old .dbenv files and obsolete environment variables on the failing systems ;)