Someone who I considered to be a friend suggested that I might benefit from applying the latest PSU taking my Exalogic to 2.0.6.0.2, so following this advice I set about the necessary procedure, 3 days later having taken lots of precautions, like headache tablets, caffeine and the unavoidable backups I rewarded myself with a pat on the back and some celebration beer and signed off thinking that all was well.
The next morning I woke with a renewed vigor, eager to march on and make full use of all those CPU's and processing cycles, so I set about creating some new VM's, that's when my head hit the desk in despair again. So I tried a different template, same problem, by now I'm thinking WTH did I bother, should of stayed on 2.0.6.0.0.
The way I figured it I had three choices;
- Commit hari kari;
- Use the backup and recover the the last known position; or
- Soilder on to the finish line
Obviously option 1 was a bit extreme and option 2 could have gone equally horribly wrong so I went for my usual option of unavoidably hacking my way to the finish line.
After several hours of trawling MOS I stumbled on;
- Bug 18161671 : POST UPGRADE VSERVER CREATE FAILS WITH OVM REPOSITORY NOT SET IN TASK PARAMETERS
- Bug 18229800 : ERROR DURING VIRTUAL MACHINE CREATION: OVM REPOSITORY NOT SET IN TASK PARAMETER
- Bug 18254689 : OVM REPOSITORY NOT SET IN TASK PARAMETERS. (60391)
As is all too often the case the out look didn't look good, varying degrees of "you're fecked" and "could not reproduce" all drawing a blank. No to be deterred I marched on.
I spotted a glimmer of hope though, one of the bugs suggested that I could verify the problem by looking at my cache;
To do this I needed to;
- navigate to https://<ECIP>/xvm/#StorageRepository
- authenticate as root user
- open the link under the Name column which should look similar to NORM-sr$///0004fb00000300005cacceasd1c1ede
- then find the section "Attributes"
- followed by the subsection "StorageRepository" and check the attributes.
BINGO! null values.
So now what! Apparently I need to refresh my cache in EMOC by starting it in development mode. Great but how the dickens do you do that !
Calling in the cavalry from the OpsCenter team I started EMOC in debug mode;
#/opt/sun/xvmoc/bin/ecadm stop
#/opt/sun/cacao2/bin/cacaoadm set-param -i oem-ec
java-flags="-Xms200M -Xmx8192M -server -XX:StringTableSize=27001
-XX:PermSize=128m -XX:MaxPermSize=384m -Xss256k -XX:+UseParallelOldGC
-XX:SoftRefLRUPolicyMSPerMB=10000 -XX:-UseCompressedOops
-Dsun.security.pkcs11.enable-solaris=false
-Djava.endorsed.dirs=/opt/sun/cacao2/share/lib/endorsed -Dxvmserver=false
-Dcom.sun.cacao.debug"#/opt/sun/xvmoc/bin/ecadm start
BANGO, Revisiting the domain model in OpsCenter using https://<ECIP>/xvm/#StorageRepository I now have access to refresh the cache.
Going to the bottom of the page and I clicked on the "Cache Refresh" button;
and then I clicked on the magic "selfHeal" button;
BONGO, we're back in business.
