[OmniOS-discuss] 64/libjvm_db.so in runtime/java on bloody does not seem to be built correctly, pstack doesn't work

[Dale Ghent]

Thanks for reporting this. 

It does seem that when openjdk builds the binary tree, it does put the 32bit object in the location where the 64bit one should be. Why it's doing this is unknown to me after a quick glance at things, so we'll need to dig in and find out.

/dale

On Jul 11, 2013, at 4:15 PM, Christopher Siden <christopher.siden at delphix.com> wrote:

> Hi,
> I was recently playing around with the OpenJDK build in the bloody
> repo and noticed that pstack was only able to resolve Java stack
> frames when I used the 32-bit binary (/usr/bin/i86/pstack):
> 
> fa00f402 * java/lang/Thread.sleep(J)V+0
> fa00367b * org/apache/tomcat/util/net/JIoEndpoint$AsyncTimeout.run()V+13
> (line 294)
> 
> But the 64-bit pstack binary (/usr/bin/amd64/pstack) just prints
> question marks instead of class and method names:
> 
> fa0f08a6 ???????? (85f4a28, 0, 858a778, 0, 0, 0)
> fa0fae7c ???????? (0, a4d, 0, 4c3, 0, 9ca8ab8f)
> 
> Looking at the code in illumos's pstack.c it seems like pstack looks
> for a library called "libjvm.so" loaded into the target Java process,
> then looks for a library called "libjvm_db.so" in the same directory
> as libjvm.so and opens it with dlopen(3). libjvm_db.so provides pstack
> with the jvm-specific knowledge it needs to read java stack frames
> from the target process. When you are using a 64-bit pstack to debug a
> 32-bit java process it uses a special 64-bit version of the
> libjvm_db.so library stored in a sub directory called "64", the
> directory structure looks like this:
> 
> /usr/java/lib/i386/server/libjvm.so
> /usr/java/lib/i386/server/libjvm_db.so
> /usr/java/lib/i386/server/64/libjvm_db.so
> 
> Note that this 64-bit libjvm_db.so is still meant for debugging a
> 32-bit JVM, it's just compiled as a 64-bit library so that a 64-bit
> pstack process is capable of using it.
> 
> The problem seems to be that the 64-bit version of libjvm_db.so in the
> runtime/java package in bloody right now is actually a 32-bit library:
> 
> $ file /usr/java/lib/i386/server/64/libjvm_db.so
> /usr/java/lib/i386/server/64/libjvm_db.so: ELF 32-bit ...
> 
> So the call to dlopen(3) from a 64-bit pstack process fails. The
> 32-bit Java 7 JDK distributed by Oracle does not have this problem:
> 
> $ file /usr/java/jre/lib/i386/server/libjvm_db.so
> /usr/java/jre/lib/i386/server/libjvm_db.so: ELF 32-bit
> $ file /usr/java/jre/lib/i386/server/64/libjvm_db.so
> /usr/java/jre/lib/i386/server/64/libjvm_db.so: ELF 64-bit
> 
> I was able to use both 64-bit and 32-bit pstack to see the Java stack
> frames of 32-bit Java processes running with Oracle's binaries.
> 
> I downloaded and started building OpenJDK myself mimicing what it
> looks like the omni-build repository scripts do (I didn't run the
> scripts themselves). I let the build run until the 64-bit library was
> created and confirmed that it was in fact a 64-bit library:
> 
> $ file ./build/solaris/solaris_i486_compiler2/product/libjvm_db.so
> ./build/solaris/solaris_i486_compiler2/product/libjvm_db.so: ELF 32-bit
> $ file ./build/solaris/solaris_i486_compiler2/product/64/libjvm_db.so
> ./build/solaris/solaris_i486_compiler2/product/64/libjvm_db.so: ELF 64-bit
> 
> This seems like there is a problem with the way the 64-bit
> libjvm_db.so files currently in the runtime/java package were built,
> but I'm not familiar with how OmniTI builds those packages. Could it
> be they were built on a 32-bit system or something? Otherwise there
> might be some difference between OpenJDK and whatever repo Oracle
> bases their builds on?
> 
> Thanks,
> Chris
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss