This is one of my most dreaded tasks … one that’s not exactly easy in a complex, high distributed and highly concurrent server based system like ours. None of the normal profiling tools really help it seems …
- We have commercial license to JProfiler. It works fine under functional scenarios. Crashes under linux even under very modest load (2-3 calls).
- We have tried BEA JRA – a wonderful tool that has helped a bit. But…like a lot of things, you have to pay beyond 1 hour snapshots.
- HPROF – ok, doesn’t work too well, hard to customize without reverting to coding in JNI and the makefiles require porting … argh
- YourKit – another nice tool, but doesn’t seem to work well under JRockit
Another option. Looking at two snapshots (one call scenario that leaks, one that doesn’t) and comparing the two JRA outputs after a GC, it seems like we are growing HashMap$Entry and String/char[] objects. Surprise, Surprise! Growing by like hundreds of thousands .. what’s going on?
So, we just want to know, after a GC and leak, what’s in the darn Maps? Why is this so hard?
OK, after thinking about different options … I resorted to a simple, semi-non invasive method.
JDK Patch
I patched HashMap class:
- extract src.zip and copy java/util/HashMap.java and patch it
- compile it into a standalone folder
- extract jrockit.jar into temp folder (second try, I did rt.jar at first which is wrong if using jrockit)
- copy class files (all of them) into the jrockit temp folder
- re-jar the jrockit.jar (back it up first!)
- drop in JSP controller page (in this case, I’m running inside jboss)
- start jboss
Now you can navigate to the controller page and start/stop collection on demand, as well as request a GC or list this current entries file.
Ruby to the rescue
I had to use Ruby for something, of course. I wrote a simple ruby script to parse out my collections file and re-create the current entries.
The entries are re-created as:
==================================================
HashMap@114059333364119280909 => created by…
net.vocalocity.common.util.xml.AbstractXmlElement.
net.vocalocity.browser.vxml.model.AbstractElement.
net.vocalocity.browser.vxml.element.PCDataElement.
net.vocalocity.browser.vxml.parser.VoiceXMLPre2Builder.createPCDataElement(VoiceXMLPre2Builder.java:77)
net.vocalocity.common.util.xml.SchemaParser.newTextNode(SchemaParser.java:585)
net.vocalocity.common.util.xml.SchemaParser.endElement(SchemaParser.java:561)
==================================================
The above entry basically is a strongly reachable HashMap with no entries. Hmmm..
==================================================
HashMap@114059333362610715406 => created by…
net.vocalocity.browser.vxml.extension.VoiceXMLCallAdapter.addCallInfoParams(VoiceXMLCallAdapter.java:309)
net.vocalocity.browser.vxml.extension.VoiceXMLCallAdapter.
net.vocalocity.browser.vxml.extension.VoiceXMLController.prepareDialog(VoiceXMLController.java:347)
net.vocalocity.common.domain.dialog.Dialog$DialogAdapter.performStart(Dialog.java:667)
net.vocalocity.common.util.jmx.component.AbstractComponent.start(AbstractComponent.java:315)
net.vocalocity.common.domain.dialog.Dialog$DialogAdapter.prepareDialog(Dialog.java:634)
-134605059==>session.connection.local.port=11
622566888==>session.connection.local.channelid=VocalOS SIP Line 11
902589610==>session.connection.remote.uri=sip:Jeff%20Haynie@11.99.12.13:5560
-49624567==>session.connection.local.uri=sip:jhaynie@jhaynie:5960
==================================================
This entry is a HashMap with 4 entries … the key/value pairs objects string representation is printed out.
… When I have some time, I’ll post my scripts – maybe some other poor old chap can benefit in some small way.
Popularity: 9% [?]
Other popular articles you might enjoy:
If you enjoyed this post, make sure you subscribe to my RSS feed!
