Tuesday, March 29, 2011

AppDynamics Lite

I was recommended AppDynamics as a good tool to monitor the performance of a Java application without quite the hit that profilers (YourKit, JProfiler etc) have.

Environment:
  • Sun Sparc v9 5.10
  • Azul JVM v1.5.0_18
  • WebSphere 6.1.0.29


Issue 1

After putting the agent JVM parameter on the WebSphere application server (-javaagent:/tmp/AppServerAgentLite/javaagent.jar) and starting it up, I noticed there wasn't an agent log file and in the native_stdout.log there was the following:

[ERROR]: JavAgent - Could Not Start Java AgentException org.apache.commons.logging.LogConfigurationException: java.lang.ClassNotFoundException: org.apache.commons.logging.impl.Log4jFactory (Caused by java.lang.ClassNotFoundException: org.apache.commons.logging.impl.Log4jFactory)

This was caused by another JVM parameter, which was overriding the default log configuration of the agent:

-Dorg.apache.commons.logging.LogFactory=org.apache.commons.logging.impl.Log4jFactory

To remove this error, I removed the following files from the agent lib/tp directory:
  • commons-logging-1.1.1.jar
  • commons-logging-adapters-1.1.1.jar
  • commons-logging-api-1.1.1.jar
I replaced these with:
  • commons-logging-1.0.4.jar
  • commons-logging-api-1.0.4.jar
  • log4j-1.2.6.jar


Issue 2

Due to access restrictions I put the AppDynamics Lite agent in the /tmp directory. As I result I started receiving error messages such as these in the agent log file:

[AD Thread Pool-Global] 29 Mar 2011 14:17:26,134 WARN com.singularity.CONFIG.ConfigurationChannel - Could not connect to the controller/invalid response from controller, cannot get initialization information, controller host [127.0.0.1], port[8990], exception [Fatal transport error: Connection refused]

This was caused by the agent not having permission to connect to the viewer port on 8990 as it was not covered by the standard WAS policy files. Therefore, I added this additional policy to the app.policy file of the application server:

grant codeBase "file:/tmp/AppServerAgentLite/-" {
permission java.awt.AWTPermission "accessClipboard";
permission java.awt.AWTPermission "accessEventQueue";
permission java.awt.AWTPermission "showWindowWithoutWarningBanner";
permission java.lang.RuntimePermission "exitVM";
permission java.lang.RuntimePermission "loadLibrary";
permission java.lang.RuntimePermission "queuePrintJob";
permission java.net.SocketPermission "*", "connect";
permission java.net.SocketPermission "localhost:1024-", "accept,listen";
permission java.io.FilePermission "*", "read,write";
permission java.util.PropertyPermission "*", "read";
};


Issue 3

Once the agent had permission to connect to the socket on port 8990 I started receiving the following error in the agent log file:

[AD Thread Pool-Global] 29 Mar 2011 14:43:43,233 WARN com.singularity.CONFIG.ConfigurationChannel - Error refreshing configuration from controller, in reading responsecom.singularity.ee.rest.c: java.io.InvalidClassException: java.util.concurrent.locks.ReentrantLock$Sync; local class incompatible: stream classdesc serialVersionUID = -5179523762034025860, local class serialVersionUID = -9038320944266814859

I had started the LiteViewer using a different JVM (standard Sun JDK rather than Azul), which was caused this compatibility issue. I restarted the viewer using the Azul JVM including the necessary policy arguments:

-PX:PolicyVhost=host -PX:CPMDomain=domain

After this, the agent was able to communicate with the viewer successfully.