Diagnostic Tools for Java Performace on App Service Linux

5 minute read | By Kendrick Dubuisson

When dealing with High CPU, High Memory, or general performance scenarios on App Service Linux, it’s recommended to profile the application locally, but this may not always be possible nor reflect the same results. This may be due to the difference in load or environment configuration.

The App Service Linux pre-built images for Java include the various JDK tools for profiling & this article will walk through the steps required for collecting traces on App Services.

As a pre-requisite, assure persistent storage is enabled on your App Service. To enable it, set the WEBSITES_ENABLE_APP_SERVICE_STORAGE app setting value to true.

You can select which instance you’d like to console into if scaled out to multiple instances before collecting your trace. This is helpful if a performance issue has been narrowed down to a particular instance.

https://<app-name>.scm.azurewebsites.net/newui# Java Perfomance

Heap Dumps - High Memory

Heaps dumps provide us with a snapshot of all the heaps live objects in use at a specific moment which is useful when troubleshooting memory leaks or understanding resource constraints.

  1. Detemine the PID using jcmd.

     # Example Output

 root@4ddc09ac647f:/# jcmd

 Picked up JAVA_TOOL_OPTIONS: -Xmx4865M -Djava.net.preferIPv4Stack=true 
 42 org.apache.catalina.startup.Bootstrap start
 223 jdk.jcmd/sun.tools.jcmd.JCmd

  2. Collect your heap dump using jmap. $JAVA_HOME/bin/jmap -dump:format=b,file=/home/LogFiles/dump1.hprof <pid>

     # Example Output

 root@4ddc09ac647f:/# $JAVA_HOME/bin/jmap -dump:format=b,file=/home/LogFiles/dump1.hprof 42

 Picked up JAVA_TOOL_OPTIONS: -Xmx4865M -Djava.net.preferIPv4Stack=true 
 Dumping heap to /home/LogFiles/dump1.hprof ...
 Heap dump file created [53217296 bytes in 0.209 secs]
  3. The dump file will be created at /home/Logfiles/dump1.hprof, which can be downloaded using FTP, KUDU API, or Kudu File Manager.

    https://<webapp-name>.scm.azurewebsites.net/newui/fileManager#

  4. Eclipse Memory Analyzer can then be downloaded locally to analyze heap dumps. Below we can see the dominator tree of a heap dump showing the objects of my retained heap.

    Java Perfomance

Thread Dumps - High CPU

Thread dumps provide us with a snapshot of all the threads executing at a specific moment. This includes information like the state of the threads, stack trace associated & reference addresses. We’ll be using jStack to collect our thread dumps which can help us identify any blocked threads or running threads.

It’s a best practice to collect at least three thread dumps. Each thread dump should be taken within 10-second intervals of one another to account for thread state changes.

  1. Detemine the PID using jcmd.

     # Example Output

 root@adb27c7aa8af:/# jcmd

 Picked up JAVA_TOOL_OPTIONS: -Xmx4865M -Djava.net.preferIPv4Stack=true 
 74 org.springframework.boot.loader.JarLauncher
 191 jdk.jcmd/sun.tools.jcmd.JCmd

  2. Collect your thread dump using jstack. $JAVA_HOME/bin/jstack <pid> > /home/LogFiles/threaddump1.txt

     # Example Output

 root@adb27c7aa8af:/# $JAVA_HOME/bin/jstack 74 > /home/LogFiles/threaddump1.txt

 Picked up JAVA_TOOL_OPTIONS: -Xmx4865M -Djava.net.preferIPv4Stack=true
  3. The thread dump will be created at /home/Logfiles/threaddump1.txt, which can be downloaded using FTP, KUDU API, or Kudu File Manager.

    https://<webapp-name>.scm.azurewebsites.net/newui/fileManager#

  4. Thread Dumps can be reviewed manually or with any of the thread analyzers mentioned below. At the same time, thread dumps contain application data, and local or client-side analyzers can be used to avoid leaking any potentially sensitive information.

    Java Perfomance

Java Flight Recorder - Profiling

Java Flight Recorder (JFR) is a monitoring tool that collects information about the events in a Java Virtual Machine (JVM) while the application is running.

JFR is designed to have minimal perforamce impact & continuously saves large amounts of data about the running system. This profiling information includes thread samples (which show where the program spends its time), lock profiles, and garbage collection details.

Depending on the scenario & if the issue is reproducible, recordings can be manually started or ran continuously at startup.

Flight Recordings can be reviewed using Zulu Mission Control

Java Perfomance

Timed Recordings

  1. Detemine the PID using jcmd.

     # Example Output

 root@adb27c7aa8af:/# jcmd

 Picked up JAVA_TOOL_OPTIONS: -Xmx4865M -Djava.net.preferIPv4Stack=true 
 74 org.springframework.boot.loader.JarLauncher
 191 jdk.jcmd/sun.tools.jcmd.JCmd

  2. Execute the command below to start a 300-second recording of the JVM. This will profile the JVM and create a JFR file named jfr_example.jfr in /home/Logfiles. Additional JFR parameters

    jcmd <pid> JFR.start name=MyRecording settings=profile duration=300s filename="/home/LogFiles/jfr_example.jfr"

     # Example Output

 root@adb27c7aa8af:/# jcmd 74 JFR.start name=MyRecording settings=profile duration=300s filename="/home/LogFiles/jfr_example.jfr" 

 Picked up JAVA_TOOL_OPTIONS: -Xmx4865M -Djava.net.preferIPv4Stack=true 
 74:
 Started recording 1. The result will be written to:

 /home/LogFiles/jfr_example.jfr

  3. Flight Recordings will be created at /home/LogFiles/jfr_example.jfr which can be downloaded using FTP, KUDU API or Kudu File Manager.JFR can be reviewed using Zulu Mission Control.

    https://<webapp-name>.scm.azurewebsites.net/newui/fileManager#

Continuous Recordings

  1. Update your application’s App Setting to pass the flight recording arguments to the JVM. This will start a continuous recording that will start a flight recording at JVM startup that will exit after one day or if the size reaches 1024MB. Additional JFR parameters

    JAVA_OPTS=-XX:StartFlightRecording=disk=true,name=continuous_recording,dumponexit=true,maxsize=1024m,maxage=1d

  2. The continuous recording can also be manually stopped with the following command. Note: The name parameter specified must match was is passed in the JAVA_OPTS to stop the JFR. jcmd <pid> JFR.dump name=continuous_recording filename="/home/LogFiles/manuallystopped_recording1.jfr"

     # Example Output

 root@97c687c874b5:/#     jcmd 74 JFR.dump name=continuous_recording filename="/home/LogFiles/recording1.jfr"
 Picked up JAVA_TOOL_OPTIONS: -Xmx4865M -Djava.net.preferIPv4Stack=true 
 74:
 Dumped recording "continuous_recording", 1.9 MB written to:

 /home/LogFiles/recording1.jfr

  3. Flight Recordings will be created at /home/LogFiles/recording1.jfr, which can be downloaded using FTP, KUDU API, or Kudu File Manager.JFR can be reviewed using Zulu Mission Control.

    https://<webapp-name>.scm.azurewebsites.net/newui/fileManager#

You may also enjoy