In-Depth
Top 5 Application Performance Challenges
Tips to identify and prevent the top headaches Java developers face.
By Bhaskar Sunkara, Director of Engineering, AppDynamics
Given the complexity of today's application environments, it comes as no surprise that many application developers are faced with multiple challenges when managing applications in production -- and particularly those written in Java, which is one of the most popular and widely used languages.
To help relieve a few headaches, I've outlined some of the top challenges facing Java teams, as well as tips for identifying and (hopefully) preventing them.
Challenge #1: Memory Leaks
Memory management is a critical part of Java. As a developer, you want to manage memory automatically. One of the great benefits of Java is its ability to take care of the memory model for you. When objects aren't in use, Java helps you out by doing the clean up.
Memory leaks typically happen as a result of improper programming – usually it's a case where the developer didn't relieve all references to an object. If memory management is done poorly, you will run out of space for the heap. Collections add up and then the JVM crashes.
There are two primary methods -- heap dumps and profilers -- to diagnose memory leaks in a development or test environment. A heap dump allows you to see which object is holding a reference to the collections. It gives you a good idea of what objects are causing the problem, but it doesn't tell you who is accessing the collection and who's not. It tells you where the collection is, but not the characteristics of who's using it. Heap dumps are also usually quite large, in gigabytes, and a big heap dump can create system overhead and can be tedious to review.
The second method, a combination of a heap dump and a profiler, gets you a little bit closer, but not much. Memory profilers try to help you analyze your heap dump.
Some memory leaks can be very difficult to reproduce in a development or test environment. To diagnose the root cause of a memory leak in a production environment, you will need to use a different tool -- one that has been designed for low overhead. Pick a solution that both automatically detects leaks as well as identifies which code path or business transaction is causing it.
Challenge #2: Slow SQL
Almost every application uses a relational database. A common problem with applications is badly performing SQL due to a variety of reasons such as tables that are not being properly indexed or too much data being fetched. This affects application performance adversely because most applications use multiple SQL invocations per user request.
Slow database access is a topic that can go in multiple directions, so let's take one particular aspect of slow SQL -- the Object Relational Mapper (ORM). You may be familiar with a popular ORM known as Hibernate.
The ORM has become a method of choice for bringing together the two foundational technologies that we base business applications on today -- object-oriented applications (such as Java or .NET) and relational databases (e.g., Oracle, mySQL, or PostgreSQL). For many developers, this technology can help eliminate the need for them to drill down into the intricacies of how these two technologies interact. However, ORMs can place an additional burden on applications, significantly impacting performance even though everything looks fine on the surface.
Although a developer may emphasize that the benefits of using the ORM component (without having to understand how it's working) greatly increase productivity, it does pay, however, to review an ORM's data access strategy to get a solid understanding of the burden on your application. Make sure you can correlate business transaction activity with the SQL activity to identify the effect of ORM on application performance.
Challenge #3: SOA Remoting Issues
In a service-oriented application, bottlenecks often occur in service communication. This problem becomes difficult to isolate because you need to have a view of the distributed communication flow to be able to understand where the bottlenecks are. Without cross-tier visibility, it is difficult to identify where to begin troubleshooting because many pieces are involved in servicing the request. Once the service causing the bottleneck is identified, the process of isolating the actual root cause begins.
Create a process that gives you an overview of the traffic flow between services and also allows you to look at how much time is usually spent between two services. This way you can always compare the current response times with the baseline to identify slow communication. The process must also drill down into the particular service for root-cause analysis.
Challenge #4: Runtime Exception Errors
Runtime exceptions are triggered within Java applications. When they occur frequently, they do appreciably slow down your applications. Runtime exceptions affect the application by consuming resources during throwing, handling, and logging exceptions.
I recently visited a company that was running one high-volume production application (hundreds of calls per second). At that location, we observed an NPE (Null Pointer Exception) rate of more than 1000 exceptions per minute (17,800/15 min). This particular NPE occurred in the same line of code and propagated some 20 calls up in the stack before a catch clause handled it. The pseudo code for the line read if the string equaled null and the length of the trimmed string was zero, then do x; otherwise, just exit. When we read the line out loud, the problem was immediately obvious – you cannot trim a null string. Although the team agreed that they needed to fix the code, they remained skeptical that simply changing '=' to '!' would significantly improve performance.
Using basic simulations, we estimated that this error rate imposed a 3-4 percent slow down on basic response time at a minimum. You heard me, 3-4 precent slow down at a minimum.
Slower performance due to runtime exceptions becomes contagious to all transactions being served by the application. Don't dismiss or ignore them, and don't convince yourself they are harmless. If you want a simple way to improve application performance, start by fixing these regularly occurring exceptions.
Challenge #5: Threading and Synchronization Issues
Of the many issues affecting the performance of Java applications, synchronization ranks near the top. There is no question that synchronization is necessary to protect shared data in an application. The fundamental need to synchronize lies with Java's support for concurrency. This happens by allowing the execution of code by separate threads within the same process.
Using shared resources efficiently, such as connection pools and data caches, is very important for good application performance. Being too aggressive with shared resources causes data inconsistencies, and being too conservative leads to too much contention between threads (because resource locking is involved). This affects the performance of the application largely because most threads servicing users are affected and slowed down -- they end up waiting for resources instead of doing real processing work.
If you want to improve synchronization issues, application performance management tools can help; the right tool can enable you to monitor application execution under high loads (aka "in production") and quickly pinpoint the execution times. In doing so, you will increase your ability to identify thread synchronization issues become greatly increase -- and the overall MTTR will drop dramatically.
The Bottom Line
These are just a few of the performance challenges that developers and application owners run into with Java. With the right tools, these challenges can be prevented and IT operations and developers can stop the bleeding, find the problems faster, and sleep at night knowing their applications are performing at healthy levels.
Bhaskar Sunkara is the director of engineering at AppDynamics. You can contact the author at bsunkara@appdynamics.com