Introducing glassbox: J2EE troubleshooter

Ever wondered what’s happening inside the application server? Ever had a fight with your developers about performance of a J2EE (tomcat/jboss/oc4j/iAS/jrun/…)? Or had a fight with developers who blame the database while you think (or know) it is not causing the problems?

There is a solution for that: Glassbox.

Glassbox is an application which can be deployed in a J2EE (java) application server, which uses AOP (aspect oriented programming) to monitor behavior of the J2EE server, and outputs operations ordered by average response time (total responsetime / number of executions). Besides that (which on itself is a godsend for application server administrators), it highlights operations which exceed the threshold (default 1 second) and it highlights operations which throw an error.

The operation can be selected (conveniently using a browser), so specifications of this operation can be investigated. like:
– stack trace of the code executed
– input URL
– diagnosis
– number of times it ran within the threshold, exceeded the threshold and failed

This is how the glassbox application looks like:
Glassbox interface

Even more interesting, it can diagnose a slow database operation (as seen in the screen above), and display the SQL which is responsible:
Glassbox slow database diagnosis detail screen

The installation of glassbox is very straightforward for jboss, the procedure for installing it in the OC4J in iAS version 10.1.3 requires some tweaking of xml files. Contact me if you are interested in the installation procedure.

You can find glassbox, including the free download (and forums for support) here

  1. Wilfred said:

    I would be very interested in the installation procedure for Oracle OC4J. We’re currently experiencing some performance issues with our application and struggle to find the cause. This sounds like a great solution.

  2. Gert Jan said:

    We are (also) very interested in the installation notes for an OC4J config. We’re using Oracle Application server (Forms and Reports services)
    We’ve been using JXInsight (from Jinspired) for a few weeks now and that’s a great tool, but it’s licensed (maybe for the future, since gathering funds for it will take a lot of time -when working for the government-). Many thanks in advance! kind regards, Gert Jan Willems

  3. Hi Frits, I think you will find that commercial tools such as JXInsight offer much performance trace and transaction analysis across processes, tiers and clusters. JXInsight is reasonably priced considering the time it takes to solve real world problems that cannot be solved by simple SQL string pairs used as JMX beans names. Patterns of behavior are much more important that counters and context is everything which is not what is offered by any of the tools in the open source domain today.

    I hope that after reading the following articles you might agree that production problems are much more difficult and cannot be solved by a small list of checks with limited scope.

    The are more than 1 million tx events summarized in the screen shots contained in the last article.

    We are just finishing up JXInsight 5.0 scheduled for release first week in Feb. In March at TSSJS we will be announcing a complete AOP addon to the product that integrates many of the extension points in the product: metrics, diagnostics, traces, and probes (coming next week).

    You should also check out our corporate blog which has some nice entries.

    kind regards,

    William Louth
    JXInsight Product Architect
    CTO, JInspired

    “Java EE tuning, testing, tracing and monitoring with JXInsight”

  4. Administrator said:

    William, I understand you want to plug your product. But for the sake of me reacting on it, I let it pass through. (the other 3 reactions I did mark as spam, though (..))

    Of course a commercial product which has an entire company behind it can do more, and more easy. I guess you missed the point here. Glassbox gives me really all I want to know: where time is spend. That is all there is. Proving where the time is going.

    Guess I know a few holy houses over, but: I can understand GUI development centric developers want a fancy GUI to do the tuning in.
    But tuning isn’t about fancy screens and important looking tree structures. Tuning is about proving where time is spend. Presentation of the figures is optional, it doesn’t add anything to the facts.

    And about pricing: glassbox is free. The jinspired website FAILS to give me an actual price, which gives me an ‘itchy’ feeling…

    Have you ever considered GUI’s can overkill in a tuning situation?

    Also, have you ever considered that due to the high abstraction level of the java language, most developers have no clue what the technical impact is of the code? (oops, there goes another holy house)
    And, because of that, if these people must tune, their actions are useless and pointless? (or: have a lottery effect)

    Or, even if they are succesfull at tuning, they tuned the application IN THEIR DEVELOPMENT ENVIRONMENT, which, in almost all cases, is not representative for production?

  5. First off when submitting the original blog entry the image code is not rendered in the browser. It is reported as missing. Looking at the pattern one could have guessed there was something amiss.

    I provided 3 links above and I would dearly love to see how you could have detected the problems presented in these articles by looking at the million or so lines of textual ouput.

    With regard to the GUI side remarks, which to be honest betray you, you might want to look around at other related fields in the computer industry that focus on real added value: Information Visualizations.

    Have you ever considered that developers using performance tools might not have actually tested the performance overhead of such tools and whether they actually report the correct values? Well we have a report coming out soon that refutes all those claims of “1% overhead” – performance tools with inherent performance problems in their architecture.

    >> “Presentation of the figures is optional, it doesn’t add anything to the facts.”
    Please take a moment out and invest a little in reading about visualization. Here is a FREE reference for you.



  6. Administrator said:

    William, first let me tell you that JInspire probably the most advanced performance profiling tool today. (at least that I am aware of). I know there are some other products (including free ones), but the JInspire suite is probably the most capable. There is no discussion about that.

    But I am not a java developer. And I don’t want to profile each line of code.

    I am a person who gets hired to combat performance problems. I do not have access to source code in most of times, or even have any access to the developers. I need to discover the problem. This also implies I can not make any significant changes to the environment.

    Using the glassbox tool, I just need to make minimal changes to the startup script in order to get it running. Then, glassbox reports per object instead of per line of code, and it shows the object with the most time spend on top, together with an usage count.

    This means I am able to pinpoint slow database connections, or much data which is fetched and processed in java, or blocking threads, or … I do not fix the problems, I just can pinpoint it, and instruct the vendor or the developers to fix that.

    It is a known fact that measuring something will influence the measurement. Of course will measuring a java instance influence performance, and decrease it. But one shouldn’t profile an application any longer than needed. So I do not understand your point about overhead.

    Visualisation of information is needed if you want to create an overview of large collections of results. Like the database example from your site. Line, by line, by line, by line…

    If you look at an higher level (object level for example), there is no need in visualisation, because the number of results is tremendously smaller.

    So, to conclude it, JInspire can have real added value, as well as glassbox does. JInspire is focussed strongly on development, whereas glassbox is focussed more towards troubleshooting of production problems.

    Oh, and JInspire isn’t free. For ad-hoc performance analysis, I do not find it “reasonably priced”.

    Thanks for the discussion!

  7. “JInspired is focussed strongly on development, whereas glassbox is focussed more towards troubleshooting of production problems.”

    This is not the case. Most of our sales are for pre-production and production systems where the performance issues are related to workload peaks (metric monitoring), high concurrency (service times), memory capacity planning (object alloc counting and gc activity), intermittent errors (diagnostics), database dead locks (tx analysis)…. none of which is handled by Glassbox today. In fact in our benchmarking of various open source tools including P6Spy, JBoss Profiler, and Jamon Glassbox comes out by far the slowest (biggest overhead) and with the weakest level of accuracy in the collected data. Considering how poorly coded and unreliable some of the other tools are this is not saying much for the tool itself.

    The articles below relate to production systems how else would you get a cluster of servers hitting a database with each hitting rates of 500 tx per second. One of the case studies relates to one of the biggest J2EE applications in Europe handling the largest postal system in Europe. After testing “performance management solutions” from all the big vendors JXInsight was found to be the only one that was able to keep under the 5% overhead requirement. 5,000 compared to 150,000 looks very reasonable.

    Please note my background is in CORBA (Visigenic/VisiBroker) & J2EE (Borland AppServer) – all server side and high performance production solutions. JXInsight is a tool that comes with analysis modes: metric, profile, timeline, and insight. Each one focused on the different roles defined in a performance engineering team (developer:[profile,timeline], tester:diagnostics,profile, operations:[metric,insight], dba:[profile, timeline]…..

    I understand that the presentations on the website seem developer focused and the amount of data comprehensive and detailed but please do not be fooled into thinking that this comes at the cost that other tools try to overhead. Our product is heavily optimized with over 500 system properties that allow refinement of the type of data collected and analyzed. When we add a feature we do a complete benefit analysis and then when we decide to include it we heavily optimize the implementation.

    By the way we are just finishing up an extension to JXInsight that provides the ability to introduce light weight probes into an application for resource metering which will make our overhead even lower when running in this mode.

    regards, William

    PS: JInspired is based in Holland and England. I live in Voorhout. When I first came to Holland 10 years ago after finishing up my studies in Ireland I worked for both Logica and CMG. I then left Holland to work with Borland and some US banks before returning a few years back.

  8. Administrator said:

    William, I do not question JInspired, nor your knowledge.
    And I do understand you measured yourself that you have the lowest overhead and is the most accurate. I also believe that it could very well be true.

    I do question the way you plug your product. It is in a very unpleasant and predetermined way. I can not find any CMG in the way you are communicating.

    Ever considered there are more ways of doing one thing?

    Most customers I work for have looked at JInspired or even used it, and stay away because of pricing.

    I need to be able to pinpoint problems very quick for what I do. In order to steer problem resolution. I do not need extensive suites.

    The way you present JInspire, the could be environments where JInspired is feasible, but I do encounter a very little amount where a process of performance profiling is done throughout the entire development cycle.

  9. Regarding Pricing. 750-1000 USD for a workstation edition seems extremely reasonable for local application server profiling and tx testing. The cost of ownership over a 3 year period is 1,500 USD. Note the currency – USD.

    A CMG consultant charges between 1,000 – 2,000 EURO a day with a simple performance investigation taking between 3-5 days (on-site visit + off-site reporting) this is excluding any other follow-ups as it is unlikely an application could be tested, analysed, and tuned in any serious environment without going through proper change management both with the version control system and the infrastructure. You are talking about a minimum of 5,000 EURO for each performance assignment which happens to be the price of a performance management solution that can track performance historical and in great detail as well as helping with problem and incident management (I assume ITIL training is mandatory for such consultants with CMG in the context of operations management).

    Look JXInsight was designed to solve problems across all phases (dev, test, intg, prod) and under different contexts (software assurance, app performance mgmt, incident management, problem mgmt, and configuration mgmt). When viewed as a whole the offering is extremely competitively priced.

    “I think offers the best feature/price ratio”

    Frits I am going to leave it there as I think your mind set is fixed at this point in time and my efforts will unlikely to change this. Thanks for the workout – it is always good to reassert why you believe your solutions meets the needs of enterprise customers more than others.

  10. Administrator said:

    William, you are the one with the fixed mindset. I try to explain in 3 comments now there is a world outside of JInspire, but with no result.

    Your comments are a bit lame and out of proportion:

    If a company is not able to deal with a performance problem itself, extern knowledge must be hired. If the problem is to be found in an application server, a tool must be installed to profile the performance. Only THEN a tool like JInsipre, but also Glassbox comes to question.

    If you’ve been in the real word, you know how often a strict ITIL procedure is followed for development.

    I do not often encounter environments where performance profiling is done throughout the whole development cycle.

    It’s good to have a discussion about things, especially because I am not an authority on java development.

    I am a bit disappointed you can not and do not want to view it from my perspective (whilst accusing me of it). Again, I think glassbox is a very simple tool to deliver only one thing, probably not in the most efficient way, and JInspire is the MacGyver swiss-army knife for performance profiling. But I just can guess there is a financial and business reason for that.


  11. Performance management is not about ad hoc assignments. If a performance consultant was really interested in solving the underlying issues within a production environment and not just trying to secure another consultancy gig when the next change is applied to the software and/or infrastructure then they would be pushing for a continuous and pro-active management approach. This is the difference (and I believe the conflict). Our solution is not just about fixing the immediate solution after the service level agreements have been violated and the external performance consultant called in (if available). With our solution we look to detect execution behavior and resource consumption changes as early as possible across all phases building a comprehensive repository for improved knowledge transfer across teams especially when performing capacity management.
    Capacity management is concerned with:
    * Monitoring the performance and throughput or load on a server, server farm, or property
    * Performance analysis of measurement data, including analysis of the impact of new releases on capacity
    * Performance tuning activities to ensure the most efficient use of existing infrastructure
    * Understanding the demands on the Service and future plans for workload growth (or shrinkage)
    * Influences on demand for computing resources
    * Capacity planning – developing a plan for the Service

  12. Administrator said:

    I agree whole hearted. But your describing an ideal situation. Where money is available for capacity management, and enough knowledge is available to understand the results of the measurements taken.

    I also completely agree that if performance is not taken into account, it will result in problems in that area in much of the cases.

    But, still there are projects out there that do not do that (in fact, most of the projects), and get stuck in a fingerpointing game between developers, database administrators, windows or linux administrators and network administrators, once deployed. I do not start these projects, nor have any influence in how these people (both inside and outside of the company I work for) do run these projects.

    And they run into problems. Most of the time in production, whilst a deadline is approaching or even passed, and have no clue what to do next. At that point an inventory of the main problems has to be made. That is the point I am aiming at. No detailed description, just the ability to identify what is going on, and finger point it.

    I am looking forward in seeing JInsight some day. I think it can be a valuable tool for performance management throughout the development cycle. Maybe we even run into each other here in Holland.


  13. Giuseppe said:

    Hi, I am trying to install glassbox on OAS 10.1.3. In this blog you say that you can provide xml configuration files for this AS, I am having a lot of problems.. can you give me some help? Thanks in advance. Giuseppe

  14. I found your blog by chance . but i have to say that it’s great blog very useful information and very interesting subjects just greetings and good luck
    i’m not going i will be always checking for updates.I’m very interested in CMS and all its related subjects.

  15. balaji said:

    Can you explain the procedure what are the configuration things requried to run glassbox with jboss server , how to install glassbox in jboss and how to run glassbox
    and how to run a sample jboss project with glassbox for to test the performance.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: