Thread Dump Analyzer

All

Intro

    Sometimes, on Java we can’t trace or profile since instrumentation would take too long, so then thread dumps seem an appropriate tool.  This article is partially based on Pierre’s course[2] and Confluence Support [4].

   Although not a tracing technique, using this tool is possible to do a sampling recording, getting a set of snapshots with the same delay.

Thread Dump Basic

    Well, the dump itself is an execution snapshot of the threads created in Java. The skill to differentiate the threads you care is something to be developed and can be partially solved using the tools – that I talk below.  It is indeed a collection of stack trees. The threads have basically 6 states [4]:

  • NEW
  • RUNNABLE
  • BLOCKED
  • WAITING
  • TIMED_WAITING
  • TERMINATED

Thread Dump Generation

       It’s possible to use JPS to find the applications running Java. 

        1 – kill -3 [pid] –> signal to the process to tomcat

        2 – Jstack command                      

        3 – Java Visual VM [5]

                     Very visual method but we can see the live threads on the console.

        4 – Console – dump straight from the console 

Thread Dump Analysis

        It is technically a screenshot of the threads. It will bring info and several tools can be used to analyze it, though it’s possible to analyze straight from the file, which highlights some details the on the file.

       Don’t analyze all the threads but rather try to find a pattern from a few patterns, with locked threads and all this. This type of analysis can show Heat Depletion [3], in which Pierre-Hugues describes a Thread Dump analysis only using txt analysis.

TDA Tools

   But the manual analysis will become difficult, either because is long or is complex threads, blocked threads, and deadlocked threads are easier to find using a TDA tool, like TDA or Samurai.

Thread Dump Analyzer

      The so-called TDA is an analyzer, is a visualization tool that is used after having the log data using jstack utility, which is part of the JDK not the JRE. 

     Then you can run TDA using  java -jar tda.jar

     We can see then the threads using the visualization tool.

Other tools

     The article [1] compares in some aspects Samurai and TDA tool, according to the author, John Mazzitelli, Samurai loses points by the threads names but appreciated the grouping mechanism used that combines threads with a similar state.

REFs

[1] https://planet.jboss.org/post/simple_tools_to_analyze_thread_dumps

[2] https://www.youtube.com/watch?v=1qzHSEjU8Hc&list=PLeLNWvESQ0GaJv8VCelD0bXiTIcVCRuSC

[3]https://www.youtube.com/watch?v=3dKufRRT_3E

[4] https://confluence.atlassian.com/confkb/how-to-analyse-thread-dumps-788039144.html

[5] https://visualvm.github.io/



Garbagecat

All

Intro

There are many GC log tools [3], the one we’ll talk about today is Garbagecat, which is opensource btw.  I plan to do a post about GCViwer[6].

Garbagecat

This tool is for users to parse and analyze GC log files of jvm. More technically is a command line tool, for either JVM tuning and troubleshooting for JDK’s: OpenJDK and Sun/Oracle JDK. It’s similar to a *log* analyzer like BabelTrace, but it’s actually on the terminal.  There is a wiki here [5].

To get all the data for the tool to work is necessary to use the following flags:

-XX:+PrintGC -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps

This tool will parse the gc log and it will produce a file with several sections, which are explained below:

Sections

Bottlenecks Events this section will say the throughput. Throughput = (Time spent not doing GC) / (Total Time). Throughput of 100 means no time spent doing GC (good)[4], therefore the closer to 100 the better since it means less time doing GC.

JVM they present the summary of the system, with swap info, JVM, and the command line – the ones you selected and the default ones.

Summary it will give more information on the GC events, with the event types. The GC throughput will be shown too. The timestamps will be given in milliseconds (divide by 3600000 to have the hours).

Analysis in this part categorized in warns | errors | info will be given. The analysis is done on the flags that are present and the ones are actually not present on the command line and how this could be done.

-XX:+PrintGCTimeStamps

-The -XX:+PrintHeapAtGC option is enabled. The additional data output with this option is not being used for analysis, so it is extra logging overhead. If there is not a good use case for enabling this option, remove it.
                             [Unidentified Events] the events that occur and could not be identified by the tool. This section only appears if unidentified events occur.

Command Options

garbagecat -r -p -t 50 -o [report.txt] [log file]

        garbagecat: is actually the java -jar [jar]

         -r : reorder: in case the timestamps are not in order

         -p : parse: 

         -t : threashold, i.e. <show me any log line that the thoughput was less then x%>

         -o : output file: write a file name 

GC tuning

I will soon create an entire post about the GC tunning as I go deep on the details, but I also see this reference too [2], which is from Plumbr, and I’ll keep referencing this source. But basically, when tuning the JVM one can focus on one of the three aspects: latency, throughput, and footprint (there are other aspects explained on The Garbage Collection Handbook, like completeness, but won’t talk about it this time).

After seeing the aspect one wants to improve, the next step is to search a GC algorithm that matches it and then improve from that aspect, for example the CMS that can be fined tuned with Ratio, and NewSpace (and MaxNewSpace) or the G1GC that can be tuned by the size of the regions (to avoid humongous regions). ZGC/Shenandoah can be improved in several aspects (one of the is the pacing and the heuristics).

The analysis part of this tool is powerful but give many suggestions on the GC flags.

JBOSS in Eclipse

Well, I use JBOSS diretly on Eclipse, where I develop my servlets, and I change the settings my VM options using the VM arguments  in Edit Configuration opening the launch configuration, in Overview. Follow this link.

OR it’s possible to change the standalone.conf diretly But it’s up to you what to change actually.

Source analysis

Analyzing the code, we can see that it’s organized similarly to another software I worked with – TraceCompass, in the sense of several classes implementing events that extend one main one – with name, id, and timestamp. And because it’s java, which both are. Though Gargabagecat is much smaller than TraceCompass.

I used ObjectAid, tutorial here, tool in Eclipse to create the diagram, and saw that Analysis is one of the biggest classes – not because of the methods, but rather the properties that are many.

REFs

[1] https://github.com/mgm3746/garbagecat

[2] https://plumbr.io/blog/garbage-collection/garbage-collection-increasing-the-throughput

[3] https://stackoverflow.com/questions/541832/know-of-any-java-garbage-collection-log-analysis-tools

[4] https://www.techpaste.com/2011/12/memory-monitoring-bottlenecks-optimization-gc-logs-garbagecat-tool/

[5]https://code.google.com/archive/a/eclipselabs.org/p/garbagecat/wikis/Documentation.wiki

[6] https://github.com/chewiebug/GCViewer/wiki

[7]Link to change the Jboss VM arguments in Eclipse

VisualVM – JvisualVM

All

Intro

While developing Java and tunning your application is quite interesting to see the features of the JVisualVM tool. This post is probably just the first about this topic.

VisualVM View

screenshot from 2019-01-07 17-55-37

> Yes, PermGen not MetaSpace hehe!

 

JVisualVM Description

It’s a visualization tool which is already in the JDK toolkit, and jdk/bin/jvisualvm. I myself created an alias on bashrc so I can use it straight away. This tool will provide you with info about the execution of Java stuff. VisualVM will detect the execution of java applications as JBoss and the java application you execute with java <option>

Auto Detect

I think the first feature of this tool I already described, which is to detect the java application execution right away and therefore

Thread counting

The thread counting is on the right down corner.  It’s also possible to do thread dumps on this tool by using the

PermGen/Heap

We can see the heap/PermGen size on the right corner as well. On the monitor tab we can do the heap dump and the GC.

Class

The number of classes is on the left down corner.

Investigating applications

We can see the evolution of the applications we study by watching how it evolves through time. The monitor is live so we can even set up a small experiment to measure the Heap Size using -Xmx option on JVM. If we put -Xmax too small, we will get straight away a java.lang.OutOfMemoryError: Direct buffer memory error, which is pretty much expected.

 

References

[1] JVisual VM Source

[2] Tutorial for JVM

Any reference of Pierre-Hugues Charbonneau is good, although the last one he did was in 2016.

 

Station F

All

Intro

I plan to launch 10 unicorns in the next 3 years. Xavier Niel

I like to have goals and work diligently towards them, Xavier Niel is certainly one of them and has a similar story of Elon Musk, make money to re-invest in new and more important things. Xavier created the Station F, which he invested 250 mi Euros, and it’s basically the biggest startup campus – in the world.

Station F

If you don’t know about Station F, is a startup incubator, actually the biggest in the world and getters people of all shorts for the same reason – be an entrepreneur.

I miss so much not have seen the campus on my last trip to Paris, it’s huge and so many people, you certainly find someone with similar – or totally different – for a nice chat.

Job Board

Take a look at the boarding of job opts there, it’s huge and it’s great, with 1000+ jobs from Front-end to Data analysis, it’s great[2]. There are volunteers and internships too, ofc.

REFs

[1] https://noticias.uol.com.br/tecnologia/noticias/bloomberg/2018/12/27/quem-e-o-genio-bilionario-que-investe-em-novatos-ate-ex-ladroes-de-carro.htm

[2] https://stationf.co/

 

Junit – TDD Java

All

Intro

I usually do my small prototypes with python and therefore I used pytest and other libraries for as unit test framework. However, I started giving a try on Junit in java. It’s quite simple actually. A good starting tutorial is [1]. I’m a very supporter of TDD and always trying new frameworks and we can see

JUNIT

1. Write the java class –> compile the class

2. Write the java test class –> compile

3. Run test

$JAVA_HOME/bin/java -cp .:junit-4.10.jar:hamcrest-core-1.3.jar org.junit.runner.JUnitCore CalculatorTest

Sucess

JUnit version 4.10
.
Time: 0.002

OK (1 test)

       Failure

JUnit version 4.10
.E
Time: 0.003
There was 1 failure:
1) evaluatesExpression(CalculatorTest)
java.lang.AssertionError: expected:<6> but was:<-6>

FAILURES!!!
Tests run: 1, Failures: 1

 

Obs: I wonder why my tests are taking in avg 1/3 of the time in the original tutorial. 3 times more is quite a different, but it might be the JVM optimization.

PyTest Comparison

Python is a scripting language and therefore does not need to be compiled. So a 1-1 comparison would be decontextualized. However, if we only compare the pytest I think python would win basically the stack is shorter, therefore, simpler to debug actually. Just less noise, but JUNIT is pretty simple also.

The time evaluation is very useful actually.

REFs

[1] https://github.com/junit-team/junit4/wiki/Getting-started

 

Tools for API dev

All

Intro

Postman [already post about it]

I already did a post about PostMAN, quite useful tool. There are two versions actually, on for the browser and another for desktop running.

cURL [1]

Straightway to test requests since gives results right on the spot in the console, therefore you can see it or send it to a file.

combined with jSON placeholder

curl https://jsonplaceholder.typicode.com/posts/3

curl -i: gives several things

curl -I: header

to send to file: curl -o test.txt https://jsonplaceholder.typicode.com/posts/3

to send data: curl –data “title=hello&body=Hello World” https://jsonplaceholder.typicode.com/posts

 

JSONPlaceholder

Basically, a fake API to do tests.

Getting 100 posts:

https://jsonplaceholder.typicode.com/posts

Getting a post with id n

https://jsonplaceholder.typicode.com/posts

– You can actually post data, though it won’t be saved, it will be accepted.

 

REFs

https://jsonplaceholder.typicode.com/

https://curl.haxx.se/

 

 

 

Mars outline

All

Below are some outlines of the evolution in Mars starting in 2024

1. People will go to explore
2. People will go to make science
3. If there is people you need places for the them to stay and food
4.The more people, the more business
4.1 Mining – Iron and stuff
5. People will use it to make exploratory research
6. The amount of people grow more ideas of jobs
7. The data flow (Earth – Mars) will increase
7.1 Satelites, cables, stuff will grow
8. Data centers/Data storages will be created in Mars, the bigger than Earth
8.1 People will realize that mining is not enough, they will start to make products on mars and ship to earth.
8.2 Mars gravity is smaller, therefore is easier to make a rocket there and send to other planets
9. High demanding jobs for technicians will be created
10. Mars becomes a pole for deep space technology
11. Number of people on mars reaches 50mi, Mars is the richest nation and request a formal separation from earth
12. As a separate entity more services and people will come
13. The nations on Earth create similar nations on Mars
14. Creation of MUN – Mars United Nations