Intro: problem statement

For some reason, my ./add-user.sh user was not working on WildFly/JBoss EAP 7.3, so I couldn’t actually open the console page on http://localhost:8080/.

So decided to add the user directly, on $wildfly-19.0.0.Final/standalone/configuration
mgmt-users.properties.

mgmt-users.properties File

So the `mgmt-users.properties` will be encrypted and will look like:

         #admin=2a0923285184943425d1f53ddd58ec7a

This happens because the passwords are hashed using:

mechanism:HEX( MD5( username ‘:’ realm ‘:’ password))

Workaround

Found a blog post that implements this mechanism on How to generate the password on JBoss AS, by JaySenSharma – based directly on org.jboss.crypto.CryptoUtil lib hashing method:

1. Did some small modification on the EncryptPassword.java file, that is actually broken for 1.8.191, removing some stuff.
2. Exported the Env variable – CLASS_PATH
3. compilated with javac (since was running in JDK 8, not 11 that has java directly)
4. All fine, but forgot to add the <:.:> on the classpath export. So of course, java was not finding my class. As in:

$ export CLASSPATH=/jboss-client-4.0.2.jar://picketbox-5.1.0.Final.jar
$ /jdk1.8.0_191/bin/javac EncryptPassword.java
$ /jdk1.8.0_191/bin/java EncryptPassword testUserOne ApplicationRealm testPasswordOne
Error: Could not find or load main class EncryptPassword

5. Fix adding :.: so adding the current directory on the classpath:

$ export CLASSPATH=/jboss-client-4.0.2.jar:/picketbox-5.1.0.Final.jar:.:
$ /jdk1.8.0_191/bin/javac EncryptPassword.java
$ /jdk1.8.0_191/bin/java EncryptPassword testUserOne ApplicationRealm testPasswordOne
clearTextPassword: testUserOne:ApplicationRealm:testPasswordOne
hashedPassword: cf8f98f5b90ccc568e1ffc7767ac9d8b
If you will create user using add-user.sh script then you will see the same Hash Value of Password.nn

6. Edit directly the mgmt-users.properties with the user/hashedPassword values, and could access the management page.

#admin=2a0923285184943425d1f53ddd58ec7a
testUserOne=cf8f98f5b90ccc568e1ffc7767ac9d8b

Using this workaround, was able to access the console page directly.

Just-In-Time Java compilers: Client and Server

As described to the Hotspot Glossary

Highly optimizing bytecode compiler, also known as ‘opto’. Uses a “sea of nodes” SSA “ideal” IR, which lowers to a machine-specific IR of the same kind. Has a graph-coloring register allocator; colors all machine state, including local, global, and argument registers and stack. Optimizations include global value numbering, conditional constant type propagation, constant folding, global code motion, algebraic identities, method inlining (aggressive, optimistic, and/or multi-morphic), intrinsic replacement, loop transformations (unswitching, unrolling), array range check elimination.

With two compilers, plus interpreter.

To disable the C2 Compiler, you can use:

   -XX:TieredStopAtLevel=1.

(BTW, some people use TieredStopAtLevel=1 to limit the total time of optimization during compilation process, therefore reducing the overhead of the compiler and gain some speed) See [3]

Basically, this flag, disables intermediate compilation tiers (1, 2, 3), so that a method is either interpreted or compiled at the maximum optimization level (C2).

# $JAVA_HOME/bin/java -XX:+PrintFlagsFinal -version | grep CompileThreshold
intx CompileThreshold = 10000 {pd product}
uintx IncreaseFirstTierCompileThresholdAt = 50 {product}
intx Tier2CompileThreshold = 0 {product}
intx Tier3CompileThreshold = 2000 {product}
intx Tier4CompileThreshold = 15000 {product}
java version “1.8.0_191”
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

Issues

There are several issues that can happen, for example:

1. A method get stuck on the C2 compilation

On this case, you will need to see which method is, using htop/top -H, then jstack -m or (pstack) and then see the line with C2Compiler::compile_method, so then you can use exclude the method of C2 using -XX:CompileCommand flag.

      2.CodeCache Full

On this case, the Code Cache is full, so you will need to increase the initial reserved space for code cache, and use codecache flushing, as below:

-XX:ReservedCodeCacheSize=384m
-XX:+UseCodeCacheFlushing

References

[1] https://phauer.com/2017/increase-jvm-development-productivity/

 

Glassfish is pretty easy to change the log level directly on the console.

0. Open console adm: http://localhost:4848/common/index.jsf

1. Configuration

2. Logger Settings

3. Log Levels

 

I thought I had already had written this blog post long ago. I might be mistaken, maybe when I wrote about the fact there is no more Jhat after OpenJDK 9, or talking about jVisualVM.

Well, I will miss it.

Jhat

‘jhat’ is a heap analysis tool that parses a Java heap dump and enables web-browsing a parsed heap dump, and even write Object Query Language interface [1]

Pretty straight forward, the help is below only with the options I actually use:

Usage: jhat [-stack <bool>] [-refs <bool>] [-port <port>] [-baseline <file>] [-debug <int>] [-version] [-h|-help] <file>

-J<flag> Pass <flag> directly to the runtime system. For
example, -J-mx512m to use a maximum heap size of 512MB

-port <port>: Set the port for the HTTP server. Defaults to 7000

-baseline <file>: Specify a baseline object dump. Objects in
both heap dumps with the same ID and same class will
be marked as not being “new”.

-debug <int>: Set debug level.

0: No debug output
1: Debug hprof file parsing
2: Debug hprof file parsing, no server
-version Report version number

-h|-help Print this help and exit
<file> The file to read

1. Get the hprof file – aka heap dump
2. Execute jhat to analyze:

$JDK/bin/jhat -J-Xmx3g -port 7001 example.hprofReading from java_pid30212.hprof…
Dump file created Thu Feb 27 05:32:01 EST 2020
Snapshot read, resolving…
Resolving 15958718 objects…
Chasing references, expect 3191 dots…………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………..
Eliminating duplicate references…………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………..
Snapshot resolved.
Started HTTP server on port 7001
Server is ready.

This tells the jhat will use 3g will affect the effect of the heap size of JVM. In this case, I do not use the parameter file, but rather, pass the file directly at the end.

-f= file.hprof

3. Open the browser with port 7001 on this case, specified with -port 7001.

http://localhost:7001/

4. Do the OQL queries, example ~ to investigate native memory leak:

    SELECT k, k.capacity FROM java.nio.DirectByteBuffer k

Example with MAT for Query:

All Classes (excluding platform)

Package <Arrays>

(…)

(…)

5. See ~ which is quite useful to count objects:

Execute Object Query Language (OQL) query

Select * from org.hornetq.core.client.impl.ServerLocatorImpl o WHERE o.initialConnectors != null

You can use this OQL or the class would be on jhat console directly.

http://localhost:7001/allClassesWithPlatform/

Reference

[1] https://www.infoq.com/news/2015/12/OpenJDK-9-removal-of-HPROF-jhat/

[2] This book covers a little bit of the tool and has several OQL queries for guidance.