Month: July 2019

Metaspace OOME

All

Intro

how can I investigate an OOME?

Metaspace

XX:MaxMetaspaceSize=[size] we can set the Metaspace upper bounds.

However, in case the MetaspaceSize extrapolates an OutOfMemory error.

GC only removes objects that are no longer in use. The GC process is triggered when the max size is reached. Metaspace by default auto increases and keeps meta-information about loaded classes. Instances of the class Class are in the regular heap.

Investigating Metaspace OOME

How to investigate:
~Do a heap dump and analyze it with Eclipse MAT. Look at the classes you have loaded. Check if there's something unexpected, especially duplicate classes. It also has a classloader explorer.

~https://github.com/mjiderhamn/classloader-leak-prevention

Issues on Metaspace ~ OOME
======
The main cause for the java.lang.OutOfMemoryError: Metaspace are one of the following:

~ too many classes:
   For dynamic languages, this might happen.

~too big classes being loaded to the Metaspace:
    A classloader leak happens when an application is redeployed, but there are lingering instances of the previous deployment’s classes left.

You can use MAT for investing those kinds of issue, and acquiring a heap dump to analyze. But on this case, the heap should be dumped after at least one ClassLoader instance has leaked. You can then analyze it with Eclipse Memory Analyzer ~ MAT.
There is a through explanation on details [1].

References
https://java.jiderhamn.se/2011/12/11/classloader-leaks-i-how-to-find-classloader-leaks-with-eclipse-memory-analyser-mat/

REFs

Classloader leaks I – How to find classloader leaks with Eclipse Memory Analyser (MAT)

Audit log ~ EAP

All

Quick note for JBoss ~ enabling Audit log

The audit log can be used ~ as the name says ~ to Audit information as who and when are logging. The original reference is here.

Audit log xml:

The following XML can be used as a model:

        <audit-log>
            <formatters>
                <json-formatter name="json-formatter"/>
            </formatters>
            <handlers>
                <file-handler name="file" formatter="json-formatter" relative-to="jboss.server.data.dir" path="audit-log.log"/>
            </handlers>
            <logger log-boot="true" log-read-only="false" enabled="true"> 
                <handlers>
                    <handler name="file"/>
                </handlers>
            </logger>
        </audit-log>

Output

The output will be on the file:

EAP_HOME/standalone/data/audit-log.log

Logger configuration ~ three configurations

JBoss Audit log has three configuration modes – that basically will filter the amount/type of data that will be recorded, as below:

Attribute Description
enabled true to enable logging of the management operations
log-boot true to log the management operations when booting the server, false otherwise
log-read-only If true all operations will be audit logged, if false only operations that change the model will be logged

 

Btw: log-read-only when is used, increase the overhead considerably, since there are many operations that happen every 15s/20s in EAP.

 

CLI command for the configuration attributes above:

cli-command#  /core-service=management/access=audit/logger=audit-log:write-attribute(name=enabled,value=true)

cli-command#  /core-service=management/access=audit/logger=audit-log:write-attribute(name=log-boot,value=true)

cli-command#  /core-service=management/access=audit/logger=audit-log:write-attribute(name=log-read-only,value=true)

Disclaimer

I work at Red Hat with EAP.

TCP Dump Linux

All

Intro

Understanding bugs on Java/Python is quite interesting, but sometimes we have issues with the network. On those cases, we have the tcpdumps to help:

TCP Dump

tcpdump basically is a common packet analyzer that runs under the command line, as in:

Command

        #sudo tcpdump -D

Btw, reading is much more difficult than actually capturing the data.

For only IPV6 traffic you can use:

#tcpdump ip6

For HTTPS get/post dump, the command is here:

 sudo tcpdump -s 0 -v -n -l | egrep -i "POST /|GET /|Host:"

Some examples can be found here

REFs

inline

OmitStackTraceInFastThrow

All

Intro

Oracle Release notes

The compiler in the server VM now provides correct stack backtraces for all “cold” built-in exceptions. For performance purposes, when such an exception is thrown a few times, the method may be recompiled.

After recompilation, the compiler may choose a faster tactic using preallocated exceptions that do not provide a stack trace. To disable completely the use of preallocated exceptions, use this new flag: 

-XX:-OmitStackTraceInFastThrow.

OmitStackTrace

-XX:-OmitStackTraceInFastThrow.

Well, there is a JVM optimization (OmitStackTraceInFastThrow, which is on by default) which optimizes “built-in” exceptions that are thrown frequently enough to be cached and have to stack trace. So somebody at one point thought that was a worthwhile optimization and allocated engineering resources to it.

REFs

inline

Interesting English words

All

anomaly: unexpected

equivocal: not easily understood or explained

lucid: unambiguous

assuage: less intense

prodigal:

laudable: admirable

pedant: excessive display of learning, show off

vacillate: fluctuate

capricious: temperamental

engender: generate, provoke

loquacious: talkative

apathy: lack of enthusiasm

corroborate: confirm, verify, authenticate

ephemeral: transitory, momentary

laconic: use a few words

 

 

 

 

 

 

 

 

 

 

 

Oracle JDK usagetracker

All

Intro

On Oracle JDK you have this usage tracker that can record information about the JDK usage. It’s quite interesting and useful for debugging and profiling Java applications.

Usage Tracker

It’s a usage tracker – i.e a file with several details about usage. The values are in ” ” and separated by commas (default, you can change this configuring additional options)

 

Usage Tracker Output

An example of the output is below:

"javaws application",
"Mon Feb 01 14:52:58 PST 2016",
"MY-COMPUTER/192.0.2.0",
"https://docs.oracle.com/javase/tutorialJWS/samples/deployment/
  dynamictree_webstartJWSProject/:
  sourceURL=https://docs.oracle.com/javase/tutorialJWS/samples/
    deployment/dynamictree_webstartJWSProject/dynamictree_webstart.jnlp
  app_model=eJytU9FqE0EUvWlSk9pQsSKCIgSpKCIzD4I++BQThZS1LU1FfSrTzbTZMDszztxt
    E4WCP1HwwT/wP3z3G3z1DwS9s7tpF5Q8dR9md/aeueeec+98+wXL3sGd2KTMZ5qN
    pFVmxpw8ZF1rX5uRVFA8tSW4GkErpl8HwkuEtWgijgXXEvmb3eh5BKsj6WOXWEyM
    RlgvwkroIz5El+gjgqzLKUrtCbArvclcLD3CzQKZYaJ41zkxixKPBG5PhDuHfYBT
    qBO/MrEIBGFfi2AlFYnuKeFzwBId8jLOXIKzUDvCo4iUcVLGC2WclPG5so1hBUuE
    y5igkmWiBs4sJbi3MMEeYejglWOpR8blJ6eW/GwHRYysYWTN2f1nT2///unq0BxA
    ayz8uEeHB9CwxiFJEBmOTShjznyYXFQxNh7L75Z1Bk1sVLmvUykl5Z/yQbg2MrFn
    xolYSUalI7zioRgvOWZINEJtvh1yL1KrpC9FpVIjH820SJMYnZT7J/LAo3BI0B1n
    JjJGHphWEJbHiNbbXOWNXGXoGzvv2/Tzj7tfvouvdaiRQp98lFMbxuekEdZp3qZ2
    GKf8R42yPFgwexubFyNwMYdL/85B81g6X25DC8K7OXelLP395RjRL8J7FO7L1DCa
    0pwgLKtkTKGxyc+efKqQ718O+f/CbKKVrZaA8Hge65nUGk2Juy4es0rp5LBKCgtP
    HTxc1IPqLSmbAOvUSwdr+QCEK85e6iytBi1Cc9jd6r/YfodwvSTuBOZOcI04O4s4
    w8WqpgvKbiGsdnd2okGvuzfY3kJon6eVIv0L97SYOQ==",
"C:\Java\jre1.8.0_71",
"1.8.0_71",
"25.71-b15",
"Oracle Corporation",
"Oracle Corporation",
"Windows 7",
"x86",
"6.1",

The example above is output and shows several details about the usage, including the architecture, x86, the JDK, the JRE version/location, the OS, in one file.

To enable Usage Tracker

  1. First option: create usagetracker.properties – as follows:
# UsageTracker template properties file.
# Copy to <JRE directory>/conf/management/usagetracker.properties
# (or <JRE directory>/lib/management/usagetracker.properties for
# JRE releases prior to 9) and edit, uncommenting required settings, to enable.
 
# Settings for logging to a file:
# Use forward slashes (/) because backslash is an escape character in a
# properties file.
# com.oracle.usagetracker.logToFile = ${user.home}/.java_usagetracker

# Settings for logging to a UDP socket:
# com.oracle.usagetracker.logToUDP = hostname.domain:32139
  
# (Optional) Specify a file size limit in bytes:
# com.oracle.usagetracker.logFileMaxSize = 10000000
 
# If the record should include additional Java properties,
# this can be a comma-separated list:
# com.oracle.usagetracker.additionalProperties = 
 
# Additional options:
# com.oracle.usagetracker.verbose = true
com.oracle.usagetracker.separator = ,
com.oracle.usagetracker.quote = "
com.oracle.usagetracker.innerquote = '

 

  1. Second option: apply directly on <file in the central file system location>
java -Dcom.oracle.usagetracker.config.file=/path/usagetracker.properties MyApplication

 

Additional options

The options below, are for the file output itself.

# Additional options:
# com.oracle.usagetracker.verbose = true
com.oracle.usagetracker.separator = ,
com.oracle.usagetracker.quote = "
com.oracle.usagetracker.innerquote = '

OpenJDK

Up to my knowledge, at this moment, the OpenJDK version is not ready yet. Ofc, I might be wrong. The is the Jira bug.