Protocol buffers

All

Google’s mechanism for data serialization, Protocol Buffers language neutral. There are three versions of the protocols: proto2, proto3.

Proto

Proto is the abbreviation of the file but also is the name of the protocol.

Message

Defining a (using .proto file)

message SearchRequest { <-- similar to python:: define message `SearchRequest`
  required string query = 1;
  optional int32 page_number = 2;
  optional int32 result_per_page = 3;
}

Advantages

> Fast ~ binary

> Lightweight ~ binary 😀

> Language independent (if you are prototyping in python and implementing in java)

Benchmarks

An example of Protobuf tutorial, given by Ten Loh, where he does a comparison of JSON parsing and Protobufs.

Protostream

On Infinispan/Data Grid, to develop custom Protostream as default serialization mechanism, this replaces the JBoss Framework.

For customization, there are two possibilities on DG:

1. Generate one given that the serializable classes have the proper annotation (@ProtoField and/or @ProtoFactory). To automatically generate it, one relies Data Grid `protostream-processor` (after adding the proper annotations) that on compile time generate the .proto file.

2. OR you manually  implement a proto message buffer

jcmd baseline

All

I would like to know exactly where the JVM is spending the memory, almost bit by bit! how to do so?

I already mentioned jcmd on my previous reply. But now, I will show this feature to actually take an compare baselines:

$ jcmd 26283 VM.native_memory 26283: Native Memory Tracking: Total: reserved=3075884KB, committed=1815600KB - Java Heap (reserved=1335296KB, committed=1335296KB) (mmap: reserved=1335296KB, committed=1335296KB) - Class (reserved=1149983KB, committed=118375KB) (classes #20513) (malloc=3103KB #30610) (mmap: reserved=1146880KB, committed=115272KB) - Native Memory Tracking (reserved=5453KB, committed=5453KB) <--- Native memory (malloc=255KB #3577) (tracking overhead=5198KB)

Generating diff:

$ jcmd 26283 VM.native_memory detail.diff 
26283:
Native Memory Tracking: Total:
reserved=2984385KB +296KB, committed=1725525KB +296KB -
Java Heap (reserved=1335296KB, committed=1335296KB) (mmap: reserved=1335296KB, committed=1335296KB) -
Native Memory Tracking (reserved=5572KB +79KB, committed=5572KB +79KB) <--- this means that +79Kb were added
(malloc=348KB +63KB #4926 +890) (tracking overhead=5224KB +16KB)

This means that 79Kb were added tot eh native tracking and using a simple baseline one can see even how much the GC increased/decreased the usage.

GC (reserved=135778KB, committed=66846KB)
(malloc=24654KB #37248)
(mmap: reserved=111124KB, committed=42192KB +79KB)

Compiler (reserved=1142KB, committed=1142KB +79KB)
(malloc=1033KB #1307)
(arena=110KB #5)

jcmd

All

If one needed to learn/use JUST ONE tool when working with java (not a GUI tool, like Jconsole or JvisualVM) it would be jcmd (MacDonald’s tools, ask it will give you):

$ jcmd <-- list processes
4131 sun.tools.jcmd.JCmd
19353 /home/fdemeloj/jboss-eap-7.2//jboss-modules.jar -mp /home/fdemeloj/jboss-eap-7.2//modules org.jboss.as.standalone -Djboss.home.dir=/home/fdemeloj/jboss-eap-7.2/ -Djboss.server.base.dir=/home/fdemeloj/jboss-eap-7.2//standalone <-- jboss

list all possible features on specific PID:

$ jcmd 19353 help
19353:
The following commands are available:
VM.native_memory
ManagementAgent.stop
ManagementAgent.start_local
ManagementAgent.start
GC.rotate_log
Thread.print <-- thread dump
GC.class_stats
GC.class_histogram
GC.heap_dump
GC.run_finalization
GC.run
VM.uptime
VM.flags <-- VM flag
VM.system_properties <-- system properties
VM.command_line
VM.version <- version
help
For more information about a specific command use 'help '.

JVM version on the spot:

$ jcmd 19353 VM.version
19353:
OpenJDK 64-Bit Server VM version 25.161-b14
JDK 8.0_161

Thread dumps:

$ jcmd 19353 Thread.print
19353:
2020-10-13 12:57:03
Full thread dump OpenJDK 64-Bit Server VM (25.161-b14 mixed mode): <--the jvm version
#And the list of threads below
"Attach Listener" #176 daemon prio=9 os_prio=0 tid=0x0000000003505060 nid=0x4d3b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"ServerService Thread Pool -- 95" #175 prio=5 os_prio=0 tid=0x00000000071fa4d0 nid=0x4cf0 waiting on condition [0x00007fd86dcf3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ae808ad8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:485)

Using just jmcd one can basically compare and even confirm there is a leak and where the leak occurs.

$ jcmd 5492 VM.native_memory
5492:
Native Memory Tracking:
Total: reserved=2987852KB, committed=1727872KB
Java Heap (reserved=1335296KB, committed=1335296KB)
(mmap: reserved=1335296KB, committed=1335296KB)
Class (reserved=1151986KB, committed=118666KB)
(classes #20430)
(malloc=3058KB #30925)
(mmap: reserved=1148928KB, committed=115608KB)
Thread (reserved=96759KB, committed=96759KB)
(thread #95)
(stack: reserved=96524KB, committed=96524KB)
(malloc=125KB #482)
(arena=110KB #188)
Code (reserved=254842KB, committed=30790KB)
(malloc=5242KB #8293)
(mmap: reserved=249600KB, committed=25548KB)
GC (reserved=105512KB, committed=105512KB)
(malloc=23192KB #17603)
(mmap: reserved=82320KB, committed=82320KB)

But of course, combined with a `jinfo`, this can be very powerful (i.e. change the JVM runtime behavior) easily:

XX:CICompilerCount=4 -XX:ConcGCThreads=2 -XX:G1HeapRegionSize=1048576 -XX:GCLogFileSize=3145728 -XX:InitialHeapSize=1367343104 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=1367343104 -XX:MaxMetaspaceSize=268435456 -XX:MaxNewSize=819986432 -XX:MetaspaceSize=100663296 -XX:MinHeapDeltaBytes=1048576 -XX:NumberOfGCLogFiles=5 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:-TraceClassUnloading -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation
[fdemeloj@fdemeloj red-irc]$ jinfo -flag -PrintGCDetails 14865
[fdemeloj@fdemeloj red-irc]$ jcmd 14865 VM.flags
14865:
-XX:CICompilerCount=4 -XX:ConcGCThreads=2 -XX:G1HeapRegionSize=1048576 -XX:GCLogFileSize=3145728 -XX:InitialHeapSize=1367343104 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=1367343104 -XX:MaxMetaspaceSize=268435456 -XX:MaxNewSize=819986432 -XX:MetaspaceSize=100663296 -XX:MinHeapDeltaBytes=1048576 -XX:NumberOfGCLogFiles=5 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:-PrintGCDetails -XX:+PrintGCTimeStamps -XX:-TraceClassUnloading -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation

With a simple GC.class_histogram, one can see the details of the number of instances per class:

$ jcmd 10197 GC.class_histogram
10197:
num #instances #bytes class name
1: 164217 20851032 [C
2: 240025 7680800 java.util.HashMap$Node
3: 162967 3911208 java.lang.String
4: 21784 3194040 [Ljava.util.HashMap$Node;
5: 46342 2609232 [Ljava.lang.Object;
6: 21603 2429160 java.lang.Class
7: 17678 1697088 java.util.jar.JarFile$JarFileEntry
8: 5805 1013840 [B
9: 20079 963792 java.util.HashMap
10: 16127 645080 java.util.LinkedHashMap$Entry
11: 19221 615072 org.jboss.vfs.spi.JavaZipFileSystem$ZipNode <--- zipnode always there!

Sure, jstat, jhat … several others have their intentions and usefulness, just watching the generations with jstat I think is a pretty good deal.

$jdk1.8.0_191/bin/jstat -gc 14512 10000 30
S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT
18944.0 17920.0 224.0 0.0 408064.0 75522.4 890368.0 32466.4 72576.0 64107.4 10368.0 8014.6 28 0.191 2 0.147 0.338

I will talk about talking a baseline on the next post.

JNLP Example

All

Continuing the JNLP part

Let’s create an HelloWorld in jnlp, i.e. an `file.jnlp` that when opened, shows Hello world:

Creating a java swing/awt class:

//Based totally in http://www.mkyong.com :D I hope he allows me
public class TestJnlp {
  static BasicService basicService = null;
  public static void main(String args[]) {
    JFrame frame = new JFrame("Example JNLP");
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
    JLabel label = new JLabel();
    Container content = frame.getContentPane();
    content.add(label, BorderLayout.CENTER);
    String message = "Yo Gabriel";

    label.setText(message);

    try {
      basicService = (BasicService)
        ServiceManager.lookup("javax.jnlp.BasicService");
    } catch (UnavailableServiceException e) {
      System.err.println("Lookup failed: " + e);
    }

    JButton button = new JButton("Button");

    ActionListener listener = new ActionListener() {
      public void actionPerformed(ActionEvent actionEvent) {
        try {
          URL url = new URL(actionEvent.getActionCommand());
          basicService.showDocument(url);
        } catch (MalformedURLException ignored) {
        }
      }
    };

    button.addActionListener(listener);

    content.add(button, BorderLayout.SOUTH);
    frame.pack();
    frame.show();
  }
}

To compile, don’t forget to add `jnlp.jar` in the classpath,

:jnlp.jar:.:

Create a jar with the file: jar -cf TestJnlp.jar TestJnlp.* And sign the jar with your keystore:

[fdemeloj@fdemeloj JNLP]$ jarsigner -keystore eap7console.jks TestJnlp.jar alias

Don’t forget to consult the JAVAWS syntax.

Results

Just do javaws Test.jnlp:

Troubleshooting

For troubleshooting, I would suggest Developer Guide JavaWS

The issues you can find:

1. Wrong location

net.sourceforge.jnlp.LaunchException: Fatal: Read Error: Could not read or parse the JNLP file. You can try to download this file manually and send it as bug report to IcedTea-Web team.
	at net.sourceforge.jnlp.Launcher.fromUrl(Launcher.java:487)
	at net.sourceforge.jnlp.Launcher.launch(Launcher.java:287)

2. JAR not signed

Caused by: net.sourceforge.jnlp.LaunchException: Fatal: Application Error: Cannot grant permissions to unsigned jars. Application requested security permissions, but jars are not signed.
	at net.sourceforge.jnlp.runtime.JNLPClassLoader$SecurityDelegateImpl.getClassLoaderSecurity(JNLPClassLoader.java:2481)
	at net.sourceforge.jnlp.runtime.JNLPClassLoader.setSecurity(JNLPClassLoader.java:385)

3. Main-class

Caused by: net.sourceforge.jnlp.LaunchException: Fatal: Initialization Error: Unknown Main-Class. Could not determine the main class for this application.
	at net.sourceforge.jnlp.runtime.JNLPClassLoader.initializeResources(JNLPClassLoader.java:774)
	at net.sourceforge.jnlp.runtime.JNLPClassLoader.<init>(JNLPClassLoader.java:338)
	at net.sourceforge.jnlp.runtime.JNLPClassLoader.createInstance(JNLPClassLoader.java:421)

Don’t try to execute as jar, this will return to you:

[fdemeloj@fdemeloj test_jnlp]$ java -jar TestJnlp.jar 
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: javax/jnlp/UnavailableServiceException
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)

At the end of the day, javaws -help is a friend

To debug:

export JAVAWS_VM_ARGS="-verbose -J-Xdebug -J-Xnoagent -J-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=y -Xnofork"

javaws Test.jnlp $JAVAWS_VM_ARGS #this starts Iced Tea Web


But this is for the third part of this series …

SCTP Protocol

All

Intro

Stream Control Transmission Protocol, or SCTP, this is very well explained in Video explaining, by Russell DeLong.

Basically, is a protocol sending streams of data between two points, with a connection previously done, and is also called TCPing.

One of the usages of SCTP, is VoIP particularly to support the telephone system’s Signaling System 7 – SS7, together with voice . SCTP also is intended to make it easier to manage connections over a wireless network and to manage the transmission of multimedia data.

SCTP is a standard protocol (RFC 2960) developed by the Internet Engineering Task Force (IETF).

Similarities with TCP:

1. SCTP manages “reliable transport” (ensuring the complete arrival of data units that are sent over the network) over the Internet’s basically connectionless Internet Protocol (IP), the protocol responsible for moving the data but not for managing whether all the data arrives.

Differences with TCP

2. SCTP ensures the complete concurrent transmission of several streams of data (in units called messages) between connected end points. SCTP also supports multihoming, which means that a connected end point can have alternate IP addresses associated with it in order to route around network failure or changing conditions.

In Java

In Java we have the `com.sun.nio.sctp` that implements SCTP.

import java.net.*;
import java.util.*;
import java.nio.*;
import com.sun.nio.sctp.*; <---- sctp

public class SctpExample
{
	public static void main(String[] args) throws Exception {
         //Server Channel:
		com.sun.nio.sctp.SctpServerChannel sc = com.sun.nio.sctp.SctpServerChannel.open(); // Open the channel
         
		com.sun.nio.sctp.SctpChannel rc = null;
		InetSocketAddress localAddr = new InetSocketAddress (InetAddress.getByName (args[0]), Integer.parseInt (args[1]));
         // Server Channel Open   
		sc.open(); //Open
		sc.bind(localAddr); //Bind the local address
		
		sc.close(); //Close
	}
}

JNI in Java AND Testing JNI with a few GCs

All

Intro

Although I did some stuff with JNI in Java during my time in schooling, I never wrote it about it here. Nor gave examples.

JNI

Java Native Interface is used basically for C/C++ interfaces, in my example, I was doing the processing of some data in Java but the data was generated (compiled, traced, generated) in C/C++ (where I would collect the data as well). (Btw, it is just me or this new way to write in wordpress with blocks sucks, like really)

That being said, the quickest example with JNI is by doing in a JAVA class

public class JNIExample{  //
   static {
      System.loadLibrary("hello");
   }
 
   private native void sayHello();
 
   public static void main(String[] args) {
      new HelloJNI().sayHello(); 
   }
}

After creating the file, we compile it:

fdemeloj@fdemeloj jni]$ javac JNIExample.java 

Then create the header with javah, and not javac -h as used to be back in Poly times. Write a HelloWorld.c:

#include <jni.h> //import jni header
#include <stdio.h> //Import the stdio
#include "HelloWorld.h" //Insert the header
  
JNIEXPORT void JNICALL 
Java_HelloWorld_print(JNIEnv *env, jobject obj)
{
     printf("Hello World!\n");
     return;
}

Write a Make file to make the compilation process easy for our Shared library

all:
	@echo "Generating Shared Library"
	cc -I. -I${JAVA_HOME}/include -I${JAVA_HOME}/include/linux -fPIC -shared -o libHelloWorld.so HelloWorld.c HelloWorld.h

clean:
	@echo "Cleaning the generated SH files"
	rm *.so

To run the code, just run java (no need for LD_LIBRARY_PATH if you are using -Djava.library.path=.)

export LD_LIBRARY_PATH=. 
[fdemeloj@fdemeloj jni]$ java HelloWorld 
Hello World!

The cool thing is, IT is a shared library totally independent from the JAVA implementation (tcharam, that’s JNI), so then if we want to change the library to this:

 Java_HelloWorld_print(JNIEnv *env, jobject obj)
 {
     printf("After change!\n");
     return;
 }

Just recompile the shared library with Make clean;Make. and It’s all good:

[fdemeloj@fdemeloj jni]$ java HelloWorld 
After change!

Next thing is to test the GC collections with JNI, from this link. From the same process, we run Makefile, that takes the java and compile it.

#include <jni.h>
#include <CriticalGC.h>

static jbyte* sink;

JNIEXPORT void JNICALL Java_CriticalGC_acquire(JNIEnv* env, jclass klass, jintArray arr) {
   sink = (*env)->GetPrimitiveArrayCritical(env, arr, 0);
}

JNIEXPORT void JNICALL Java_CriticalGC_release(JNIEnv* env, jclass klass, jintArray arr) {
   (*env)->ReleasePrimitiveArrayCritical(env, arr, sink, 0);
}

Running it with Shenandoah we see the difference and how it is so fast:

[fdemeloj@fdemeloj critical-gc]$ make run-shenandoah 
time java -Djava.library.path=. -Xms4g -Xmx4g -verbose:gc -XX:+UseShenandoahGC CriticalGC
Acquired
Releasing
Acquired
Releasing
Acquired
Releasing
Acquired
Releasing
Acquired
[Pause Init Mark, 0.533 ms]
[Concurrent marking 2880M->2988M(4096M), 216.052 ms]
[Pause Final Mark 2988M->920M(4096M), 2.220 ms]
[Concurrent reset bitmaps 922M->926M(4096M), 0.946 ms]
Releasing
Acquired
Releasing
Acquired
^CCancelling concurrent GC: Stopping VM
Releasing
Acquired

I mean, comparing with the other GC’s approaches this Shenandoah doesn’t even stop.

REFERENCE:

Following this reference

PodCast `Hora do Lancamento`

All

Hora do Lancamento

Gabriel Alabarse and me, we started a podcast to talk about some of the launches that currently happen: basically SpaceX and RocketLab. We are trying to fight the HUGE amount of MISINFORMATION there is around, particularly in Youtube in Portuguese. Abnormal amount. Big channels, with millions of views spreading conspirational theories.

The explanation for being in Portuguese more than English, is because we have several very good channels that explain the basics for the general population in English, examples: Everyday Astronaut, SpaceXcentric, Scott Manley, 2 The Future Together, SpaceXfans, and the Angry Astronaut, besides the launches from SpaceX, ULA, RocketLab, all content in English.

It is just the first try:

-Francisco

Infinispan/JDG Quickstarts Pt5 – Java HotRod

All

Intro

Another Hot Rod tutorial, this time with java, similar concept but using a Hot Rod java client. Just a reminder, basically TCP client that access the cache on JDG.

Core part

There is the `​FootballManager` and the Team, which are pretty much the classes that play a role here (except for the user). So then a FootballManager has team(s) and so forth. The team just encapsulates the list of players and the name. The core part (talking with the cache itself) is on the FootballManager class.

ConfigurationBuilder builder = new ConfigurationBuilder();
builder.addServer()
.host(jdgProperty(JDG_HOST))
.port(Integer.parseInt(jdgProperty(HOTROD_PORT)));
cacheManager = new RemoteCacheManager(builder.build());
cache = cacheManager.getCache(“teams”);

The main part is then the builder adding the server with jdgProperty (from a file) and the port. Then it adds the players. The file has basically the host and the port:

jdg.host=localhost
jdg.hotrod.port=11222

To read the commands,  it is used java.io.Console in comparison with the javascript version (that uses Vorpalo=||==>).

Interesting Topics

– Originally the players added were Messi, Pedro, and Puyol. I changed for Pele, Tostao, and Rivelino (Tri-world champions on the1970th World Cup in Mexico)

– Comparing with the JS version: This version is done in Java,that’s the only difference. Although js has a simpler connection with JDG.

– Interestingly, java.io.Console reading is faster than bufferedReader

REFs

inline

 

Mocking request http

All

Intro

Recently I needed to simulate some requests http so then I used EasyMock. Quite useful indeed and very simple:

Code

HelloServletMock mock = new HelloServletMock();

HttpServletRequest request = createMock(HttpServletRequest.class);
HttpServletResponse response = createMock(HttpServletResponse.class);

expect(request.getParameter(“yourName“)).andReturn(“Jose Carlos“).times(1, 2);
replay(request);

On this case the request will return “Jose Carlos” when I get the paramter “yourName”, example:

String yourName = request.getParameter(“yourName”);