JVM flags for work both with kubernetes port-forward and cryostat

[grzesuav]

I am experimenting with having both options working:

1. JMX work via port-forward to pod from localhost, i.e. in JavaMissionControl
2. JMX work via cryostat

JVM options for those to work are:

1. -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9091 -Dcom.sun.management.jmxremote.rmi.port=9091 -Djava.rmi.server.hostname=127.0.0.1 came across this in https://iceburn.medium.com/how-to-expose-jmx-in-kubernetes-b1faad450451
2. -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9091 -Dcom.sun.management.jmxremote.rmi.port=9091

What is important, adding -Djava.rmi.server.hostname=127.0.0.1 to target JVM make it work with port-forward, but then cryostat stops working - it throwns bunch of connection exception in the logs and is unable to connect/start recording/create JVM identifier`.

I am still experimenting to make both options working, but if anyone has encountered something similar I would appreciate any hints.

[grzesuav]

To showcase

1. with -Djava.rmi.server.hostname=127.0.0.1

Cryostat has issues with connecting to target vm -

port-forwarding works

2. without -Djava.rmi.server.hostname=127.0.0.1 -

cryostat correctly detect JVM and can connect

however then port-forward do not work -

[grzesuav]

on separate note I tried to use -Djava.rmi.server.hostname=0.0.0.0 but it does not work either - in that case it don' work for boths cases - cryostat and port-forward

[igorbljahhin]

Same issue here, @andrewazores can you elaborate with this?

[andrewazores]

This isn't really a Cryostat question, it's more of an OpenJDK JMX and Kubernetes port-forwarding question. The problem is that the JVM just wants to bind its JMX port (ex. 9091) on one local (within its container) interface. I think by default it chooses its own default machine hostname, ex. sample-app, which in Kubernetes will work for Cryostat because that hostname is also network resolvable via DNS. However, when you use kubectl port-forward you are making a tunnel from that container's port to your local machine's localhost, and then trying to use JMC (or other tool) to connect to localhost:9091, but your JVM application is only accepting connections on sample-app:9091.

If you change the configuration so that the application is binding to localhost:9091 or 127.0.0.1:9091 then this does match the port-forward case, but it no longer accepts the incoming JMX connections from Cryostat, which is trying to use sample-app:9091.

So it seems like what we need in order for this to directly work, is that the target application's JVM needs to bind on both localhost and sample-app, or else bind on all available interfaces. Usually the 0.0.0.0 value can be used to mean "bind on all interfaces", but in my testing it seems like the JVM doesn't like that setting at all and ends up accepting no connections whatsoever.

Another workaround could be to use the Cryostat Agent with JMX enabled. Then Cryostat will be able to talk to the application via the Agent and its embedded HTTP(S) server, but you can also have JMX enabled and bound on localhost which you can then access via port forwarding.

[anguzo]

TL;DR Setting java.rmi.server.hostname={service}.{namespace}.svc.cluster.local works, allowing it to be mapped locally for JDK Mission Control with port-forwarding as well as works in Cryostat.

I was experimenting to determine which hostname (or port) RMI defaults to when java.rmi.server.hostname is not explicitly set. It seems that it defaults to the pod's hostname and address. However, I also tried setting java.rmi.server.hostname={service}.{namespace}.svc.cluster.local, and it actually worked!

In both cases, InetAddress.getLocalHost().getHostName(), InetAddress.getLocalHost().getCanonicalHostName(), and InetAddress.getLocalHost().getHostAddress() returned the same values, so I believe setting java.rmi.server.hostname={service}.{namespace}.svc.cluster.local is a viable option.

This allows you to set this hostname in the local "hosts" file to 127.0.0.1 and specify {service}.{namespace}.svc.cluster.local:9091 to use JDK Mission Control with a port-forwarded app.

[andrewazores]

Yep, that'll work. Editing your workstation machine's hosts file so that the port-forwarded Pod's hostname resolves back to 127.0.0.1 from the workstation is a bit of a chore and probably not the most scalable, but it should be fine for one-off work.

[grzesuav]

@anguzo can you elaborate on above ? I mean in your case java.rmi.server.hostname value is {service}.{namespace}.svc.cluster.local or pod ip actually ?

Regarding

This allows you to set this hostname in the local "hosts" file to 127.0.0.1 and specify {service}.{namespace}.svc.cluster.local:9091 to use JDK Mission Control with a port-forwarded app.

how actually you do it automatically on nodes? I wonder if it is viable solution when you have multiple pods on the same node

[anguzo]

@grzesuav We don't need automating the hosts files across nodes, this is only needed on local machine to work with JMC.
As I stated when java.rmi.server.hostname={service}.{namespace}.svc.cluster.local (the FQDN) the InetAddress.getLocalHost() values still return pod hostname and address (not the FQDN).
Also in Cryostat pod names are shown, not FQDN, example:

1. service-664ddf56d7-bdvlz (service:jmx:rmi:///jndi/rmi://11-97-12-3.namespace.pod:1093/jmxrmi)
2. service-664ddf56d7-pprrp (service:jmx:rmi:///jndi/rmi://11-97-7-5.namespace.pod:1093/jmxrmi)

We are still testing how it works with multiple pods of the same service but from the looks of it they are monitored in Cryostat correctly and independently.