[kn-admin] Proposal: support downloading profiling data for knative components

[xliuxu]

Currently we are using profiling data to debug knative performance issues. I think it would be nice to support download related profiling data directly using kn admin plugin.
Here are three steps to get a profiling data for knative components:

1. Enable tracing by updating the config-observability ConfigMap and setting profiling.enable = "true"
2. Enable port-forwarding to get access to Knative Serving pods.
3. Download specific profiling data at http://localhost:8008/debug/pprof/

There are 8 types of profiling data could be fetched according to https://github.com/knative/serving/blob/master/test/performance/profiling.md

• heap
• profile
• block
• trace
• allocs
• mutex
• goroutine
• threadcreate

For profile and trace, a optional seconds parameter could be specified.

I think for kn-admin plugin, we could download profiling data in one command like

• kn admin profiling --enable/--disable to enable/disable profiling.
• kn admin profiling -t/--target autoscaler --heap to download heap profile of auto scaler. Will enable profiling if it is currently not enabled. The same goes for the following commands.
• kn admin profiling -t/--target networking-istio --trace 2m for a 2 minutes CPU trace profile of networking-istio.
• kn admin profiling -t/--target activator --all to download all profiling data avaliable for all activator pods with a default 30s duration for tracing and profile.

If there are more than one pod avaliable, the default behaviour should download profiling data for all pods. Also we can set a specific pod name using -t/--target flag

• kn admin profiling -t/--target activator-5979f56548 --all to download all profiling data avaliable for pod activator-5979f56548.

The plugin should be responsible to provide a meaningful filename for downloaded profiling data. e.g.
2020-07-09T10:09:48Z-activator-5979f56548-54crk-heap.pb.gz. A -d flag could be supported to specify the download directory.

cc @zhanggbj

[chaozbj]

Thanks @lanceliuu for refining this proposal and make it more clear. @lanceliuu and I had a short meeting to discuss how to co-work on this issue. @lanceliuu will focus on the utility functions to serve profiling command, such as: forwarding to local port, download profile data, and I will focus on the command part and use the utility codes to implement the command functions.
/assign

[zhanggbj]

@maximilien @julz @duglin this is the proposal as discussed^^^^^, thanks!

[maximilien]

Thanks @zhanggbj @lanceliuu

Will review this DRAFT. Let's move this quickly so you all can demo it soon to wider audience. Thanks 🙏