feat(k8s): add kubernetes metrics source

[zeldanut]

fix(headers): remote task custom headers additions

allow for the passing of extra headers through an env var to downstream
requests for remote task executions including tap & metric reporting

ref: LOG-23416

---

feat(k8s): add kubernetes metrics source

port of mezmo agent module that allows for the scraping of kubernetes
metrics-server api and the format created is specific to the format
mezmo expects for the kubernetes enrichment feature

ref: LOG-23416

[kristof-mattei]

Nit/correctness:

Going from &str to String works with to_string(), but ideally we'd use to_owned(). to_string() goes via std::fmt::Display, and is used to create a user-suitable representation of the receiver.

It happens to be that for String to_string() yields the same as to_owned() but semantically they're different.

[kristof-mattei]

Prefer not using mutable variables as it makes it harder to reason about the code.

[kristof-mattei]

Instead of using mutable you can write this as follows:

        let extract = |map: Option<_>| {
            map.map(|m: &BTreeMap<_, _>| {
                (
                    m.get("cpu")
                        .and_then(|q: &Quantity| convert_cpu_usage_to_milli(q.0.as_str())),
                    m.get("memory")
                        .and_then(|q: &Quantity| convert_memory_usage_to_bytes(q.0.as_str())),
                )
            })
            .unwrap_or_default()
        };

        let (cpu_limit, memory_limit, cpu_request, memory_request) = c
            .resources
            .as_ref()
            .map(|resources| {
                let (cpu_limit, memory_limit) = extract(resources.limits.as_ref());
                let (cpu_request, memory_request) = extract(resources.requests.as_ref());
                (cpu_limit, memory_limit, cpu_request, memory_request)
            })
            .unwrap_or_default();

[kristof-mattei]

Thoughts on implementing AddAssign?

[kristof-mattei]

HEre we can also get rid of the let mut denominator:

let denominator = match unit.as_str() {
    "m" => {
        1.0
    },
    "u" => {
        1000.0
    },
    "n" => {
        1000000.0
    },
    &_ => {
        error!("Unknown CPU unit");
        return None;
    }
}

[kristof-mattei]

Can we make headers already a HeaderMap here? I understand this is a more foundational change. String is case-sensitive, HeaderMap's HeaderName is case-insensitive (as per the HTTP spec).

Regardless of that:

I'd like to suggest the following change:

        let mut headers = env::var("MEZMO_REMOTE_TASK_EXTRA_HEADERS")
            .ok()
            .and_then(|v| serde_json::from_str::<HashMap<String, String>>(&v).ok())
            .unwrap_or_default();

        // Option 1: if `MEZMO_REMOTE_TASK_EXTRA_HEADERS` provides `Authorization:`, DON'T overwrite
        {
            // note that if we'd switch to hashbrown, we can delay the `to_owned()` until it's actually needed to be created. 
            headers
                .entry("Authorization".to_owned())
                .or_insert_with(|| format!("Token {token}"));
        }

        // Option 2: if we want our `Authorization:` to ALWAYS win:
        {
            headers.insert("Authorization".into(), format!("Token {token}"));
        }

[kristof-mattei]

Do we want log something here? This quietly eats any headers that cannot be represented as HeaderName / HeaderValue.

[kristof-mattei]

Actually, if we don't expect the value of MEZMO_REMOTE_TASK_EXTRA_HEADERS to change during the runtime of the application, we might want to use a LazyLock to create the hashmap once, and then clone it each time this call happens. That way we skip over the deserialization step.