LPS Network Endpoint Enhancements and Signaling

[milan-zededa]

Description

Implements the EVE side of the API additions introduced in
lf-edge/eve-api#144.

Three API enhancements plus two ancillary improvements:

1. Signal endpoint — low-latency LPS config notifications (GET /api/v1/signal)

EVE now maintains a persistent long-lived HTTP GET connection to the optional
/api/v1/signal endpoint on the Local Profile Server. LPS can push a
Signal proto message (NDJSON-framed) listing which endpoints have a pending
configuration change. On receipt, EVE immediately polls the listed endpoints
instead of waiting for the next scheduled tick.

Implementation details:

• New goroutine in pkg/pillar/localcommand/signal.go handles the stream
  lifecycle independently of the watchdog.
• TCP keepalive at 60 s is the sole liveness mechanism (no app-layer heartbeat),
  as required by the spec.
• Exponential backoff on failure (1 s initial, 2× doubling, 30 s cap).
• If LPS returns 404 (endpoint not implemented), reconnect attempts are throttled
  to one per hour so a non-signaling LPS does not generate excess traffic.
• Rate limiter (one signal per 3 s, burst 3) guards against a buggy or malicious
  LPS overwhelming EVE with spurious triggers. Periodic polling is the
  correctness guarantee; dropped signals are safe.
• The stream is restarted whenever the LPS address changes (UpdateLpsConfig).
• Unknown ConfigEndpoint enum values are silently ignored for forward
  compatibility.

2. Per-port runtime status (NetworkInfo.port_status)

EVE now populates the new NetworkPortStatus repeated field in every
POST /api/v1/network request. For each network port it reports the
kernel-observed runtime state: link up/down, MAC address, assigned IP
addresses (CIDR), active default gateways, DNS servers, DNS search domain,
NTP servers, and MTU.

3. local_modifications_allowed per port in NetworkInfo.latest_config

EVE sets NetworkPortConfig.local_modifications_allowed when reporting
latest_config in NetworkInfo, mirroring the controller-provisioned
SystemAdapter.allow_local_modifications flag. LPS can use this to know
upfront which ports accept locally submitted configuration, rather than
discovering it via a trial-and-error error message.

4. Reactive LPS network POST on config/status change

EVE now triggers an immediate POST /api/v1/network when either:

• DevicePortConfigList changes (new network configuration applied), or
• DeviceNetworkStatus changes with a meaningful state update (link state,
  IP assignment, etc.).

Previously the network endpoint was only driven by the periodic ticker.

5. Configurable LPS polling intervals

All six LPS polling intervals are now tunable via controller config
properties (previously they were compile-time constants):

Property	Default	Min	Max
timer.lps.profile.interval	60 s	3 s	1 h
timer.lps.radio.interval	5 s	3 s	1 h
timer.lps.appinfo.interval	60 s	3 s	1 h
timer.lps.devinfo.interval	60 s	3 s	1 h
timer.lps.network.interval	60 s	3 s	1 h
timer.lps.appbootinfo.interval	60 s	3 s	1 h

Defaults match the previous hard-coded values, so behaviour is unchanged
without explicit configuration.

How to test and validate this PR

Prerequisites

You need a running EVE device with a deployed Local Profile Server application.
Consider using my LPS implementation.

1. Validate local_modifications_allowed in NetworkInfo

LPS side: In the handler for POST /api/v1/network, inspect each entry in
network_info.latest_config.ports. Ports provisioned by the controller with
allow_local_modifications = true must arrive with
local_modifications_allowed = true; those provisioned without it must arrive
with false.

Expected behaviour: LPS can display a read-only indicator next to ports
that it is not permitted to reconfigure, and can pre-validate a
LocalNetworkConfig submission before sending it.

2. Validate per-port runtime status (port_status)

LPS side: In the same POST /api/v1/network handler, inspect
network_info.port_status. For each network port you should see:

• logical_label — matches the port label from latest_config
• interface_name — kernel interface name (e.g., eth0)
• link_up — true when the port has carrier
• mac_address — colon-separated hex (e.g., aa:bb:cc:dd:ee:ff)
• ip_addresses — CIDR notation (e.g., 192.168.1.10/24, fd00::1/64)
• gateways — active default routers
• dns_servers / dns_domain — effective resolver configuration
• ntp_servers — NTP peers in use
• mtu — effective MTU in bytes

To verify reactivity (enhancement 4): Change a network configuration on
the device (e.g., add a static route, trigger a DHCP renewal). LPS should
receive an updated POST /api/v1/network within a few seconds, not after the
next 60-second tick.

3. Validate the Signal endpoint — basic trigger

LPS side: Implement GET /api/v1/signal as a streaming endpoint that:

1. Keeps the HTTP connection open indefinitely (use chunked transfer encoding).
2. When a user submits a configuration change (e.g., via a UI that calls
   PUT /api/v1/local_profile), immediately writes a single NDJSON line:

   {"pendingChanges":["CONFIG_ENDPOINT_LOCAL_PROFILE"]}\n
   

   and flushes the response buffer.
3. Does NOT write anything until there is actually a pending change.
4. Does NOT write heartbeats or empty lines.

EVE side: After receiving the signal, EVE must immediately fetch
GET /api/v1/local_profile and apply the new profile — without waiting for
the next 60-second tick.

Verification: Measure the latency between submitting the config change on
LPS and EVE applying it. With signaling it should be under 2 seconds; without
signaling it would be up to 60 seconds.

4. Validate the Signal endpoint — multi-endpoint coalescing

Send a signal listing multiple endpoints simultaneously:

{"pendingChanges":["CONFIG_ENDPOINT_NETWORK","CONFIG_ENDPOINT_APP_INFO"]}\n

EVE must trigger both the network POST and the app info POST immediately.

5. Validate configurable polling intervals

Using the controller, set:

timer.lps.profile.interval = 10

Observe in EVE logs that the local profile is now fetched every 10 seconds
instead of every 60 seconds. Restore the default when done.

Changelog notes

• Low-latency LPS configuration updates. EVE can now receive near-instant
  notifications from the Local Profile Server when new configuration is ready,
  reducing the delay from up to 60 seconds to under 2 seconds. LPS
  applications must implement the optional GET /api/v1/signal streaming
  endpoint to take advantage of this; existing LPS deployments without the
  endpoint continue to work exactly as before.

• Richer network status in LPS. EVE now reports the live kernel-observed
  state of every network port to LPS on each network POST: link up/down,
  assigned IP addresses (with subnet mask), default gateways, DNS and NTP
  servers, and MTU. LPS applications can display this information to operators
  without needing a separate management channel.

• Per-port modification permission flag. LPS now knows which network ports
  it is allowed to reconfigure (as provisioned by the controller), so it can
  give operators clear feedback instead of a generic error when they attempt to
  modify a controller-managed port.

• Reactive LPS network updates. EVE now pushes an updated network POST to
  LPS immediately when the device's network configuration or link state changes,
  rather than waiting for the next periodic tick.

• Tunable LPS polling intervals. All six LPS polling intervals (local
  profile, radio, app info, device info, network, app boot info) can now be
  adjusted via controller config properties. The defaults are unchanged.

PR Backports

- 16.0-stable: Yes. Even though this is strictly speaking a new feature, we must make an exception here...
- 14.5-stable: No 
- 13.4-stable: No

Checklist

• I've provided a proper description
• I've added the proper documentation
• I've tested my PR on amd64 device
• I've tested my PR on arm64 device
• I've written the test verification instructions
• I've set the proper labels to this PR
• I've checked the boxes above, or I've provided a good reason why I didn't check them.

[codecov]

Codecov Report

❌ Patch coverage is 5.05837% with 244 lines in your changes missing coverage. Please review.
✅ Project coverage is 22.01%. Comparing base (63fa63d) to head (d22cfef).
⚠️ Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/pillar/localcommand/signal.go	0.00%	146 Missing ⚠️
pkg/pillar/controllerconn/send.go	0.00%	38 Missing ⚠️
pkg/pillar/localcommand/network.go	0.00%	28 Missing ⚠️
pkg/pillar/localcommand/agent.go	0.00%	12 Missing ⚠️
pkg/pillar/cmd/zedagent/zedagent.go	0.00%	7 Missing ⚠️
pkg/pillar/localcommand/profile.go	0.00%	5 Missing ⚠️
pkg/pillar/localcommand/appbootinfo.go	0.00%	2 Missing ⚠️
pkg/pillar/localcommand/appinfo.go	0.00%	2 Missing ⚠️
pkg/pillar/localcommand/devinfo.go	0.00%	2 Missing ⚠️
pkg/pillar/localcommand/radio.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5904      +/-   ##
==========================================
+ Coverage   21.62%   22.01%   +0.38%     
==========================================
  Files         464      475      +11     
  Lines       83994    85923    +1929     
==========================================
+ Hits        18166    18912     +746     
- Misses      64300    65304    +1004     
- Partials     1528     1707     +179     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[eriknordmark]

Run tests

But some yetus annotations to fix (however, yetus shows as passing which is new to me!)

[rene]

Run tests

But some yetus annotations to fix (however, yetus shows as passing which is new to me!)

If you look to the artifacts, Yetus is 100% passing because it checks only the patch (what it was changed), so if you go to the final result files, they are all blank (no errors).

However, the annotation step runs over the patch-*-result files, which shows all errors for the file and not only what was changed.... the annotation step will not fail (I did it on purpose). I think we can keep this approach while we solve the issues as they show up on files we are changing...

But it's important to point out that any error introduced by the PR will make Yetus to fail.... so the check is working.....

[milan-zededa]

Run tests
But some yetus annotations to fix (however, yetus shows as passing which is new to me!)

If you look to the artifacts, Yetus is 100% passing because it checks only the patch (what it was changed), so if you go to the final result files, they are all blank (no errors).

However, the annotation step runs over the patch-*-result files, which shows all errors for the file and not only what was changed.... the annotation step will not fail (I did it on purpose). I think we can keep this approach while we solve the issues as they show up on files we are changing...

But it's important to point out that any error introduced by the PR will make Yetus to fail.... so the check is working.....

I think this approach is quite annoying for the reviewer. Furthermore, these are not "issues", but suggestions. We may not agree with everything pointed out by Yetus, so we will intentionally ignore some, but they will keep popping up...

[eriknordmark]

However, the annotation step runs over the patch-*-result files, which shows all errors for the file and not only what was changed.... the annotation step will not fail (I did it on purpose). I think we can keep this approach while we solve the issues as they show up on files we are changing...

This is at best confusing (because those annotations show up in the diffs the same way as when they cause failures) and a bit annoying. Can we have the annotations be limited to the changed lines as before.

[milan-zededa]

Rebased on the top of the latest master.

[eriknordmark]

Does it make sense to add a general note about all these below the table that if the device gets a 404(??) it will back off the timer a lot?

[eriknordmark]

If/when the long-lived connection dies and is re-established is the re-establishment done automatically by EVE?

Who is responsible for ensuring that the LPS can track the current state after the re-establishment?
Should the LPS notice that there is a new TCP/HTTPS connection and use that to trigger fetching/updating the current state? If so, would be be useful if EVE sent a signal over the long-lived connection immediately after it has been re-established the connection so the LPS doesn't need to detect new connections at the lower layers?

[milan-zededa]

If/when the long-lived connection dies and is re-established is the re-establishment done automatically by EVE?

Who is responsible for ensuring that the LPS can track the current state after the re-establishment? Should the LPS notice that there is a new TCP/HTTPS connection and use that to trigger fetching/updating the current state? If so, would be be useful if EVE sent a signal over the long-lived connection immediately after it has been re-established the connection so the LPS doesn't need to detect new connections at the lower layers?

This signal endpoint is really just an optimization to minimize latency. If we loose some signal from LPS for whatever reason, the only consequence is just longer latency (like it is today without this enhancement).
So we do not impose that LPS must react to re-connection in any way. A good implementation could track all pending changes and re-emit them when reconnection is detected (a new GET request is received for /signal). But a "dumb" implementation which does not keep track of pending changes and simply sends&forget signal on any user input will functionality-wise work as well (even though user-experience is not the best in these edge-cases).

[eriknordmark]

A few comments:

1. No unit tests for signal.go. A new ~300-line state machine with five
   outcomes, an exported public API (OpenLocalStream), and intricate
   cancel/restart semantics ships without a single test. At minimum, handleSignal
   (rate-limit + dispatch table) and readSigHandlerStream (NDJSON framing +
   malformed-line tolerance) are easy to unit-test with an io.Pipe. Worth adding
   before merge given the long-lived-connection complexity.

2. sigHandlerLimiter config inconsistency. Burst is 3 over a 3 s refill, so a
   single legitimate signal listing all 6 endpoints will not exceed the limit (it
   dispatches synchronously inside handleSignal and consumes 1 token, not 6 —
   Allow() is called once per signal, not per endpoint). That is fine for normal
   use, but the comment says "periodic polling is the correctness guarantee;
   dropped signals are safe" while the limiter is actually quite generous.
   Consider documenting that the limiter is per-message, not per-endpoint, so
   future maintainers don't tighten it incorrectly.

3. TCP keepalive is key for the robustness, but if we fail to set it there isn't even any logs in. Makes sense adding such logs.
   (controllerconn/send.go:111-115):
   if tcpConn, ok := conn.(*net.TCPConn); ok {
   _ = tcpConn.SetKeepAlive(true)
   _ = tcpConn.SetKeepAlivePeriod(keepAlive)
   }

4. If DialerWithResolverCache ever wraps the connection (e.g., for proxying),
   keepalive — the only dead-peer detection mechanism per the design — is
   silently disabled. At least log at Trace level when the assertion fails so
   this doesn't become a silent debugging problem later.

5. Empty mask → no /N in CIDR string. dnsPortsToProto builds &net.IPNet{IP:
   addr.Addr, Mask: addr.Mask} and calls String(). If addr.Mask is nil/empty
   (possible on a freshly persisted DPC before updateDNS has run, or if a code
   path forgets to populate it), IPNet.String() returns just the IP — violating
   the proto comment that promises CIDR format. Either guarantee Mask presence by
   construction, or fall back to /32 or /128 based on len(addr.Addr) so the wire
   format is always CIDR.

6. Log line could be large (signal.go:864): Warnf(... line=%q) logs up to 64
   KiB of attacker-controlled bytes per malformed line. Truncate to ~256 chars in
   the format string.

7. Unbounded pendingChanges slice.
   A malicious LPS can send a Signal with pendingChanges containing millions of
   entries (within the 64 KiB line cap, that's roughly 2k–5k enum repetitions).
   Each is iterated in handleSignal. Iteration is O(n) with cheap per-element
   work (one switch + one tickNow), so a 5k-entry signal completes in
   microseconds. Not exploitable, but a len(pendingChanges) > 32 early-return
   would cost nothing.

8. add a one-line // see eve-api
   PROFILE.md ### Signal — auth model is intentional
   comment near OpenLocalStream

For #3 and #4 a suggestion is:
Push keepalive into the dialer itself. net.Dialer has a KeepAlive
time.Duration field that handles this internally. Plumb a KeepAlive field
through DialerWithResolverCache so the inner net.Dialer is constructed with
it:

stdDialer := net.Dialer{
Resolver: resolver.getNetResolver(),
LocalAddr: &net.TCPAddr{IP: d.localIP},
Timeout: d.timeout,
KeepAlive: d.keepAlive, // <-- new
}