[Jumpcloud] - Add support for a 'source_lag_time' config option #15248
Description
Jumpcloud integration currently uses start_time and last_event.timestamp to control pagination. The issue happens when these to values overlap i.e ( last_event.timestamp == start_time of next cycle) leading to duplication of the first event in almost every pagination cycle. Recent issue have also highlighted that there is often times a delay at the APIs end in populating the relevant events and our current method of polling miss out on these delayed events.
To mitigate this we need to provide an option for configuring a pagination time window for the polling to occur in. A source_lag_time can help us achieve this. According to the official API Docs, end_time is supported as a request parameter.
We would set end_time = now-{{source_lag_time}},
At the same time we need to update start_time according to the following logic to maintain a proper polling window.
if start_time > now - {lag}:
start_time = now - {lag}
else
start_time = '[[.cursor.timestamp]]'
end_time = now - {lag}
This gives us a buffer which helps us avoid pulling duplicates on each pagination and also gives us the window to capture events which might have been missed due to a delay in getting populated as observed in recent issues.