Live bookmarks are not loading new posts from www.flyertalk.com, even when doing a manual refresh

[zem99m]

Add-on version
Add-on version: 2.8 in Firefox 77.0.1, Windows 10 Pro 1909 (x64). New installation on new laptop out of the box.

Describe the bug
New Windows 10 install. Firefox 77.0.1 installed and Livemarks add-on installed.
Set up Livemark for an RSS feed on www.flyertalk.com for the "British Airways | Executive Club" forum at this URL: https://www.flyertalk.com/forum/british-airways-executive-club-446/ (see reproduction notes below for details)
The posts (threads) from the forum page were successfully loaded by Livemarks and I verified that links are followed successfully.
Today I noticed that the live bookmark had not updated since June 19th, and I verified that there have been new posts (threads) in that forum since that date.
I performed a manual "Reload Live Bookmark", but this did not update the list of posts in the live bookmark.

I found that if I select the Livemarks icon in the address bar (as if I were adding a new livemark), the page generated by Livemarks lists the same posts which are currently in the live bookmark.
The bookmark for the RSS feed is https://www.flyertalk.com/forum/external.php?type=RSS2&forumids=446
Then I clicked on the "View source" link at the top of the page generated by Livemarks - this successfully opened the source code. After doing this, I used the browser's back button to go back to the list of posts that had been generated by Livemarks, at which point the page was automatically updated with the latest posts since June 19th!
After doing this, it was possible to update the live bookmark by selecting the "Reload Live Bookmark" option, and the live bookmark now shows all recent posts.

Please note that RSS on www.flyertalk.com is oddly configured. On selecting the Livemarks icon in the address bar, two feeds are offered - the first is a feed of recent posts in all forums on www.flyertalk.com - the second is the feed just for the single "British Airways | Executive Club" forum.
The odd thing (as far as I am concerned) is that when clicking on the live bookmark and selecting the option "Open " at the top of the list, the page that is opened is the top level of the www.flyertalk.com "All forums" feed, not the specific "British Airways | Executive Club" forum page, which I had actually bookmarked.

To Reproduce
Steps to reproduce the behavior:

1. Go to https://www.flyertalk.com/forum/british-airways-executive-club-446/
2. Click on the RSS icon in the address bar.
3. Select the second of the two feeds offered - "Flyertalk forums - British Aiways | Executive Club - RSS feed"
4. On the generated page, select "Add Livemark"
5. Wait for a few days - there are always a few new posts (threads) in that forum each week.
6. Go to https://www.flyertalk.com/forum/british-airways-executive-club-446/ to verify that there have been new posts in the days since you added the live bookmark - the date when a thread was started is shown directly beneath the bolded title of the thread.
7. Reload the live bookmark and check if the new posts (threads) are displayed in the live bookmark. If the new posts are not displayed, refer to the instructions in the bug description above.

Expected behavior
The live bookmark should be updated to add in all recent posts when it is reloaded. Ideally, this should happen automatically.

Extra context
This problem only affects live bookmarks from www.flyertalk.com. I have created live bookmarks for several other sites, which function as expected. For the live bookmarks I've created for other sites, where more than one RSS feed is offered when clicking on the Livemarks icon in the address bar, I have selected the first feed in the list - this is different from the live bookmark for www.flyertalk.com, where I have selected the second feed in the list offered.

[zem99m]

Since reporting this issue, I've noticed that my live bookmarks for forums on www.flyertalk.com are occasionally being refreshed by Livemarks automatically, This does not happen every day - on 7 of the past 9 mornings when I check the live bookmarks for www.flyertalk.com they contain exactly the same links as the previous day. I do not know what Livemarks is looking at to decide whether the live bookmark needs to be updated. All my other live bookmarks are being automatically updated throughout the day whenever a new link is added to the target RSS feed.

[evilpie]

Hard to say what is going wrong. By default we refresh all livemarks every 5 minutes. Because other feeds seem to work correctly for you I suspect the flyertalk.com server is maybe to blame.

Or maybe we are not actually fetching the feed, but instead using a cached copy.

[zem99m]

Thanks for your response. I've watched live bookmarks for this site over the past few days, and they finally updated automatically this morning after not changing for five days. The native live bookmarks in the browser on my old laptop - still running (ancient) Firefox 52.9.0 ESR - update throughout the day every day, so I know that Firefox's native live bookmarks worked fine for this site's feeds before Mozilla discontinued support in FF64.

For the example URL - https://www.flyertalk.com/forum/external.php?type=RSS2&forumids=446 - what is Livemarks checking to see if the feed has been updated and needs to be refreshed?

I ask because I wonder if the site is not generating the RSS feed data in the precise format expected by Livemarks - for example, if Livemarks is checking a particular ctime value, maybe that value is not being set or updated correctly.

[zem99m]

Here is an additional data point...

In Livemarks I have another live bookmark for the RSS feed from another web site, which publishes a web comic three days a week on Monday, Wednesday and Friday, at this URL: https://www.gunnerkrigg.com/rss.xml

Using native live bookmarks in FF 52.9.0 ESR on my old laptop, the newest link in the RSS feed read by Firefox is a link for today's (Monday's) comic, with the title "Chapter 77: page 9".

With Livemarks in Firefox 78, the link for today's comic is not in the feed - the top entry in the live bookmark displayed by Livemarks is last Friday's comic, with the title "Chapter 77: Page 8". This behavior persists if I manually reload the live bookmark, or if I close and restart Firefox.

By using "view-source:https://www.gunnerkrigg.com/rss.xml" to view the content of the rss.xml file, the copy loaded by Livemarks in FF78 contains this line, which shows the XML file was built on Friday July 10th.:

     <lastBuildDate>Fri, 10 Jul 2020 00:00:00 -0700</lastBuildDate>

Using "view-source" in the same manner on my old laptop, the rss.xml file loaded by FF52.9.0 ESR in native mode contains this line, which shows that XML file was built today:

     <lastBuildDate>Mon, 13 Jul 2020 00:00:00 -0700</lastBuildDate>

It appears that Livemarks 2.8 under FF78 is loading an older copy of the rss.xml than FF52's former native live bookmark feature did. How could this happen? I'm guessing that there might be a difference in the way that Livemarks and Firefox's (now unsupported) native live bookmarks feature obtain the RSS feed data from the web site.

I attach the output of "view-source:https://www.gunnerkrigg.com/rss.xml" that was generated by FF52 (native) and FF78 (Livemarks).
FF52-gunnerkrigg.com-rss.xml.txt
FF78-gunnerkrigg.com-rss.xml.txt

[zem99m]

Looked at this problem again yesterday and today. As you suggested, this looks like a caching problem. If I notice that a live bookmark for flyertalk.com hasn't changed since yesterday, and I try "Reload Live Bookmark", nothing changes. If I clear my browser's cache and retry the reload, the live bookmark is immediately updated with new recent links added to the list. I assume that the XMLHttpRequest open is returning a 304 for the GET request. I guess we don't want to force a bypass of the cache every time. I don't know enough to suggest what could be going wrong here, or an appropriate way forward.

[ehasis]

I'm having the same issue, in my case is https://devblogs.microsoft.com/feed/landingpage If I open the address in other browser, it shows correctly.

[zem99m]

This problem has been resolved by the feed request caching fix in version 3.4. Thank you.