Checking for url's with content that may change over time

[tkelman]

Related to JuliaLang/julia#10743 (comment)

We already have a case of someone setting LIBGIT2_VER to master and that ending up in the cache, so now there's an out-of-date copy of libgit2-master.tar.gz on the server. We should probably blacklist downloads that include master, and openblas' develop for example.

There's a copy of busybox.exe on the cache which can be cleared out, I modified the url for that to point to a versioned copy but the unversioned exe will go stale. objconv.zip is also not versioned upstream and should maybe be blacklisted... or mirrored on github so we can get versioned downloads.

[staticfloat]

All good points. I'm in favor of ensuring that only versioned downloads get cached, as opposed to caching things that could change and then hoping to detect when they do change.

[tkelman]

We could send Agner Fog an email and ask if he'd be willing to use a versioned url, or use some form of publicly viewable version control.

[staticfloat]

I can't figure out a good way to ensure that a tag is a tag and not a branch name, so for now, I think I'll just refuse things called master and develop and hope for the best.

[tkelman]

was this suggestion JuliaLang/julia#10743 (comment) not fruitful?

[staticfloat]

Oh! I just realized that was a HEAD request. That doesn't seem too ridiculous to do every time someone hits the cache; I'll go ahead and do that for github URLs. >_>

[staticfloat]

@tkelman I added a bunch of infrastructure that is cool, but ultimately useless for a very simple reason: I completely misunderstood what the etag header was giving us.

The etag header is not calculating the SHA1 of the .tar.gz; it's giving me the "git" SHA1, e.g. a commit hash. This means that, instead of just checking whether the SHA1 of the github HEAD response matches the SHA1 of the file I have stored locally, I'll have to store the gitsha alongside every github download I make. This isn't super easy to do retroactively however, since I don't store the full url of where a file comes from when I store it in the cache.

I think what I'm going to do is on every request we make out, I'm going to check for an etag header, see if it's included, and save it alongside the file. Then, every time someone makes a request for that file, I'll validate the etag. It won't retroactively apply to files we've already cached, but that's fixed easily enough. (just clear the cache. :P)

[tkelman]

Wha-oops. I think your new new plan sounds good. What will you do if the etag doesn't match? Forward along to the unmirrored url and replace the file? Or blacklist that file to not be cached in the future?

[staticfloat]

The former. That seems the most sane option, and I'm hoping that we don't run into this very often. I have a kinda-sorta log file, and I'm noting when this happens, so hopefully we can see if this really happens in the real world.

[staticfloat]

Alright, this is pretty cool. I did not know about this ETAG business, props to @vtjnash for pointing it out to me. The new caching server pings for the ETAG via a HEAD request before forwarding every time, which doesn't seem to slow it down much (~160ms measured for github.com). If it can't get the current ETAG, it just goes ahead and serves its file. If the ETAG doesn't match, it deletes the file and downloads the new one, making a note of it in the log file. I've cleared the cache, let's see how things go from here on out.

[tkelman]

👏

[staticfloat]

I'm going close this, as everything seems to be working so far. :)

[tkelman]

With a todo (for me I guess) to compose a nice friendly email to Agner Fog.

[staticfloat]

Indeed, that is one thing that should probably be done. Failing that, we could start up our own little archive of versions, but I'd rather upstream did it.