`kolide_program_icons*` performance issues

[RebeccaMahany]

We've noticed that the kolide_program_icons table frequently uses a large amount of memory, and often CPU. Currently, for cpuprofiles in performance flares generated thus far by the v1.28.4 launcher release, 3 of the top 5 functions within launcher point to this area of the codebase:

github.com/kolide/launcher/pkg/osquery/table.generateUninstallerProgramIcons		found in 533 samples across 7078 cpuprofiles		5179 total seconds
github.com/kolide/launcher/pkg/osquery/table.parseIcoFile		found in 534 samples across 7078 cpuprofiles		6274 total seconds
github.com/kolide/launcher/pkg/osquery/table.generateProgramIconChecksums		found in 534 samples across 7078 cpuprofiles		6806 total seconds

We attempted to solve at least the memory usage issue in #2401 by adding the kolide_program_icon_checksums table -- however, switching to using the checksums table had no discernible effect.

Next, we can try confirming that the package we use for ico parsing is our best option. We currently use https://pkg.go.dev/github.com/mat/besticon/ico for our icon parsing. We should try benchmarking it against the following options:

• https://pkg.go.dev/github.com/sergeymakinen/go-ico
• https://pkg.go.dev/github.com/thatoddmailbox/go-ico
• https://pkg.go.dev/github.com/ur65/go-ico
• Any others you may find!

Completing this issue may have the side effect of solving #2271 as well.

[RebeccaMahany]

$ go test -bench=. -count=10 -run=^$ -benchmem -v
goos: windows
goarch: amd64
pkg: github.com/kolide/launcher/pkg/osquery/table
cpu: 13th Gen Intel(R) Core(TM) i7-13800H
Benchmark_parseIcoFileWithBesticon
Benchmark_parseIcoFileWithBesticon-20                          7         152115314 ns/op        1376952713 B/op     2659 allocs/op
Benchmark_parseIcoFileWithBesticon-20                          8         137973388 ns/op        1376935718 B/op     2624 allocs/op
Benchmark_parseIcoFileWithBesticon-20                          8         137077462 ns/op        1376933796 B/op     2622 allocs/op
Benchmark_parseIcoFileWithBesticon-20                          8         136347888 ns/op        1376934050 B/op     2621 allocs/op
Benchmark_parseIcoFileWithBesticon-20                          7         145814529 ns/op        1376936156 B/op     2625 allocs/op
Benchmark_parseIcoFileWithBesticon-20                          7         145919157 ns/op        1376936262 B/op     2627 allocs/op
Benchmark_parseIcoFileWithBesticon-20                          8         138693550 ns/op        1376935088 B/op     2623 allocs/op
Benchmark_parseIcoFileWithBesticon-20                          7         144343443 ns/op        1376934982 B/op     2624 allocs/op
Benchmark_parseIcoFileWithBesticon-20                          8         139612725 ns/op        1376934066 B/op     2620 allocs/op
Benchmark_parseIcoFileWithBesticon-20                          7         145222500 ns/op        1376934316 B/op     2620 allocs/op
Benchmark_parseIcoFileWithSergeyMakinenGoIco
Benchmark_parseIcoFileWithSergeyMakinenGoIco-20               90          12867620 ns/op         7006634 B/op       2540 allocs/op
Benchmark_parseIcoFileWithSergeyMakinenGoIco-20               98          12846410 ns/op         7004726 B/op       2538 allocs/op
Benchmark_parseIcoFileWithSergeyMakinenGoIco-20               96          13066471 ns/op         7005120 B/op       2539 allocs/op
Benchmark_parseIcoFileWithSergeyMakinenGoIco-20               93          12618229 ns/op         7005562 B/op       2540 allocs/op
Benchmark_parseIcoFileWithSergeyMakinenGoIco-20               96          12882919 ns/op         7004718 B/op       2538 allocs/op
Benchmark_parseIcoFileWithSergeyMakinenGoIco-20               97          12853423 ns/op         7004789 B/op       2538 allocs/op
Benchmark_parseIcoFileWithSergeyMakinenGoIco-20               94          13325805 ns/op         7004015 B/op       2538 allocs/op
Benchmark_parseIcoFileWithSergeyMakinenGoIco-20               91          12966491 ns/op         7006282 B/op       2540 allocs/op
Benchmark_parseIcoFileWithSergeyMakinenGoIco-20               98          12863258 ns/op         7005063 B/op       2538 allocs/op
Benchmark_parseIcoFileWithSergeyMakinenGoIco-20               96          12971542 ns/op         7003652 B/op       2537 allocs/op
Benchmark_parseIcoFileWithThatoddmailboxGoIco
Benchmark_parseIcoFileWithThatoddmailboxGoIco-20               7         158752329 ns/op        1381579393 B/op   447477 allocs/op
Benchmark_parseIcoFileWithThatoddmailboxGoIco-20               8         142872688 ns/op        1381579355 B/op   447476 allocs/op
Benchmark_parseIcoFileWithThatoddmailboxGoIco-20               7         145929786 ns/op        1381580186 B/op   447478 allocs/op
Benchmark_parseIcoFileWithThatoddmailboxGoIco-20               7         145175814 ns/op        1381579773 B/op   447477 allocs/op
Benchmark_parseIcoFileWithThatoddmailboxGoIco-20               7         143758243 ns/op        1381579185 B/op   447475 allocs/op
Benchmark_parseIcoFileWithThatoddmailboxGoIco-20               8         142654038 ns/op        1381579712 B/op   447477 allocs/op
Benchmark_parseIcoFileWithThatoddmailboxGoIco-20               7         145180943 ns/op        1381582836 B/op   447479 allocs/op
Benchmark_parseIcoFileWithThatoddmailboxGoIco-20               7         147299629 ns/op        1381579689 B/op   447477 allocs/op
Benchmark_parseIcoFileWithThatoddmailboxGoIco-20               8         141315962 ns/op        1381579133 B/op   447474 allocs/op
Benchmark_parseIcoFileWithThatoddmailboxGoIco-20               8         142225075 ns/op        1381578828 B/op   447474 allocs/op
PASS
ok      github.com/kolide/launcher/pkg/osquery/table    34.456s

[RebeccaMahany]

https://pkg.go.dev/github.com/sergeymakinen/go-ico has significantly better performance than our current option -- but out of my 5 test ICOs that I expect to be parsed correctly, it only parses 4 of them. (It's returning data for the 5th, but the cloud is unable to display it, so something's not quite right there.)

I'd like to see if #2491 helps first.

[RebeccaMahany]

Looking at trace data, we've definitely seen a performance improvement for speed for this table:

version	number of spans	p50	p90	p95
1.29.2	4,184,979	0s	10.741ms	32.335ms
1.28.4	88,195	0s	408.522ms	702.958ms

[directionless]

That's a great perf boost!

I also notice a big jump in spans. Do we need them all?

[RebeccaMahany]

@directionless the difference in number of spans is because traces are only stored for 30 days, and we released 1.29.2 to stable on Dec 8 -- so we don't have a lot of devices on 1.28.4 reporting up these spans in the last month. (I probably should've looked at this span data in mid-December to get a more even split, but ah well!)

[directionless]

I think we can close this? Feel free to reopen if you think we should