Fixes: Stability, Estimates, Source Reports

The site has been a little unstable over the past few days — if you’ve been here you’ve probably noticed the yellow banner at the bottom of the page. The issues were triggered by series of large data imports, combined with some increased traffic.

As I write this, the total number of log entries in the database is 396,795,265, documenting 219,698,959 searches, with tens of thousands of visitors per day poking around all of it. Those are big numbers for a little $15/mo virtual server.

Many of the reports and statistics are not calculated in real time — most are updated once per day. That’s why you’ll sometimes see the notice that newly imported data is still being processed and results may be inaccurate or incomplete. But processing all that data (especially the statistical info) takes some doing.

So, after some rejiggering of things and whatnots, the reports are now (internally) broken up into smaller batches so that the server doesn’t catch on fire trying to read 200–400M records all at once.

Some statistics — specifically, some “number of unique X” counts — have been replaced with estimates, using HyperLogLog. This should give accurate estimates within 2%. Where estimates are now used this is noted through tooltips, language like “approximately” or indicators like “~”.

If reports fail to refresh properly in the next few days, some further adjustments may be needed, but those should be relatively minor and have less impact on overall site stability.^[1]

#Source reports

The source reports (example), which aggregate multiple other reports did not work correctly. Queries would timeout, and pagination was broken, limiting results to a maximum of 200. Both issues should be fixed.