Spot database

Along with the server move last night, I have begun to make some structural changes to the spot database to address some scalability issues and pave the way to support more data mining & analytics on the site. The major change is that I am in the process of separating the spot database into two tables: a realtime "live" table which will be limited to some recent history (I plan to start with 14 days), and an archive table containing the full history. Spots will go into the live table, which will be used for the standard database query page. New spots will migrate into the archive table in batches, every 30 minutes for now.

By separating the transactional "live" data, it will be easier to add more sophisticated search features to the archive database (date ranges, wildcard matching on calls (e.g., VK*), path analysis, extraction of data for download, etc.) without locking things up for new spots and auto-refresh pages. It also makes it easier to back up the archive without locking things up for new spots.

The largest performance issue on the site has been the "old" database query page, which commonly made queries which effectively caused the whole history to be read. I have put a temporary limitation on the old database so that it will only show data from the last two weeks of spots. I'll figure out whether/how to enhance that interface moving forward, but most of the temporary slowness you see is from that issue.

Please bear with me as I try to catch up with the tremendous data collection we have all done!