Data scrubbing

You may recall, I partially implemented data scrubbing on incoming data in April of this year. It looks for bogus calls, duplicates (including mixing product harmonics...receiving the same signal at multiple freqencies in the passband), etc. Because of some controversy, I never did roll out the "wrong band" detection code, because of some concerns it might be overly aggressive. Anyway, the implementation deleted the bad spots by moving them into a separate table. I have about 250k of them.

I made a decision to handle this differently in the future. I've added a field to each spot which allows the scrubbing program to record what it thinks is wrong with the spot, but it won't delete it from the main database. Instead, it will (eventually) mark suspicious spots and be able to flag them as such in the database display. That will allow me to turn on the "wrong band" detection code, which might have some false positives, but it won't prevent the data from being found. I'll restore the previously set-aside data as well.

The only things which I plan to truly delete are pure duplicates (identical report from same time)...most of those are rejected when they are posted, but sometimes manual uploads of old data can put int dupes.