Today is a brute-force day. I needed to col­lect second-by-second data from a data­base over a 4-hour period. There were 2 queries that needed to be run, each just gen­er­at­ing a sin­gle num­ber for each sec­ond (count­ing events/sec). If I knew any­thing about SQL these could prob­a­bly be done in exactly 2 queries. I don't know any use­ful SQL so I ran indi­vid­ual SQL queries for every sin­gle sec­ond. That's 28,800 total queries.

I didn't hand-write them, of course. I first wrote a python script to gen­er­ate the queries and craft a sim­ple SQL script. After gen­er­at­ing my two SQL scripts (each 3 MB) and run­ning them, I had 2 data files each con­tain­ing 14,400 data points. Then I used my text edi­tor, Edit­Pad­Pro, to trim the white­space and con­dense to a sin­gle col­umn of num­bers. Then I pasted into Excel.

Then I made a line chart from the two num­ber sets which almost brought Excel to its knees, but it worked! It was cool but there was no way to zoom in on the smaller more inter­est­ing sec­tions, plus the thing took sev­eral sec­onds to redraw after any scrolling. I ended up cre­at­ing sep­a­rate charts for each hour so the graphs ren­der much faster. Now I am cal­cu­lat­ing some inter­est­ing max, mean, and mode num­bers to reduce it all to use­ful sum­mary numbers.

The last few hours were there­fore a major tri­umph of brute force effort: writ­ing a pro­gram to gen­er­ate two other pro­grams (the SQL queries) to gen­er­ate the 28,800 data points. All because I only know very basic SQL queries.

It was a blast!

BONUS — my gen­er­a­tor pro­gram under­stands nat­ural lan­guage, so when I ran it I did 'elog_count.py today 01:00:00 to today 05:00:00 by sec­ond for "Inform­Pe­ri­odic" '. Then once again for 'Infor­m­Val­ueChanged' for the other query.