The brother has a problem today, his dispensing system kept keeling over when he dispensed to some people, but not to others.
Firing up the utility which is what all normal pharmacy people would have had access to would only show that the system failed to validate a table, and that the connection to the database was broken. The only clue I had was the table name that I last saw in the utility. I’ll just say from the get-go that this is Not good diagnostic programming practice.
Firing up the SQLPlus worksheet, I ran the ANALYZE TABLE <foo> VALIDATE command over the table, and saw no error. I ran it with the CASCADE, option and the same thing happened as the utility – the database fell over.
It turns out that one of the indexes was corrupt on the table. Iterating over all the indexes using ALTER INDEX <foo> rebuild, which narrowed down the indexes. Then I extracted the DDL for the index, dropped and recreated them, re-validated the table and everything was hunky dory.
Full documentation on the ANALYZE command is in the oracle documentation. For the purposes of this exercise we used ANALYZE TABLE BLOB VALIDATE CASCADE to see the error.
Solving the problem took about an hour, most of which was spent chewing on a sandwich and booking tickets to see Star Wars, and of course cardboard programming by another friend. A backup of the database would not have helped in this case as the corruption occurred silently some time ago, and was only uncovered when certain customer’s records were accessed, so the loss of all the data input from the point of corruption to then would have been unacceptable.