WIP: port to work on Ska3

[taldcroft]

Description

This is a WIP to port basic timelines functionality to work in Ska3. Some points of note:

• Support for sybase is dropped and the database DBI is hardcoded as sqlite.
• Most of the diffs are whitespace, which can be ignored in the diff settings.
  NOTE: cmd_states H5FILE still hanging around? xija#114

Testing

So far:

cp $SKA/data/cmd_states/cmd_states.db3 ./
cp -p $SKA/data/timelines/sum_files_sqlite3.touch ./
./parse_cmd_load_gen.pl --server ./cmd_states.db3 --touch_file ./sum_files_sqlite3.touch --verbose
./update_load_seg_db.py --server ./cmd_states.db3 --verbose

The first time I ran this it did a lot more processing than I expected given that the production version basically in a static situation (no loads changed/updated recently).

ska3-kady$ ./update_load_seg_db.py --server ./cmd_states.db3 --verbose
Connecting to sqlite server ./cmd_states.db3
LOAD_SEG DEBUG: Updating from /proj/sot/ska3/flight/data/arc/iFOT_events/load_segment/2021:299:10:31:00.000.rdb
Running: ('SELECT max(id) AS max_id FROM timelines',)
LOAD_SEG INFO: Updating Loads for range 2021:279:02:52:54.460 to 2021:305:01:39:10.961
Running: ('SELECT max(id) AS max_id FROM load_segments',)
Running: ("select * from load_segments\n                               where datestart >= '2021:279:02:52:54.460'\n                               order by datestart, load_scs",)
LOAD_SEG INFO: Mismatch on these entries:
('CL294:0801', 2021, '2021:294:08:44:14.702', '2021:296:10:41:57.000', 128, 0)
(426104749, 'CL294:0801', 2021, '2021:294:08:44:14.702', '2021:297:14:00:51.886', 128, 0)
Running: ("select * from load_segments where datestart >= '2021:305:01:39:10.961'",)
Running: ('SELECT * from timelines where load_segment_id = 426104749',)
TIMELINES DEBUG:  DELETE from cmd_fltpars
                   WHERE timeline_id = 426104880 
Running: (' DELETE from cmd_fltpars\n                   WHERE timeline_id = 426104880 ',)
TIMELINES DEBUG:  DELETE from cmd_intpars
                   WHERE timeline_id = 426104880 
Running: (' DELETE from cmd_intpars\n                   WHERE timeline_id = 426104880 ',)
TIMELINES DEBUG:  DELETE from cmds
                   WHERE timeline_id = 426104880 
Running: (' DELETE from cmds\n                   WHERE timeline_id = 426104880 ',)
TIMELINES DEBUG: DELETE FROM timelines
              WHERE id = 426104880
              AND fixed_by_hand = 0 
Running: ('DELETE FROM timelines\n              WHERE id = 426104880\n              AND fixed_by_hand = 0 ',)
...
A bunch more like that.

[jeanconn]

So, do we want to retire the "states" part of cmd_states universally? I suggested "let's retire sybase first", but maybe if we are there they should just all go?

If we want to keep the 'states' product h5 file we'd need to move that updating to the sqlite job. Right now it is a "side-effect" of the sybase job.

[taldcroft]

This is a fine time to drop the 'states' product. It has been deprecated for what feels like years.

[jeanconn]

Sounds good. For that I think we'd want to reach out to MTA too? Not sure if there might have been any other customers... and we'll need to do a bit more of a survey with some updates if there are some pieces that aren't converted to kadi command states.

[taldcroft]

See email "Package deprecations: Chandra.cmd_states and Ska.ParseCM" from June 10 2020. I can re-send that same email if you are worried.

[jeanconn]

Given resources during these past 20 months it would probably be a good idea to re-send.

[taldcroft]

Done (re-send)

[jeanconn]

Thanks! To enforce "deprecation", I was thinking rename the sybase tables and remove cmd_states.h5. Wasn't sure if we also wanted to strip out the cmd_states table from the cmd_states.db3.

[taldcroft]

Yes on all three (in a week).

[taldcroft]

Timelines will be dropped.