[NO ISSUE][STO] Tolerate Corrupted System Checkpoint Files - user model changes: no - storage format changes: no - interface changes: no
Details: - Do not delete corrupted checkpoint files until a valid checkpoint is persisted. This ensure a forged checkpoint that will force recovery to start from the begining is always used until a valid checkpoint is found. - Attempt to read the latest checkpoint file right after writing it and before attempting cleaning up invalid and old checkpoint files. - Use on disk files to determine next checkpoint id to account for existing corrupted checkpoints. - Maintain two older checkpoint files in addition to the latest one. - Catch all exceptions on checkpointing failures since it is a try operation. - Add test scenairo for the new checkpoints clean up behavior.