(OS=Windows) I was having issues with power failures while processing very large files. If the power went out, then my log file would be all zeros (NUL NUL NUL...). So, I wrote a couple lines to save the progress every 50K lines, so that in the event of a power failure I can then restart from the last successful line.

toggle = toggle ^ 1
fs.writeFileSync(`progress${toggle}.txt`, progress)

Toggle alternates between 1 and 0. This way, if the power happens to go out while progress1.txt is being saved, I will still have progress0.txt. And vice-versa. I tested this and it works correctly. It toggles between the two files and saves progress. Then, after the power went out, I checked and BOTH progress files were all NULs. Not just one script, but all the scripts had their progress files overwritten with NULs.

According to my research this happens when a computer is suddenly shut off while writing to a file. How is it possible that both files could be open at the same time? In order for progress1 to be open, writeFileSync would have to save and close progress0 and then process another 50K lines since everything is being done sync style. There should be no way for both to open at the same time.

Thank you.

EDIT I was asked if I can show how writeFileSync is being called. Here is the basic program structure:

var module = {
  finish_cb: null,
  toggle: 0,
  progress: 0,
  start_batch: function () {
    this.toggle = this.toggle ^ 1
    fs.writeFileSync(`progress${toggle}.txt`, this.progress)
  },
  process_batch: function () {
    while (STILL PROCESSING) {
      // DO BATCH THINGS
      // THIS WILL TAKE 10-20 seconds to complete
    }
    this.progress++
    this.finish_cb()
  }
}

module.finish_cb = finish
module.start_batch()
module.process_batch()

function finish () {
  if (ALL DONE) {
    console.log('All done!')
  } else {
    module.start_batch()
    module.process_batch()
  }
}