Scrivener, Word, and Tracked Changes

So it's not a very closely guarded secret that I'm working on getting Looking Glass and Irreconcilable Differences back in print. (Soon. Soon.) My modern toolchain for these old books (for those keeping score) is: Nisus Writer -> RTF -> Scrivener -> Epub -> Jutoh -> Epub, Mobi, and Epub for ibooks. For the hardcopy, it's Nisus Writer -> RTF -> Scrivener -> RTF -> rtf2latex->Tex->XeLaTex -> PDF. This sounds like more work than it really is. I have the RTF manuscripts, the same ones the Flying Pen Press typeset was done from. I imported them both into Jutoh years ago when I stopped making ebooks by hand. I imported them into Scrivener for all the modern maintenance, mostly cleaning up weird typos that seemed to crop up.

Fast forward to a few days ago. While finishing up the ebook version of Looking Glass, I decided to go through it one more time with the Jutoh spell checker. (Jutoh's spell checker is awful. Slow, primitive, based on ispell or aspell (I forget which), and Jutoh likes to hide the cursor permanently when you've been spell checking a while. It also crashes occasionally.) And I turned up dozens of errors. Looking closely at them, some of them I /knew/ I changed back in the early days of 2007, when we were editing the book. Something was horribly wrong. Looking more closely, these were present in the PDF as well. Walking back through the toolchain, I discovered them in the Scrivener file. When I reimported the original manuscript into Scrivener, I got the same results, plus a lot of "typos" I'd chased through the ebook version over the years. 


My first thought was "Oh crap. Is the file corrupted? Is the import process broken? Is this somehow in the original manuscript?" So I went back to the original manuscript, last edited when I cleaned it up prior to hand-making the first ebook version. Some time in '08, I think, and opened it up in Nisus Writer, just like the old days. When I searched on one of the errors, up it popped - in tracked changes. Every one of those glitches was a change Scott or I had made with Word tracked changes. Every stinking one. About 400 of them in all. Those changes were invisible on the body of the text in Nisus, and in Word too. Somehow, during the import to Scrivener, they were being re-exposed. The book was, essentially, being un-edited.

I sat down to file a bug report. On a whim, because I know that Scrivener leans heavily on OS X's RTF libraries, I tried importing the file into textedit, Apple's default text and RTF editor in OS X. Same result. Same stinking result.

The upshot? OS X's RTF libraries do not do anything sane or intelligent when presented with tracked changes. They concatenate the original and the changed text, and send it on up the pipeline. I do not now recall how I did the original port to Jutoh, but I very much suspect, given the number of people who've complained about the typos in the ebooks over the years, that it used the same RTF library somewhere along the line.

The solution? Fairly straightforward. Open the original RTF up in Word, make a copy, and tell word to Accept All Changes, and save it to an RTF. When I reimported that RTF, it was clean. Nisus Writer doesn't seem to have that capability (or I couldn't find it.) It was fairly well hidden in Word.

So, moving forward, I remade both the ebook versions and the LaTeX typeset version yesterday and today. It gave me an opportunity to clean up some LaTeX code and some of the sloppiness in the way I'd set it up in Jutoh. And I finally solved the mystery of why my later ebooks were such a mess. If you're reading this, and you have one of those messed up ebooks, email me and I'll hook you up with the new version. Meantime... watch this space. I hope to have the thing for sale fairly soon.

-JRS