To Edit Markdown

Why, exactly, do I need a markdown editor? 

Good question. Almost any code-aware text editor can color-code Markdown so you can see what's highlighted, what's italic, and so on.  But having an editor that can and will display the markdown in its final form, tell you when you've goofed up the syntax by not doing what you had in mind, that kind of thing's useful.

I've tried quite a bunch of markdown editors. So far, I have *one* winner.

iA writer for mac is lovely. It fits my workflow, by having a library manager, a spell checker, a grammar checker, even a usage checker (although it's annoying, so I leave it off unless I'm editing.) It's rock stable. 

Downside? It's Mac only*, and it's expensive (as this sort of software goes.) I'm looking for software that is similar for Linux. No steaming piles of Python need apply (now watch, that's what makes iA writer go. :P ).  Do I *really* have to write my own?

Anyway. That's where I am.  iA writer can apparently compile individual documents direct to .docx or PDF, but as it does not handle multi-documents directly, I don't think that'll be of much use. Pandoc does it better, and uses my templates.

Anyway. That's where I am.

*It's not actually mac-only. There are Windows, Android, and IOS versions as well, just no Linux version. So of my platforms, it's Mac only, because writing long work with a touch-screen keyboard makes my hands hurt just thinking about it.

My final words on Manuskript

I no longer use Manuskript. Between the usual problems of keeping a steaming pile of Python working on both my platforms of choice, and the Manuskript/QT problems where chapters *disappear* if you move them around in the chapter list, I gave up. I had to restore data too many times by using dropbox's rollback function, and once more by resorting to my NAS backup. I stopped using it for my current writing.

However, I am in the final throes of getting "Ask the Dust", formerly known as "The Silent Dust," ready to publish, so I figured I could at least use the compiler in Manuskript and add the pieces I needed to the novel to get it ready for printing.

It lost data on me again.

It lost a whole *scene* silently. I nearly went to press with the first scene of a chapter unceremoniously hacked out.

Worse, I got into a discussion amongst the developers. Apparently the guy who runs the project was rejecting pull requests (new code contributions) because they had too much documentation in the code.

Say what?

I was assured that most of the code was "Self documenting" but there was code that needed special explanation.

That would have gotten you an F in the computer science I was taught. To my ears, it translated to "I don't know what I'm doing." This made me cringe for my data safety. Am I being unfair? Has computer science perhaps moved on since 1986, when I first studied it at the college level? Undoubtedly. Still. It's a problem for *me*.

I've poked around in the code a little, mostly to try to understand what the compiler was actually doing. It's not like it's documented *anywhere* that I could see. I finally wrote a bash program to have it masquerade as pandoc and tell me what flags were really being sent. Here's what the mighty Manuskript compiler really does, albeit not in any particular order:

1. For document file in the project, push every heading tag down one level. Basically find every # tag (how markdown spells <h1>, and add an additional # to it, so the <h1> becomes an <h2> and so on..

2. Pull the chapter name from the custom metadata for each markdown document and put that in as the top level heading. (The <h1> equivilent.)

3. Add a blank line at the end of each markdown document.

4. Concatenate all the modified markdown documents into one big document.

5. Call pandoc with a bunch of flags and feed it the one big document.

It's not exactly rocket science, and it's the only piece I've not been able to replicate with other software. (Mostly vscodium (the Microsoft-free version of Visual Studio Code) with lots of plugins.) You could do it with some bash scripting, a makefile (probably, although my make-fu has become weak). I settled (because I was in a hurry) for using the Manuskript compiler to produce just the combined markdown file, fixing a bug in that file, and throwing it into Pandoc from a bash script, and this was *still* easier and more robust than letting Manuskript do it.

 Wait, I hear you say. What about Manuskript's outlining, character database, world database and so forth and so on. You can't do *that* with vscodium.

You're right. That's a job for a wiki. In fact, I use Smeagol-wiki (A tiny little rust wiki server) for exactly that. And a web browser. And a bit of Go code I put together to build the index files in my wiki automatically, because the files change a lot. You can get smeagol here: https://smeagol.dev/, and my make_index program here https://github.com/jrstrick/Make_Index-for-Smeagol-Wiki (You'll need to install Go to build it though. Working on that.)

I think this vast and disparate architecture, even without Python and QT, is the real problem underlying Manuskript. Like Scrivener, it needs to be a suite of tools rather than one big tool that does the whole thing in one (tabbed) window. I think they've got an awful lot of code to maintain, and I've read (from the developers list) that it's horrifically intertwined with QT, so separating the QT horror out and replacing it with something better—if such a thing can be said to exist. Don't get me started about Linux GUI libraries—has proven very, very difficult. I know that Python, by itself, does not make software bad. I use FreeCAD, I use Blender, and probably a dozen other things I don't even know are python every day. But I think python plus QT plus embedded Chromium plus making a program that does too many different, non-trivial things probably is.

I wanted to love Manuskript. Here was an open source replacement for Scrivener. It worked. Writing fiction in markdown has been a godsend. But I can't love software that loses my data. I can't love software that dies every time I upgrade my system and the python environment gets changed out from under it. I can't love a wordprocessor that is so resource intensive it won't run well on a Raspberry Pi 4 with 8 gigs of RAM. I can't love Manuskript as it is, and because it's in python, I can't even try to fix it.

So. No more Manuskript for me. I don't see this changing.

btw: the my current vscode/vscodium addon stack is this: LTEX-LanguageTool grammar/spell checking, by valintįn, Markdown Fiction Writer by vsc-zoctarine, Markdown Links by tchayen, Marky stats by robole, the standard pandoc markdown syntax addon, Simple Markdown Header by sguerri, Markdown Preview Mermaid Support by bierner, Markdown Table by Takumil, and Text Toolbox by carlocardella.  Markdown Fiction Writer even has a compiler, of sorts, although it's pretty awkward to use for large scale documents. You may have to download Markdown Fiction Writer separately from the microsoft addon store (it's free) if you're using codium. I'm unclear why, but that's what worked.

 

I use vscodium for writing, because I have an entirely different addon stack in Visual Studio Code, that I use for, yanno, writing code. Yes, I know about profiles, I just don't do it that way.

New Host

 Migrating the web side of things to a different hosting company. Once DNS cuts over, you'll be going there instead of Dreamhost. Please let me know if, in the next few days, you see anything weird going on with my website.

-JRS

Un-broke the website

 Dreamhost apparently changed versions of PHP on me, and broke quite a few things in my admittedly antediluvian PHP, most notably simplepie, which feeds this blog onto my home page.

I'm strongly thinking of replacing Dreamhost (for other issues. I'm sure the PHP change was a security thing.) I'm also strongly thinking of replacing my home-grown hodge-podge CMS system with something else.

What, I have no idea, but having put PHP aside for what... eight years? I don't think I like it anymore.

New novels in the works. That'd be a great time to redo the website. I'll already be reskinning it.

Anyway, we're back.

-JRS

Manuskript, Again

So I've written a novel in Manuskript. Sixty-two thousand words later, I think I'm in a position to talk about it a little more authoritatively.

The good:

Stability:

Manuskript is reasonably stable. It crashes from time to time (I had a poorly behaved font, which I think was the underlying cause) and I've yet to lose data, unlike Scrivener, although there was a close call, and I had to renumber all the scenes in The Silent Dust's metadata at one point to fix a crashing corruption of the file. At least I could.

Markdown:

Markdown turns out to be a delight to write in. Not only is it faster than type/select/shape and the styles horror of WYSIWIG editors, it's machine readable. This means that Pandoc can translate my novel into an ebook or typeset it in LaTeX right now, at least, once I get the templates set up. I can also translate the novel into a perfectly acceptable .docx file so the folks in my writing critique group (Armadillos! Yay!) can comment on it easily. Do I have to apply the changes to the novel manually? Yes.

 

Cross-Platform: 

Manuskript runs equally well on my Mac and my Linux box. It runs adequately on my Raspberry Pi 4, more on that later.

 

Familair Workflow:

If you came from Scrivener, Manuskript will be familiar to you. Folders of folders with scenes in individual documents at the deepest level. Compile the documents to their final format. 

Human-readable files

If you've used Microsoft Word, or any of the other WYSIWYG editors, you know that getting text out of a word processor format into another format can be very painful. Even Microsoft isn't consistent from their mac .docx files to the windows ones. Libreoffice does better, and Pandoc is fairly brilliant about translating from one format to another, but I'm tired of dealing with silly proprietary formats that hide all the formatting and make such a mess of it. This started in the 1970s with Wordstar, and it's been downhill ever since.

Open Source:

At the very least, you should always be able to grab a copy of Manuskript and open your files. There's no company that decides to turn your favorite word processor into a 'software service', or simply go out of business. I've had both. Done with that.

 

Best in Breed:

For what it does, Manuskript is the best software out there. It's as good as Scrivener, arguably more stable than Scrivener has been historically, and it runs on all my platforms. And it's free.

---

The good ideas, poorly implimented:

Side Data: Manuskript has ways to store character sketches, world information, timelines, all kinds of stuff like that. Unfortunately, each one is stored in a different format, and there's no way to make a global set for a series of books. It desperately needs a database behind it, and it hasn't got one. 

 

The editor:

Text editors are hard. I get it. But for reasons unknown, the Manuskript editor will, after a while, lose its ability to select a point in the text. Inserts are done at the end of the file, and cut and paste doesn't work at all. This is maddening.

 

Search/Replace: The search function works. It's a new addition, and it's ok. I'd like to be able to select between local (this document) and project-wide, but I can't. There is no replace. The easiest way to do search and replace in Manuskript is to save the file to the un-zipped version of the file format, and open the directory that results in MS Visual Studio Code, which has a brilliant global search and replace. I fear for the metadata every time I do that, however.

 

The Compiler:

Great idea. Gather the files and feed them to Pandoc. But there's no way to feed only one scene to Pandoc, nor can you easily pass flags to Pandoc. If the file format you want to compile to isn't already in the compiler's menu, you're on your own, more or less.

 ---

The Bad.

File Format: 

Oh God, the file format. Hidden away in the neat .zip file is a dog's breakfast of markdown files, JSON, YAML, XML, and heaven only knows what else. Metadata is scattered all over the place. You can get at everything, and none of it will make a basic text editor choke, but it's a mess to try and piece things back together if something breaks. Been there, done that.

 

Metadata:

The metadata system is, to be charitable, not Manuskript's strongest suit. Even the order of chapters or scenes in chapters is stored in metadata, and if that one number gets corrupted, two documents can wind up with the same order number. If they do, one or both of them will disappear.

 

Renderer: 

If you look at how Manuskript works, it's really a web app running through a QT webview. While I get it, this makes it  whole lot easier to render markdown as rich text/html, and it handles all the really messy issues of text shaping, layout, font handling, and so on, it also means that you have a titanic binary blob that is almost a whole web browser. This murders performance on Raspberry Pis, and the damn thing isn't stable. One misbehaving font should generate errors, maybe even make a document unreadable. It should not crash the entire application every time you change the margins. News flash, it does. 

Versioning:

Manuskript has a versioning system. They recommend right in the instructions that you not turn it on. So I haven't.

Python:

I've been fairly vocal in my disdain for Python over the years. Yes, part of this is ignorance—I don't know Python at all. Part of it is that the idea of whitespace in a program having programmatic meaning makes me itch. Yes, I know there are special editors that take care of that for you. My point is it shouldn't exist. The big problems with Python in this case are 1. It's slow. Manuskript is somewhat marginal to use on a Raspberry Pi 4 with 8GiB of memory. This is embarrassing. Python is also very much a second-class citizen on the mac, where Apple dutifully breaks the python environment routinely. Just getting Manuskript running on the mac sometimes takes me hours. I see in the pull requests on the Manuskript github repo that they have someone doing a proper MacOS build with a proper bundle and everything, but it isn't in the distributions yet.

Best in Breed:

There are so many good ideas in Manuskript, and so many markdown editors out there are nicer and more stable but are missing functionality I use every day that it's very frustrating.

---

Conclusion:

I'm starting another novel in Manuskript. It's good enough. It has its foibles, as I've tried to make plain above, but as I've also said, it's the best in breed for novel writing, as far as I'm concerned. Does it frustrate me terribly at times? Yes, yes it does. 

 

Am I dusting off my programming skills to try and write something better? Yesss, but nobody should hold their breath on that ever being finished. I have a language, a gui library, and a database picked. That's it so far. 

Mostly what I've accomplished so far is to become much more in tune with how big a project this is, and how hard some parts of it are. Also that I haven't seriously studied code in over 30 years, and GUIs are a pain in the ass to write code for. I've picked on Manuskript a lot in the text above, and I think I'm being fair. That said, I used it anyway, and will continue using it anyway for the foreseeable future.

The Ethnicity of Characters, and The New Novels

For a long time, I've been reluctant to depict people too far from my own cultural/ethnic stock in my novels, for fear of getting the details wrong, and making essentially a 'blackface' character—an unintentionally (in my case) racist parody. I really try not to offend people accidentally with my work, and making a character like that would be a humiliating failure on my part.

An online friend of mine (Dale, lookin' at you here,) who is Black, set me straight. He said he would much rather see someone try and not necessarily get the 'details' right than not see characters who looked like himself at all. The implication was that the 'details' that worry me so much assume a monolithic culture of a given minority that isn't really there.

It's taken me a while to digest this.

But I'm trying.  The protagonist of the novel I just finished, The Silent Dust, and indeed the whole series, is Nina Cohen. She's Jewish, born in Romania a very long time ago, and her family came to the United States when she was 14. It's part of the plot, part of the character, so I picked her background for her. As the series is set in a fictional city in Minnesota, however, the question of 'what do all the people around Nina look like?' became critical. In The Silent Dust, I tried to keep an even hand. A lot of the characters came out White. I don't think that's necessarily a problem, given the demographics of Minnesota, but darn it, I want to be fair. I keep thinking about what Dale said.

For Dead of Winter, I've a new method. I took the demographics for St. Paul, Minnesota, and made a chart. For any character I don't have a good reason to make them any given ethnicity, I get out my much-worn D&D percentile dice, and roll against that chart. Now, my book will reflect reality. There's a fifty-one percent chance any given character not already assigned an ethnicity will be White. There's a seven percent chance they'll be mixed-race. Culturally, all the characters I know about so far except Nina herself are Minnesotans, the way I remember them from when I lived there. One of the main characters is Black. One is of mixed ethnicity. Is it a challenge for me? Yes it is. But hey. Without challenges, one does not grow, right?

 

 Does this mean I'm writing again? It does. 

The Silent Dust is complete. I'm in the process of my final edit. The Dead of Winter is started. They're paranormal detective stories. Above all, the stories are small. Nobody's saving the world, particularly, it's just a poltergeist in an abandoned human body trying to make a living using her abilities to solve difficult cases. There's a lot of humor involved, but I'm trying to write well constructed mysteries, and "because it's funny" is not a good solution to a mystery, in my opinion. The humor is part of the telling of the story. It doesn't control it.

 

What about the other projects?

Right now, I'm kind of done with steampunk. I came in late, and it seems like the genre is dying out. Also, the other steampunk stories I've read really haven't been to my taste, so it's kind of hard to want to chase that audience further. Brass and Steel: Inferno's sales were disappointing, to say the least, so there's that, too, and that book took me five years to write. I don't want any more five year missions. that result in epic 'save the world' plots. I lose touch with the characters over that kind of time, and good sequels are practically impossible.

What about cyberpunk?

I still love cyberpunk. I still love reading it, it still fires my imagination, and it's very likely I'll revisit cyberpunk at some point. The LookingGlass world is probably a dead end at this point. We now live in the timeline I was talking about when I wrote it.

 

That New Cat Smell

 Meet Puma Purrman.

Sunny, our other cat, was not doing well as an only cat, in the wake of Shadow's death, so on Saturday, we went to the Castle Rock Dumb Friends facility to look at cats. Puma (then named Uma Purrman) was first on her list, and the volunteer said "Oh, perfect. She's a doll, I've been trying to find a home for her all day.

And she was. 

Affectionate, inquisitive, practically fearless (except for the dogs when they were looking in the window), with impeccable manners (soft paws, no biting). Also sleek and black, as I've come to prefer in cats. So we brought her home. She's about four months old, with great big paws and a very long tail. IT seems likely she's going to be a big kitty.

It's been a very long time since we had a kitten. 13 years or so, not coincidentally the length of Shadow's life, since he was the last kitten we had. Still getting used to that. Next steps, get the new cat inspection done, get her spay stitches out, and get her fully introduced to Sunny, who is very curious about her.

As I told her (like she listened. Cat.) I'm glad she's here. I wish Shadow could be here too, but he can't.

What, you don't talk to your cats?