A Practical application of markdown & pandoc

As I recounted in another post to this blog, I started using the text-only markdown format for my writing a few months ago. In part I did this to avoid future problems with file formats, and in part to be able to have one central file that I could convert to various formats as needed, ideally by using pandoc.
Most people probably will understand the problem with obsolete file formats. I’ve got some not so old WordPerfect files on my computer right now, for example (and you can find a bunch of them on my employer’s servers as well), but no WordPerfect app (or “program” as we used to call it). I can still read the ones I’ve tried to open, but I need to be a bit clever about it and I doubt many of my colleagues would succeed on their own. I also had a heck of a time converting my 1998 dissertation to a newer version of Word, losing a little formatting along the way.
But the major impetus for the effort in recent months has been the second goal of having one master file in markdown (mentioned very briefly in that other post). In particular I was thinking about streamlining the workflow for updating my CV, which I do as often as every few months. I keep a version of the CV on line as a web page, and another version in PDF for me to share more formally and for the curious to download from the web page. I wasn’t too thrilled with how the html printed, so I didn’t spend much time cleaning it up for that purpose and instead generated the PDF from an OpenDoc file which is easy to format (and which I edit with LibreOffice, BTW). My workflow then was something like:

1. Edit the html with BBEdit.
2. Edit the odt file with exactly the same content changes.
3. Print the odt file to PDF.
4. Post the html on line.
5. Post the PDF on line.
6. Archive the successive .odt and PDF versions by saving the new files with an appropriate name (I stuck the date in it).

In short, not a super long workflow, but still plenty of opportunity for those two versions to go out of sync (which they did). More annoyingly though, it was just too many steps.
When I learned about pandoc, I figured this was going to do it for me. Instead of worrying about odt and html, I’d just have one markdown file and spin that off into html and a PDF, avoiding odt altogether. That turned out to be harder than I’d expected, mainly because I knew virtually nothing about the tool pandoc uses to generate PDF, TeX/laTeX, and the default format didn’t look much like what I was used to generating. (My current default document format is based on the ideas in Matthew Butterick’s Practical Typography, which I highly recommend.) I also had the problem that the html wasn’t really printable in the format I was using: the links were underlined so visitors would know where to click, and it contained some info at the bottom including links that led to my department’s website and a few other places, all of which did not need to be included in the PDF.
At the heart of the problem is that markdown is a nice little tool, but it isn’t designed for formatting the text. That’s a pretty standard text-markup thing: put the document’s logical structure in it, but do the formatting elsewhere. Content not layout. So while you can say “this bit of text is a heading”, you can’t say what a heading should look like from within markdown. For that you need to use whatever formatting system your destination file requires (css for html, stylesheets for odt, and so on) and that’s where my ignorance of latex came into play. After a little bit of trying to get up to speed on laTeX, I decided to drop the idea of going directly to PDF and continue as I had been: print a file to PDF after it had been generated by pandoc. This was still an improvement over what I had been doing…as long as I could get the format right so I could use the html file for printing.
I put the project aside for a bit, but when I had to update the CV a few weeks ago, a few things occurred to me. First, I wasn’t using css to full advantage. I’d forgotten that it allows you to change styles depending on how the page is viewed. For example, the css can indicate one font for a big computer monitor and another for a smart phone. So I used that feature (specifically @media print coupled with the css pseudo-class last-of-type) to make that pesky final material disappear upon printing. I also made the links look like the rest of the text when printed. The other thing I realized was that my two-column format could easily be handled with standard html tables which I had abandoned years ago in an earlier version of the document and was now fudging in the markdown by using standard html…ugly. The last thing to do was use another css pseudo-class (first-of-type) to get my leading table to format differently from the other tables throughout the document. Here's what that last looks like. The contact details at the start are one table (the first in the document) and the educational institutions with dates are a second:

The last table is similar and disappears upon printing:

Where am I now with that workflow? Here’s the new one:

1. Edit the markdown.
2. Convert the markdown to html
3. Print the html file to PDF.
4. Post the html on line.
5. Post the PDF on line.
6. Archive the markdown and html on Github.

Strictly speaking the PDF in step 3 is only for me now, since viewers of the web page version could just print it themselves and get the right formatting. Nevertheless I post the PDF for downloading to make it easy. (Now that I think of it, I should probably keep track of downloads to see if that’s worthwhile.) Step 6 was my solution to all the archiving work I was doing. It still requires some action on my part, but with the OS X Github Desktop app it’s much easier and quicker than the old method. In fact although overall the number of steps is still the same, they entail a lot less work. Importantly the repeated and error-prone editing step is gone.

Lessons learned?

• Pandoc is a great tool, but getting it to generate (nearly) identical-looking documents in different formats can be fairly hard. To be fair, it’s not really for that anyway.
• If you don’t want to settle for default styles, you’re going to have to do a little work in your output format to get your document to look like you want. You may also have to compromise to get it to work in markdown.
• It’s not always easy to step back and look at a problem with fresh eyes. I was so intent on rendering the original html in markdown that I didn’t realize that an easy(-ish) css solution was staring me in the face in the form of new html. Setting the project down for a while helped.

A final note: I continue to work (slowly) on my laTex knowledge, and have now got a generic document style going that looks fairly similar to my odt and html.
PS I don’t know why I’m capitalizing PDF, but not the other extensions.

List of undergrad Classics programs - more pandoc & github

I've kept ("maintained" would be too strong a word) a list of undergraduate classics program for a while now. I first started it when I was the webmaster for the American Classical League back in grad school and early in my career, but then I just kept it around because it seemed like a shame not too.

The other day I got a now rare update from someone on the list, and I thought this would be a good time to change the way I handle the whole thing.

First I figured I'd switch over the basic form of this very simple page from html to the easier to manage markdown, and then change my workflow to use pandoc to generate the html from this markdown whenever necessary. This part was fairly simple, though there were a few complications. For example, pandoc doesn't like html name tags, so in order to keep a bunch of internal links I had, I needed to convert those anchors to spans with IDs that matched the names I was using. BBEdit and its nice search-and-replace functionality to the rescue. I then did some minimal tweaks to the new pandoc-generated markdown file, so that pandoc could now generate some decent html from it.

Step 2 was to put the new markdown and html up on GitHub, instead of in my institutional filespace. Not only does this give me some version control over the files, but I can let other people edit the markdown and send me pull requests, instead of emailing me updates that I then put into the file. I don't think I'll get a lot of these, but you never know.

So I can now (1) make changes to the markdown myself, or accept pull requests with them in it, (2) sync my local github repo with the on-line version, (3) run pandoc on my local copy, and (4) sync it back to GitHub where I release a new tagged version. I then link to the files directly via rawgit. (The tag in GitHub is needed to make rawgit grab the correct version instead of the one it caches.)

The relevant links are here:

• GitHub repository
• The list itself

Check out the list and let me know about any updates via a pull request!

Self-publishing with pandoc, etc

Depending on your thinking, I'm either just done with or approaching the end of a sabbatical. ("Just done with" if you think that once commencement occurs, it's just a regular summer.) Among the things I produced in the past few months is a very short "note" on a topic that doesn't fall within my usual area of research. I sent it to a couple of OA journals, but neither wanted to publish it as is. I'm not interested in doing more with it at this time, but it seems silly to have it just sit on my hard drive doing nothing. It's the kind of thing I'd do as a conference paper, if I went to a conference at which I think it'd be welcome. But since I don't go to such conferences, I figure I'll just put it out there for people to check out anyway. (The advantages of tenure and the internet!)

I could do it as a blog post, though it's already written in a more "academic" style than I write this blog in. Instead I'm going to post it as html on github and as a pdf on my account at figshare, where it's easily accessible, archived, and even gets a DOI. I'll also link to it from my academia.edu page (as well as here, obviously).

The Workflow

I've started using markdown with pandoc to generate documents. I was inspired by Dennis Tenen and Grant Wythoff's post last year, "Sustainable Authorship in Plain Text using Pandoc and Markdown," but I've long been a fan of avoiding proprietary formats that are likely to become obsolete (no doubt in part because I work with very old texts and materials professionally). It's easy enough to do simple stuff this way, but getting to more complex documents requires some work. Here's a list of stuff I do/use:

• For editing my markdown documents, I use the free MacDown, which gives a nice split screen, showing the raw markdown on the left and the interpreted version on the right. There are a number of pandoc "enhancements" to markdown that MacDown can't handle, but it gets the vast majority of the formatting right and it prevents me from making stupid mistakes in that majority.
• I keep all my bibliography in Zotero. I export it all as a a bibtex file using Better BibTeX, which provides some nice customization of the export entries. Once this bibtex file is created, I can easily cite the works in it within markdown and then let pandoc-citeproc expand them as appropriate.
• I've given up on using pandoc to produce final versions of the same file in different formats. I'm mostly interested in html, OpenDoc, pdf and—sadly—Word. There are just too many complications in academic documents (footnotes, etc) and my skills and time are limited. LateX PDFs have a certain look to them, but anything I can print, OS X can turn into a PDF, so that's not a big deal for me. In most other cases, I don't need both html and odt/docx versions, so I can skip it there too. (I was really hoping that I could generate my CV in html and PDF directly, since I've been maintaining html and odt versions, the latter of which I turned into PDF via OS X, but I've yet to get enough into LateX to be able to reproduce my mildly complicated CV format.)
• For html, I use a few different versions of my standard css file. It's all up on github, so you can see what I've done. I just discovered rawgit.com, so now I'm converting my html files to refer to the css files there, instead of on the institutional server that I've been using (and which has recently become a bit more difficult to keep updated from home). FYI, I usually edit css field manually with BBEdit, but I'm trying out CSSEdit, which seems to have been EOL'ed.
• On the pandoc side, I've tweaked the default templates for html, odt and docx, so that they can handle multiple authors, as well as a license field in the header. Again, all on github.
• To tie it together a bit, I wrote an applescript that takes the frontmost document in MacDown and runs it through pandoc, outputting whatever type of file you want (based on extension) and also allowing user-inputted pandoc switches.

The overall process then runs like this: write in MacDown, incorporating the citation keys from Zotero; process that with pandoc to generate the desired final file type; publish/share/whatever.

It's seems easy when I write it like that.

Numbering for citation

My one concern with html output of the note that I just wrote was that html has no default pagination, and pages are usually the way one cites an article. So instead of numbered pages, I decided to go with numbered paragraphs. (Read about Sebastian Heath's approach to articles he's editing for ISAW.) But how to number them, so that the numbers were visible (for easy citation) and so that I didn't have to manually put them in? With a little help from the pandoc Google group, I combined some features of pandoc with css. Since pandoc automatically gives IDs to headers and css allows for formatting those headers and even for auto-numbering them, I put a nearly empty level-6 header at the start of each paragraph in my markdown document and I used css to number them and put that number off in the margin. (They're nearly empty because markdown won't create empty headers.) Although the numbers are visible to a human reader, the IDs aren't ideal: section, section-1, section-2, and so on, but they are sequential and linkable. The headers are also a bit ugly in markdown, but they work and they also make it possible for me to indicate logical paragraphs instead of the actual ones. This is useful, for example, when there's a block quote, which technically creates a new paragraph, right in the middle of a logical paragraph.

One more thing, since I'm using css to number the paragraphs in the html version, those numbers are technically part of the display of the article, not part of its content. So you can see them, but you can't find them if you search in your browser. That's not the case in the PDF; there the numbers are "real" and you can find them in a search.

The Article

So about the article itself...it has to do with the original nature of the Golden Calf in Exodus 32. I speculate that it was in origin a "corn calf," to be associated with a lost harvest ritual. Go have a read:

• HTML
• PDF