The L&L Blog / Scrivener

Epub, Kindle and MultiMarkdown Export in Scrivener 3

KB / 21 SEP 2017

Scrivener 3 brings with it much better support for creating Epub and Kindle files.

Note: This blog post pertains to upcoming features in Scrivener 3, which will be released on macOS later this year and will follow on Windows during early 2021.

Additional note: This blog post specifically relates to the development of the macOS version, but the improvements that resulted from this development are coming to Windows too.

One of the many areas of Scrivener I wanted to improve for version 3 was its Epub export. Scrivener 2.x creates Epub 2 files, even though the latest Epub specification at the time of writing is 3.1. Moreover, internally those Epub 2 files are a little messier than is ideal. An Epub file is essentially just a zip package containing a bunch of HTML, CSS and XML files (CSS files define the formatting used in the HTML). Scrivener 2.x creates a separate CSS file for each HTML file in the book (when it’s better practice to create a single CSS file that is used by all of the HTML files) and the HTML is a little messy.

Ordinarily, the internals of an Epub file shouldn’t worry anyone except for those who are passionate about parsimony—the most important thing is that it looks as it should on e-reader devices. However, the use of multiple CSS files has a deleterious effect on Kindle files. When you export a Kindle .mobi file from Scrivener, it is constructed using the same code as Epub export, and then converted to Kindle format using Amazon’s KindleGen utility. Scrivener-generated Kindle files look great on Kindle devices, but they don’t work so well with Amazon’s “Look Inside” feature. This is because a bug in Amazon’s “Look Inside” feature means it takes formatting from only the first CSS file in the book, regardless of whether or not it’s the one that’s supposed to be used for the text being displayed. This can result in the text using the wrong formatting—not a great introduction to your book for the reader!

All of these issues originate from a single cause: the HTML generator that Scrivener 2.x uses. To create ebook files, I have to convert the rich text in Scrivener’s editor to HTML format. Apple provides an HTML converter built in to the text system, so I have always used that. However, Apple’s converter only supports HTML 4 (whereas Epub 3 requires HTML 5), it produces rather messy HTML with a lot of cruft, and I have very little control over it (there’s no way of setting it up to use a single CSS for multiple output files, for instance).

For Scrivener 3, then, I needed to find a way of converting Scrivener’s rich text to HTML 5, and I needed more control over the HTML generation. I began by hunting high and low (sing: there’s no end to the lengths I’ll go to) for a third-party converter, but there is very little out there in the way of third-party converters that work with Apple’s text engine. I considered writing my own converter from scratch, but that would be a mammoth undertaking that would require regular maintenance.

The solution? MultiMarkdown. MultiMarkdown can already convert text to HTML 5, and it does so cleanly and allows a great deal of control and flexibility.

The problem? For MultiMarkdown to convert text to HTML 5, the text has to be written in MultiMarkdown syntax—that is, with asterisks denoting bold and italics, pipes and dashes representing tables, greater-than symbols representing the start of block quote paragraphs, and so on. You can’t just pipe rich text though MultiMarkdown and have it come out with the formatting intact.

The solution to the problem with the solution? I spent a couple of weeks writing a full rich-text-to-MultiMarkdown converter. This takes a rich text document and converts it all to MultiMarkdown syntax: bold, italics, footnotes, tables, lists, block quotes—the works. With that in place, to generate an Epub 3 file or an improved Kindle file, internally I could convert the rich text to MultiMarkdown and then the MultiMarkdown to HTML 5.

The result of all of this work is this:

Scrivener 3 can export to Epub 3 format.
Its Kindle export is also much improved, and should now work fine with Amazon’s “Look Inside” feature.
The internals of Epub 3 and Kindle files are tidier, containing only a single CSS file.
In Compile, you can optionally override the CSS that is generated. This means that those with experience in CSS will be able to add features such as drop caps (although not many e-readers support that yet) or anything else that CSS and e-readers support.
In Scrivener 3, as a nifty side effect of all this work, you can export to any MultiMarkdown format even if you don’t write using MultiMarkdown syntax: you can simply tick a checkbox and have all of the rich text converted to MultiMarkdown during export.

Compiling an Epub 3 file relies in large part on Scrivener 3’s new styles feature—for block quotes and other formatting to appear as expected in the exported ebook, you’ll need to apply styles to the text (something not available in earlier versions of Scrivener). To make the transition to Scrivener 3 as seamless as possible, for the time being it will offer the ability to choose between Epub 3 and Epub 2 export (and between older and newer versions of the Kindle export). This means that you won’t have to worry about updating the text in existing projects to use styles just to get it to work with Epub or Kindle export—you can continue to use the older exporters until you’re ready to update your text or start a new project.

Scrivener 3: Redesigning Compile

26 Comments

razyr / 21 SEPTEMBER 2017

I think you've addressed most of the major issues that I've seen with 2.0.

Will the default CSS file be well documented or will there be a way to discover what style is in use for a particular element from within Scrivener? Or is the expectation that we'll load the HTML into a browser and use the Inspect feature there?

Michael Holley / 21 SEPTEMBER 2017

With all of this work with Markdown will there be better support with actually writing in Markdown? Like auto formatting similar to TextNut or maybe syntax highlighting like Ulysses?

Ricardo / 21 SEPTEMBER 2017

The option to export to MultiMarkdown is very interesting.

@KB, I asked in a previous post ("Bookmarking Folders") about screenwriting. Any news on that front?

bewe / 22 SEPTEMBER 2017

@MichaelHolley unfortunately not: https://www.literatureandlatte.com/blog/?p=1094#comment-1461

The only thing I really miss. As a workaround I regularly compile and use Marked to preview the result

PerlinP / 22 SEPTEMBER 2017

@MichaelHolley @bewe : IMHO, I think it's a very good thing that Scrivener remains above all a rich text editor. It is one of the main reasons why Scrivener is my favorite writing software. Not a big fan of #, *, ** everywhere on the page while I write.

There are, in the market, markdown editors that work well. Specialization makes better.

However, I don't know whether or not a 'Markdown mode' (like the current 'script mode') could be implemented in Scrivener.

JTW

JohnTWolff / 22 SEPTEMBER 2017

I am now a Pavlovian dog. Literature & Latte posts a Scrivener 3 feature article, and I salivate.

Every time.

Release it soon or I'll chew your slippers.

KB / 22 SEPTEMBER 2017

@Ricardo: Sorry about that! There aren't too many changes in scriptwriting in Scrivener 3, except that the Windows version will be updated to have all the features of the macOS version. Also, a <em>lot</em> of work has been done to ensure that the Windows version works well on high DPI monitors. There are still a few issues to work out (it looks great on most machines but looks a little large on my ZenBook still), but all of the previous DPI issues should be resolved for 3.0 (I hope!).

Fountain import and export is in there, but for mid-dialogue cuts you will still need to export to a dedicated scriptwriting app, just as you do on a Mac. Likewise for dual dialogue (although you should be able to generate dual dialogue in an exported FDX file). Mid-dialogue cut-offs and dual dialogue are technical hurdles, especially for a general-purpose word processor engine such as Scrivener's. That's not to say that they will never make it into Scrivener, but they are not part of 3.0.

Scrivener was originally intended as a first-draft tool from which you would ultimately export for tweaking in dedicated layout tools. That line has blurred a little with ebook support and such, but it is still generally true for screenwriting.

@JohnTWolff: Thanks, I'm glad you're looking forward to it!

@razyr: If you look at the screenshot in the post, you can see that the CSS there is well-commented, so that you can understand what it all does. You'll know what element is in use for which style because you will use Scrivener 3's new styles system to determine this. (Even if you don't use styles in the main editor, you can apply them to text at Compile time.) You can even choose the names for the styles CSS elements if you want, overriding the defaults. Of course, ultimately you will need to test the export in several ebook readers, especially as different readers have varying levels of support for different CSS elements (drop caps, for instance, is supported by very few). For this, there's a new "Test" button in the Compile area that allows you to generate a test Compile without yet saving your settings or closing the panel, and from which you can choose to open the exported file in an app of your choice.

@Michael Holley: As bewe says, no, there is no Markdown auto-formatting mode in Scrivener 3, nor any planned for the future. Scrivener has been built from the ground up to be a rich text app. Such a mode would disable probably half of Scrivener's menus and features and require a whole slew of new settings that apply to some projects but not others. Ultimately, though, for my sins, I like rich text. There are a tonne of great Markdown apps out there, but Scrivener is firmly in the rich text camp.

Ricardo / 22 SEPTEMBER 2017

@KB Thanks. That's great news!

What about character names autocomplete? That'd be a very useful feature for writing a first-draft. Or is it already one of the features of the macOS version?

Scrivener 3 looks great! You're doing an amazing job.

@Ricardo: I haven't checked the progress of character names autocomplete in Windows, but it should definitely be in for 3.0 because it's already a feature of macOS 2.x (as well as locations auto-complete). Thanks for the kind words!

Eric Beaty / 22 SEPTEMBER 2017

@KB,

I left a question regarding working with a professional editor in the previous post about the new compiling features. Could I get an answer to that, please?

This is great news, especially about the Amazon "Look Inside" feature. For the longest I was pulling my already thinning hair out wondering why my compiled .mobi wasn't showing up correctly in it. Now I know, and now I'm excited to get it to work much better!

@Eric: Sorry about that, I missed your question there. I've answered it now. Thanks for the kind words! (And cherish your hair while you have it - you'll miss it when it's gone, trust me. :) )

elgarak / 22 SEPTEMBER 2017

I'm with JohnTWolff. Pavlovian Scrivener user.

Re. Markdown: I don't see the big problem, given that Markdown is (mostly, depending on variant) a subset of the RTF formatting possibilities, Scrivener has Markdown import, and a lot of Markdown editors and tools have RTF export. Sure, it makes the workflow a bit more complicated, but it appears to be very stable.

DavidWSnow / 22 SEPTEMBER 2017

I am glad to see that you are improving Scrivener's features for EPUB and MOBI with MultiMarkDown. It would be great to make it easier to get some of the text formatting features that I see in other people's Kindle eBooks.

Will this also improve the handling of figures so that the captions are tied to the images?

Won't this make it harder to produce both Kindle eBook and a CreateSpace paperback from the same source? Do you have or recommend a good MMD to PDF converter? I am a believer that also selling the book in paperback on Amazon boosts your eBook sales.

Torsten / 23 SEPTEMBER 2017

Great new features!, and good see that markdown is so useful. What a pity that pandoc does not support RTF as an input format, only as a output format. Otherwise, you could have built on that instead of coding your own converter. Have you considered helping pandoc make RTF work as inout format? Would be a great addition.

I would then have suggested to build on pandoc (which also knows multimarkdown). Then, we would have a real diamond working inside Srivener with a great markdown dialect and a truly universal document format converter on top!

Mike / 24 SEPTEMBER 2017

Can't wait for the new release of Scrivener, just a few questions here:
1- Do you have any plan to have a better (more intuitive) integration with tools like Zotero?
2- Using Scrivener mostly to prepare my teaching and training notes, I constantly reuse part of the information from other files. At the moment I just do an ugly "Copy-Paste" but would prefer to reference the source and be able "at compile" to import as text the reference. Any better idea how I can do that?

Thanks,

KB / 25 SEPTEMBER 2017

@elgarak - always nice to have salivating Scrivener users (as long as it's not on the keyboard :) )!

@DavidWSnow: The screenshot shows the settings for one of the built-in formats. In the screenshot, on the right you can see the CSS generated by Scrivener from all the options selected. On the left, you can see that I have chosen an option to append some custom CSS. Using this, I add some nice borders around chapter titles and tweak the way things look. You'll be able to format ebooks however you want, basically.

And yes, it allows you to tie captions to images. For captions you will need to use a style in your text, placing the caption directly above or below an image, and then you just select the style as the "Caption" style in Compile. All that's set up for you by default, though - in practice all you need to do is use the pre-provided "Caption" style for captions, then select the "Ebook" format when you compile.

Creating a paperback from the same source will be as easy as - easier than - it was before. The rich-text-to-MMD-to-HTML-to-ebook conversion is all done internally, so you can create paperback output just as you could in 2.x.

@Torsten: the code I have that converts rich text to MMD/Pandoc syntax is very specific to the Apple text system. It doesn't take the RTF format as an input but rather an NSAttributedString (Apple's class for rich text objects), so unfortunately it would be of no use as a contribution to Pandoc. For those who have Pandoc installed, Scrivener 3 offers Pandoc options in Compile as well as MMD, though, and you can even choose a custom post-processor for those who are a bit more technical (this is the sort of thing most users will never touch or need to know about, but it's there).

@Mike: We get a lot of questions about better integration with Zotero, but every time I ask someone what this would actually involve, no one is able to tell me. :) Please see this thread on our forum:

http://www.literatureandlatte.com/forum/viewtopic.php?f=4&t=34766

As for reusing information, Scrivener has a new placeholder tag. You can use this to include the text from other documents inside Scrivener, or to include the text of text files stored outside Scrivener, embedding it right into the text during Compile. So you should be able to do what you want there in 3.0.

nontroppo / 25 SEPTEMBER 2017

@MichaelHolley & @bewe: Regarding Markdown "visualisation", Scrivener 3 can do something that all other editors can only dream about. A couple of blog posts back Keith mentioned that styles allow conversion to inject a prefix/suffix. Therefore you can fully write with styles to both visually identify and semantically "label" your text sections/blocks. This makes the experience in the editor fully visual (strong / emphasis / superscript / block quote / code block etc.), you use named styles instead of (or in addition to) markup. Then, when you compile, Scrivener can converts character and paragraphs styles into the markup you specify, whether it is MMD / Pandoc / Asciidoc / you name it!!! Styles can be bound to keyboard shortcuts, so you can speed through semantic markup like a Tasmanian Devil!

The work that Keith has done on making his RTF system styled, and thus flexibly convertible to MMD or Pandoc and beyond really offers us the best of both worlds. Keith suggests that Scrivener is firmly in the "rich text" camp, yet he has given us the most elegant set of bridges to multiple plain text tools imaginable...

antony / 27 SEPTEMBER 2017

I'm just here for the A-Ha references.

KB / 27 SEPTEMBER 2017

I like your train of thought.

Torsten / 28 SEPTEMBER 2017

Wow again! Thanks so much Keith, for expanding the markdown support in such a flexible way! Can’t wait! Can you elaborate a bit more on the Pandoc options in the Compile?

The only thing missing in Compile recently (in connection with Pandoc) was for me to be able to convert comments in Scrivener texts to multi-paragraph footnotes. In Compile I can change the markers around comments to look like "^[" and "]" which allows for Pandoc’s inline footnotes, but I cannot do "[^“ and „]“ since that requires an identifier and that the comment text must moved from the actual place to some individual blocks.

The conversion from a style to some markup/markdown syntax that @nontroppo mentions, is once more a very well designed piece of code! I believe that Scrivener 3 will be my favorite markdown editor. It already is now, but the new possibilities will blow away my last tiny doubts.

KB / 29 SEPTEMBER 2017

@Torsten: Basically, if you have Pandoc installed on your system (and are running the version of Scrivener 3 downloaded from our site - the App Store has limitations) then several Pandoc formats will appear in the Compile menu in addition to the MMD formats (.docx, DocBook and epub). But also, you can create your own Compile format, choose to use MMD as the output format, tell it to use Pandoc syntax, and choose your own post-processing path and command-line arguments. So... you can do a lot, basically. :)

I'll have to check up with Ioa, our resident Pandoc and MMD expert, on the comments thing. He advises me on all MultiMarkdown and Pandoc implementations as he is more knowledgeable on those formats than I am.

IaP / 29 SEPTEMBER 2017

Torsten,

I might not be following the nature of your query, but it seems to me that you should be using Scrivener's footnotes instead of comments or inline annotations. Since version 1.0 in fact it has supported [^id1] style markers in the text, linked to references at the bottom of the document, and it has supported multi-paragraph notes in this fashion as well. So yeah, give it a try right now in 2.8. I just ran a quick test and got a clean multi-paragraph footnote via Pandoc -> DOCX & HTML.

That said, if I sat and scratched my head enough I could perhaps derive a way of generating remote blocks of content and unique markers with a combination of styles and maybe some placeholders using the new 3.0 toolset---and if not that, in conjunction with the aforementioned capabilities Keith brought up: a simple Ruby post-processor hooked into the compile settings. I wouldn't do it though---because native footnotes work better. :) Point being: I think you're all set, but even if not, I bet you can do whatever it is you need to do. There is a truckload of potential coming down the pipe (pun intended).

Torsten / 02 OCTOBER 2017

Thanks, I need to think about that and read the manual once more … What I liked is the fact, that you can just add comments in Scrivener, have them nicely displayed beside the text and then decide on Compile what to do with them. When compiling for Pandoc I use plain text compile and Scrivener will convert the comments into footnote if I wish so (or just remove them if I don’t want to have them in a clean cersion). But, I can also choose to compile to docx and then let the comments pass through as is. I love this flexibility.

nontroppo / 05 OCTOBER 2017

@Torsten: I would also like a slightly better way for handling conversion of comments, something perhaps to think about when Scrivener 3 comes out for a wishlist post...

One cool change coming to Pandoc V2.0 is that it can handle proper commenting to/from word documents. It uses two elements so you can markup the range of text as well as the comment contents. With a small change to Scrivener we could build full support for markup conversion from comments to word (and other pandoc writers if needed)

Scrivener

Scapple

Forum

Blog