What You Won’t Find in Scrivener 2.0

Earlier this week I completed the last major block of code for Scrivener 2.0 - at last. Now all that remains is a number of small issues and tweaks and then two months of beta-testing and bug-fixing before the release in October (barring any major catastrophes or problems that crop up during beta-testing).

(A note on beta-testing by the way - thanks for all the volunteers but I now have more beta-testers than I know what to do with, so I can’t accept any more at the moment, sorry. The best way of becoming a beta-tester in the future is just to be really helpful if you find any bugs and to be active on the forums with helping others.)

It’s been so long since I first mentioned that I was working on Scrivener 2.0 that I’m sure there are those who think it’s vapourware, and I can’t blame them; it has been a long, hard slog and I sometimes thought I’d never reach the end, but the finish line is now in sight. It’s been worth it, though - I’m pompously proud of Scrivener 2.0. It’s still the Scrivener you know (and hopefully love), but refined and with all its features much better integrated. This is down, in large part, of course, to the users of Scrivener 1.x, for their feedback and suggestions and the years of real-world use Scrivener has now had. I designed Scrivener 1.0 in isolation as a tool that would suit my own needs for writing long projects, and while I can’t claim that many of its individual features were particularly unique (although some were), it combined these features in ways that hadn’t been done before. I know this, because if it had been done before then I wouldn’t have spent months learning how to code and then two years putting Scrivener together; I would have just used one of the other programs out there and got on with writing (or more honestly, I would have found a different way of procrastinating). I’ve said this elsewhere, but as it’s relevant to what I want to cover in this post, I’ll recap: the three main features I ended up coding Scrivener for, were:

1) The ability to work on a long text in smaller chunks, and to have these chunks combined upon export (and in the event, I also added dynamic combination for editing - Edit Scrivenings). You should be able to structure these chunks however you want.

2) The ability to associate a synopsis with each chunk of text that could appear alongside the document (or chunk) while working on it, or in the context of other synopses so that you could get an overview of the whole (the latter was implemented as the major feature of the corkboard and outliner). Thus you could outline in advance and use the synopses as a reminder about what to write, or write like mad to begin with and then get an overview afterwards for redrafting.

3) The ability to review research files such as images, PDF files and other chapters or notes alongside your text, in the same window.

Of these, back in 2004 (when I started coding, although the idea had been gestating for at least two years before that), several great and well thought-out programs provided at least part of number (1), the ability to work in small chunks, but nothing I could find did (2) or (3), which were something I really wanted. Of course, time has moved on, and Scrivener - I am pleased to say - has done pretty well (although remaining a niche application) and has been noticed by other developers; these features aren’t as unique as they were five years ago. There are "copycat" applications on Windows (sorry, "copycat" sounds a bit disparaging, but I can't think of a better word right now; and I’m not being big-headed or presumptuous here - the developers of these programs all kindly acknowledge that they were “inspired” by Scrivener) and these features have found their way into competitor software on the Mac, too. Which is flattering, I suppose - it means you are doing something right. (A note to Windows users, though: you might want to wait for the original rather than going for the clones…)

Which brings me back to what I was saying about the users of Scrivener 1.x. I designed Scrivener 1.0 in isolation and, over all, I think I did a decent job. But there were aspects of it that just made my head hurt during the design process. (For instance: how and when do the corkboard, outliner and Edit Scrivenings features activate or integrate with the binder selection? How do you switch between them? What happens if you select multiple documents here - how do the different modes deal with that? And so on.) Because certain features hadn’t been integrated or done in the same way before, there were no existing models, so the solutions I came up with were the best I could envisage at the time. (There were plenty of good models for individual features, of course: What makes a good outliner? Look at OmniOutliner for the basics; What’s a good meta-data system? Look at Ulysses’ labels and Aperture’s keywords; What would make an efficient but simple versioning system? Look at CopyWrite’s versions and Page Four’s snapshots features. It was the gestalt that was the design problem, trying to put together certain features in one interface in an integrated way that hadn't been done before.) In the years after Scrivener was released into the wild, it became clear which features needed more thought. If one in ten thousand users has a minor problem with a feature that you think is otherwise good, you can probably put it down to user idiosyncrasy (yes, users can be idiosyncratic as well as programmers, okay?). But after you’ve seen dozens of posts asking the same question on the forums, you can’t pretend it’s user error - something in the design needs fixing.

And that has been the learning curve of Scrivener 1.x: Scrivener 1.0 was fully formed but roughly-hewn; the attentive eyes of its users have pointed up its flaws, and I’ve spent the past two years fixing them, refining. So my sincere thanks to everybody who has taken the time to send me feedback, either in the form of bug reports, criticisms or feature requests: even the ones I didn’t agree with at the very least made me look at Scrivener’s features in a new light. I hope you’ll all like Scrivener 2.0 as much as I do.

My plan for Scrivener 2.0 all along, then, has been to refine what is there rather than to throw in two hundred new features just for the sake of it. With these thoughts in mind, the purpose of this post is not to talk about what is coming in Scrivener 2.0 - I’ll be posting information about that at the beginning of September, when I get back from my Spanish holiday. Instead, I’d like to get out of the way those things that aren’t coming. The things I get asked for regularly but which won’t be in Scrivener 2.0, either because they just don’t fit within the scope of what Scrivener is, or for technical reasons.

So, without further ado (I know, you’re thinking there has already been one thousand words of ado), here are the things that you won’t be seeing in Scrivener 2.0:

A Timeline

I think I should clear this up once and for all, given that it’s a request that comes up every so often: Scrivener is unlikely ever to have a timeline feature. (I won’t say “never”, but it is highly unlikely.) I can - sort of - see the logic behind this request. At first glance it may seem that Scrivener has a corkboard, an outliner, a word processor, and hey, these are writers’ tools, and a timeline is also a writing tool, so why not have that too, all in one place? But it’s not quite like that. Scrivener’s corkboard, outliner and editor are all integrated (and even more so in version 2.0) - they are different ways of viewing the same structural information; each provides a window onto a part of the structure of the binder, showing either the subdocuments or the textual (or media) content of a document or group.

A timeline, however, is something completely different, and somewhat antithetical to Scrivener’s structure-based system. Unlike Scrivener’s outliner and corkboard, there is no way that a timeline could be directly integrated with the file/folder structure (except for perhaps the most linear and flat-structured of projects). For instance, how could you access it in the interface? The corkboard and outliner represent the contents of the folder or container you have selected in the binder. You wouldn't want a separate timeline only covering the subdocuments of each container, though, and yet that's the way Scrivener “thinks”. More importantly, there is an inherent conflict between a structure-based system (such as Scrivener’s) and a time+player-based system (which is the basis for timeline programs). There is no logical way for changes in one system to affect the other (especially given that any given story may be told chronologically, backwards, in medias res or, Slaughterhouse 5-style, all over the place).

Here’s an example of what I mean: suppose you have two chapters in your binder. The first chapter deals with things from Bob’s point of view, and the second chapter revisits the same events from Jane’s point of view (or, if you’re an academic user, let’s say that one chapter deals with Chaucer’s life and the other with Gower’s). This isn’t an unusual structure - Kundera’s Unbearable Lightness of Being uses it as does the current Booker-nominee, The Slap. In this example, you might have two folders in the binder, one for each chapter, each containing different documents for each scene. But moving a scene from Chapter One to Chapter Two (and presumably adjusting the POV) tells us nothing about its placement on the timeline, because both chapters cover the same events. Meanwhile, moving a single event on the timeline could in this case conceivably need to affect both chapters. On the other hand, you might have events happening at the same time as those described in Chapters One and Two but involving another character on the timeline, and that will only be revealed in Chapter Forty; in this case, you wouldn’t want changes to these events in the timeline to affect either chapter in the binder. And then what about concurrent events - how is that dealt with in the binder? And so on. This assumes that each event on the timeline equates to a single document in the binder, of course. But equally, you might only have two documents in the binder, one for each chapter; it depends how you work. In this case the timeline wouldn’t even be able to have its events equate to documents in the binder, as you would most likely want the timeline broken up into smaller segments than your files. In fact, my guess is that it would be rare for every event on the timeline to fit nicely with individual documents in the binder (and even in the unlikely circumstance that they did, what happens when you merge documents?).

This all assumes that a timeline would be integrated in such a way that changes in the binder structure automatically make changes in the timeline and vice versa, and there is no logical solution to these problems because of the inherently different nature of the two systems. (Note that at first glance it may seem that the excellent StoryLines in Writer’s Café - surely one of the best timeline implementations around - manages to integrate these two systems, but in fact it does not - the outliner attached to StoryLines is not comparable to the one in Scrivener.) The only alternative is that any timeline system would have to be divorced from Scrivener's main feature-set - it could not be integrated in the same way as the corkboard or outliner, but would have to sit apart from everything else, in its own window somewhere. But if that is the only way of implementing a timeline, the usefulness of having it as part of the same application is severely diminished - if it’s going to be sitting in a separate window, not interacting with the rest of the program, wouldn’t it be better to use a dedicated timeline application, the developer of which is dedicated to specific timeline features? I think so, and this is why a timeline is unlikely to come to Scrivener. (For those who want to keep track of the date of events in their stories, though, Scrivener 2.0 will feature the ability to add custom text meta-data columns to the outliner, and the outliner will be sortable.)

You are still welcome to send me e-mails or post on the forums appealing for a timeline, of course, but if you do so, I ask that you do two things:

1) Think in depth about how it would integrate with Scrivener’s other features, bearing in mind that Scrivener is used by all sorts of users from novelists to academics, screenwriters to lawyers, and provide a clearly broken-down and lucid explanation of the interface and how it interacts with the rest of Scrivener. (No, that’s not my job, because I’m not the one who wants it, so there.)

2) Provide some drawings. (And send beer. Beer is always good.)

I ask for these things not to be facetious, but because I have spent a lot of time considering all of this, drawing out ideas and so on, so just telling me you want a timeline won’t help me or you. And maybe you can see a solution where I cannot. If so, I’ll be happy to discuss it. (But not for 2.0!)

(A quick clarification: One of the reasons I've held back from explaining why I don't think a timeline fits into Scrivener before now is that I'm aware StoryMill has a timeline - in a separate window, if I recall correctly - and I don't want my reasons for not providing a timeline to come across as a veiled criticism of that excellent application. These comments are not intended in that spirit at all; StoryMill and Scrivener are very different applications and if you are still working out which software is for you, you should try them both. And if their timeline wins you over, well, I'll just have to live with it...)

A Mindmap

This one can be mostly covered by, “See above” - many of my reasons for not providing a timeline apply here, too. Mindmaps are another one of those tools that writers use and thus users request for Scrivener based on the fact that Scrivener has a corkboard. But mindmaps have no structure - they are free-form tools that allow you to get your ideas down in any order, grouping them spatially rather than linearly. There is no way that the freeform chaos of a mindmap can be mapped across to the linear grouped structure of Scrivener’s binder. So again, any mindmap feature would need to exist in isolation in a separate window, in which case it would be better to use one of the great dedicated tools that are already out there, developed by someone who can put all of his or her time into making it the best mindmap possible (which I wouldn’t be able to do given that it wouldn’t be a core feature). “But wait!” you cry. “What about just letting me move the cards on the corkboard around freely while trying out ideas?” To that I reply: this post is about features that Scrivener 2.0 won’t have, so let’s talk about that next time.

NB. Ioa tells me that what I am calling a mindmap is technically a “concept map or diagram”, as mindmaps are something different that are often tied to structural representations, but I’ll leave my loose usage the term as it stands because when most users ask for this they are talking about “thought clustering” and suchlike. But even structured mindmaps tend to fork ideas from a central, root node, because they are about brainstorming a network of ideas, and that’s not something that’s really compatible with the folder/file structure of Scrivener.

Perhaps a more salient point about timelines and mindmaps though is that to develop a really great timeline or mindmap (and there are some great ones out there - one Scrivener user is currently working on a timeline program called Aeon, too) is something that takes years. These aren’t single features but are programs unto themselves, and thus are better left to dedicated tools.

Fiction-Specific Features Such As Character and Tension Tracking

Occasionally I’m asked if I can add something that will help users keep track of characters or story tension, but I won’t be adding anything like this mainly because Scrivener isn’t solely for fiction, and such features would limit its use to fiction writing. We have lots of academic, business and other nonfiction-writing users as well as the many novelists and scriptwriters, and I much prefer to keep Scrivener flexible rather than tie it down to specific uses. (Again, personal bias comes in here, though, as I’m not especially fond of software character and tension tracking.) That said, Scrivener 2.0 will allow you to assign custom icons to folders and to set up in-project document templates, making it easy to set up character and location root folders and character sheet templates, but for now let’s stick to what isn’t getting included and I’ll return to these new features in the future.

A Proper Styles System

I fully admit that one of the reasons a proper styles system has remained a low priority is that I don’t use them myself, but this is still on the list of things to investigate further in the future. There are several problems with implementing a full styles system in Scrivener, but they are mostly technical rather than conceptual - a styles system would be a nice addition in the future. I won’t bore you with all the technical issues, but they include the following: unlike word processors, which open all of the text at once in one long file, Scrivener has to manage potentially hundreds of text documents and so would have to apply styles to documents that aren’t even open; the text system that Scrivener uses - which is the standard OS X one also used in TextEdit - has no out-of-the-box styles system, so I need to create my own; even if I did create a styles system inside Scrivener, it would then need to be incorporated into the RTF export, and that would most likely involve writing my own RTF parser - I can’t hack in a styles system to the existing RTF parser as I have done with other features. So: styles aren’t off the list, I understand why people want them and hope to return to them at some point in the future, but they won’t be included in Scrivener 2.0.

Track Changes

Microsoft Word and Apple Pages have a “track changes” feature that allows you to keep track of any edits made to a document, and to review and commit them. It’s a great feature and I know a lot of editors use it, so it comes up as a feature request for Scrivener from time to time. Unfortunately it’s not something that’s likely to make it into Scrivener for the foreseeable future. As with styles and Pages integration, the problem here is a technical one - it’s not that I think Track Changes is something that is antithetical to Scrivener, because it isn’t. It is not a coincidence that only the word processors produced by the two giants of software and computing, Microsoft and Apple, have this feature (and OpenOffice, which has been around for ten years and before that was StarOffice, and has many developers working on it). It is such a beast of a feature that I’m sure it takes a whole team of experienced programmers to implement. It has to intercept and monitor your typing (without slowing it down), keep track of deletions, display changed text in different ways, provide information about which author has done what, and much more. But even if I spent a few months cracking all of this, a Track Changes feature in isolation in Scrivener would be of only limited use. The main reason users want this feature is so that they can import Word or Pages documents that have been edited by collaborators or editors, and work on reviewing the changes in Scrivener. For this to work, on top of implementing Track Changes, I would also have to write my own .doc or .docx importer that could read all of the Track Changes information - and as explained below, as a lone developer, I rely on certain standard importers and exporters, because writing even a single custom file translator is an enormous task, and mostly beyond the reach of small shareware companies such as L&L. Thus, Track Changes remains one of those features that we would love to add in the future, if we one day become insanely rich and are able to employ a whole team of programmers. In the meantime, though, Scrivener 2.0 will feature a way of showing simple comparisons between a snapshot and the current version of a document (and between two snapshots) - but more on that next time.

iWork Pages Integration

“For the love of all that is sacred, Scrivener is a Mac program, and you don’t support Pages? What is wrong with you!!!?!?!” is something that comes up frequently. And in this regard, Scrivener 2.0 will have no advances, sorry - Scrivener 2.0 will still not support the .pages format. Is this because I hate Pages and Pages users? No, I like Pages - I think it’s a beautiful word processor and I would dearly love to provide better support for it. Unfortunately, there’s a bit of a Catch-22 situation going on here, though, which currently makes better support impossible.

The first problem is that Apple doesn’t make the .pages file format available to anybody outside of Apple, so third-party developers cannot support Pages directly. Nor does it appear that they have any intention of making it public. They published partial specs for the Pages 1.0 file format (which used to be at http://developer.apple.com/documentation/appleapplications/Conceptual/iWork2-0_XML/Chapter01/chapter_1_section_1.html#//apple_ref/doc/uid/TP40002479-CH202-DontLinkElementID_62 but no longer seems to be available), but even then they made the following caveats:

“Important: This document only covers the file formats for Keynote 2.x and Pages 1.x. Future versions of those products may use a different file format than the ones described here. Developers should understand that Apple cannot guarantee that the file formats described herein will be supported in those future versions of the iWork applications as they are currently supported. Changes to these file formats ought to be expected.”

And:

“This document does not describe the complete XML schema for either Pages 1.x or Keynote 2.x. The complete XML schema for both applications is not available and will not be made public.” (Italics mine.)

So, there’s no way I can write my own .pages importer/exporter without reverse-engineering the file format and expecting it to break with each iWork update (writing my own importer/exporter would take a long time as a single coder anyway, and would be painstaking work). Unfortunately, neither do Apple provide an importer/exporter “out of the box” for developers to use. Apple’s text system is fantastic in that it provides us with standard importers and exporters for RTF, RTFD, HTML, TXT, DOC, DOCX and ODT (although that last three are somewhat buggy and broken, admittedly), but there is no support for .pages (there was never one for AppleWorks, either).

But: Pages doesn’t just read .pages files, of course. It can also read .doc, .docx, .rtf and .rtfd files. This is where the Catch-22 situation comes in. Because .doc and .docx are massive and complex file formats and I’m just a lone developer, Scrivener relies on Apple’s standard exporters for these formats, but, as I mentioned parenthetically above, the standard exporters for these formats are somewhat broken: indents and line spacing get messed up. Moreover, they are very basic and don’t support images, comments, footnotes or headers and footers. In other words, although Pages has great support for both these formats, Scrivener does not, and won’t until such time as I can afford to hire a programming whizz who can knock me together a full-on .doc and .docx translator (at which point, no doubt, the Apple text system will change so that they’ll need writing again).

However, Scrivener does have great support for the RTF format. The RTF format is a well-established, standard rich text format supported by all major word processors (it has been around for twenty years and in the Windows world is second only to the Word .doc formats), and it is capable of storing almost all of the same information as a .doc or .docx file. So an RTF file can store images, footnotes, comments, headers and footers and so on. It’s a great exchange format because every version of Microsoft Word fully supports it, Nisus Writer uses it as its default format, Mellel and OpenOffice have support for most of its features, and most Windows word processors or rich text editors other than Word and OpenOffice-based ones use it as their main file format. Scrivener has strong RTF support simply because RTF is essentially a marked-up plain text format that has been well-documented for years. So although the standard Apple RTF exporters don’t support features such as footnotes, comments, images and so on, I have been able to add these features by post-processing RTF files in a way that is unfortunately not possible with .doc and .docx files.

The catch, of course, is that while Pages has strong .doc and .docx support but Scrivener does not, Scrivener has strong .rtf support… but Pages does not. In fact, Pages has the worst RTF support of any major word processor. Not only does it drop comments, footnotes, images and headers and footers, but it even drops the page breaks. So sadly, for now, we’re at an impasse, and this is why Scrivener 2.0 won’t have better Pages support. It’s something I’d love to provide, and something users are more than justified in demanding, but for now it remains out of reach - and, for the most part, in the hands of Apple. Most of this, incidentally, isn’t too much of a problem if you aren’t writing complex documents - if you want to transfer a novel or short story draft to Pages, for instance, you can just use RTFD and most of the formatting you want will remain intact in Pages (excepting page breaks). But if you need footnotes and suchlike, it is a problem, and there are three possible solutions:

1. Don’t use Pages, use the wonderful Nisus Writer instead. (Okay, if you like Pages then admittedly that’s probably not practical, and the quick-witted among you will retort with the ancillary, “Or don’t use Scrivener,” but I just wanted to plug Nisus given that Martin, one of Nisus’s main programmers, has helped me so much over the years, and it is a great alternative.)
2. Install OpenOffice and use it purely as a file translator. Export from Scrivener as RTF, open the RTF in OpenOffice and save it out again as a .doc file (because OpenOffice has decent support for both file formats), and then open the .doc file in Pages.
3. Drop Apple a line and ask them nicely to implement better RTF support in the next version of Pages:

http://www.apple.com/feedback/pages.html

We can hope!

An iPad Version

Okay, this one isn’t technically a feature request for 2.0, it’s just an excuse to give you an update. Since the iPad has been released and users have now experienced the strengths and weaknesses of this wonderful device first-hand, the requests have greatly died down, but it’s still worth mentioning. The good news is that I do have someone looking into an iPad version, but the bad news is that he won’t have chance to look at it properly until after 2.0 is out and even if it does go ahead, it won’t be for some time yet. I’m sorry to report that I still have no personal interest in such a version - I use my iPad as a consumption device (and it certainly is very nice for reading web pages; I also bought my Mom one and no one can get her off Bed Bugs) but I can’t stand to write on the thing (I have in fact bought an AlphaSmart for writing on-the-go, which is a thing of beauty for typing); so, each to his own. I am aware that some people (although undeniably a tiny minority of computer users in general) now like using their iPads for everything, and I do wish them good luck with it, but, crazy hippy that I am, I’m afraid I’m less interested in “following the market” for financial gain than I am in creating the best tool for my own (and existing users’) use on the platform I use and love. I’m very grateful to those who have e-mailed me showing enthusiasm for an iPad version, though - it is great that users like Scrivener enough to want it on other platforms. And thank you, too, to those concerned enough about the welfare of L&L to worry that our business will fail unless we make them an iPad version (I hope they find some comfort in the knowledge that Mac sales have increased since the advent of the iPad); it is certainly a good point that we should investigate other platforms in the long run, and to this end we are actively looking into a Windows version (which has an even bigger market share than the iPad, I am told).

But if that all seems rather negative, here is something positive: Scrivener 2.0 will feature the ability to sync text documents to Simplenote and Dropbox, so you will be able to edit and create notes for your Scrivener project on an iPad using, for instance, the Simplenote app or Hog Bay Software’s upcoming PlainText app. I imagine that for a great many users, these new features will supplant the need for a dedicated app altogether, given that even a dedicated app could only ever do a tiny fraction of what the Mac version can do anyway, and you will now be able to “pack up a lunch” (as Andy Ihnatko put it) from your project to take with you for editing on the iPad. But more on that soon. And all the same, as I say, post-2.0 someone will be looking into a dedicated app. (So let’s try to avoid too many comments debating the pros and cons of an iPad version - we hear ya.)

I apologise for yet another long and rambling post - I’ve been meaning to explain many of these things for some time now and so had a lot to get through and not enough time to edit it down into something of a more reasonable length (I have an October deadline to meet, after all.) And finally, once again, a note to all Scrivener 1.x users: thank you, sincerely. As they say in the ’States: You rock. As vic-k on the forums likes to point out, I can come across as a grumpy code monkey sometimes, and thankfully I have David and Ioa on board these days too who have far better social skills than yours truly (actually, I’m not so sure about David, and I’ve never met Ioa in person…), so I just want to express my appreciation for all that you guys have done over the past few years. We’ve still got a long way to go (and more on that in the future), but you have blogged about us, raved about us on review sites, told your friends, told fellow students and mentioned us in lectures; some of you have even mentioned Scrivener in the credits of your published books. It means a lot to us, and often when I've been tearing my hair out over some tricky piece of 2.0 code, an enthusiastic e-mail from a user or a blog mention in my Google Alerts has helped. So: thank you.

Right, that's all the negative, what's-not-coming stuff out of the way. Next time, when I’m back from my hols, I’ll spill the beans on Scrivener 2.0. I’m nervous as hell about it, because it’s taken up the past two years or so of my life. I hope you’ll like it. I think you will.