Scrivener 2.3 and the Mac App Store - Why the Delay?

UPDATE: Scrivener 2.3.1 is now available on the Mac App Store. We resolved the issues mentioned below and our icon designer worked overtime to get the new, larger icon ready, and Apple hurried 2.3.1 through the review process within a few hours of submitting the updated version. Fingers crossed our App Store users won't have such long delays again and that this was a one off.

Many of our Mac App Store users have been writing to us to ask why the Mac App Store is still showing Scrivener 2.2 as the latest version, when the version on our site was updated to 2.3 over three weeks ago. This is as frustrating to us as it is to our users - we would much rather everybody was using our most recent version - but unfortunately we have no control over how long it takes for updates to get onto the Mac App Store, and this time around things have been a little more complicated.

We submitted version 2.3 to the Mac App Store three weeks ago, on the same day we released 2.3 on our site. Every time an app is updated on the App Store, it has to go through Apple's review process again. Usually, it takes anything up to a week before a reviewer picks it up from the queue, and then it takes a day or so to get accepted or rejected. (I don't think we have ever been accepted first time - there's always a new reason for rejection!)

However, the App Store review process has recently slowed to a crawl, presumably because of all the problems Apple had last month whereby they somehow managed to corrupt many of the updated apps they uploaded, causing crashes for thousands of users of different applications. As a result, it took a little over two weeks before we even received notification that 2.3 had made it into review.

A brief digression: A few users have asked us why we can't go back to simultaneous releases - after all, for the past few updates, we have released the Mac App Store version on the same day as the version on our site. But the only way we could do this was by holding back the update from our site until Apple had accepted it for the Mac App Store. We've never been entirely happy about doing this, and have decided, especially with certain things we have had to get into place in our non-MAS version ready for Mountain Lion, that it's better to get an update into as many users' hands as possible as soon as it is ready. After all, simultaneous releases won't get Scrivener to our Mac App Store users any faster; they just make everybody else wait unnecessarily as well. And besides, we thought it would only be a matter of a week or so before App Store users had 2.3 in their hands too.

But, as I say, this time it took over two weeks to get into review. It then took another five days for us to receive the news that Scrivener 2.3 had been rejected.

The rejection wasn't entirely unexpected, although it was disappointing. On 1st June, Apple introduced a new rule on the Mac App Store - all apps sold through the App Store (by third-party developers, at least) must be "sandboxed". Sandboxing is a new security feature which essentially locks down everything an app can do on your Mac - reading and writing files, accessing the internet and so on. For an app to be able to do any of these things (which is obviously vital for a great number of programs), it must get "entitlements" from Apple, justifying them to the reviewer, and even then, some things are prohibited and certain long-standing features of many apps - such as the ability to access a folder or file between launches - are no longer automatically available and involve having the user do a little extra work to grant permission.

Which is all fine (from a technical perspective, at least; this isn't the time to talk about the pros and cons of such a security system, and I am not expert enough in that domain) - I have done a lot of work on sandboxing over the past few months to ensure that, should Apple enforce it, Scrivener would work without too much extra inconvenience to the user. However, Apple said that any apps that had been sold on the App Store before 1st June (as Scrivener was) would not need to be sandboxed for bug-fix updates. So, knowing that sandboxing would cause some minor extra inconvenience to users (users will explicitly have to tell Scrivener which folders it is allowed to access for features such as linked images, references and aliases to work, for instance), we submitted an unsandboxed version of 2.3 in the hope that it would be considered a "bug fix" update. It wasn't.

So, as soon as 2.3 was rejected for not being sandboxed, I was ready to re-submit a sandboxed version immediately. But before I did, I noticed something rather worrying. Although our sandboxed version had been extensively tested on Lion and on some pre-release versions of Mountain Lion, when I tested it out on the developer Golden Master of Mountain Lion, I discovered that some key academic features no longer worked - in particular, the integration with MathType (which relies on Apple Events).

As far as I can tell, this is a bug in sandboxing on Mountain Lion, and I've contacted Apple about this (taking out a Technical Support Incident with them) to see if there is anything I can do about it on my end. But, until I hear back from Apple, at least, it leaves us between the proverbial rock and a hard place. Apple won't let us upload the non-sandboxed version of Scrivener 2.3 (which works fine on Mountain Lion) and sell it through the Mac App Store. But if we upload the sandboxed version of 2.3, then although it will work fine on Lion, as soon as any of our Mac App Store customers upgrade to Mountain Lion later this month, they will find that the MathType integration is broken. (I have appealed to Apple to let this release through the App Store without sandboxing until the issue has been resolved, but that was a no-go.)

I know most of our users are probably not going to be worried about MathType support, and may feel that holding back from re-submitting for a few days over this one feature is unnecessary. But for some of our academic users this is a key feature, and I would like to get it resolved before resubmitting if I can, rather than having to explain to users why this is broken when they update to 10.8.

So, as things stand, I am waiting for a response from Apple and will re-submit 2.3 to the App Store, sandboxed, as soon as I can either fix the MathType issue or know that it cannot be fixed.

But now for the good news: Mac App Store users do not have to wait until 2.3 is released on the Mac App Store in order to start using it; you can just download and run the version from our site if you like. As long as you have run the Mac App Store version once, the version on our site will recognise you as a registered user. This is because the first time you run the Mac App Store version, it copies its internal machine-specific, Scrivener-specific, user-specific validation file (which is installed into the app by the Mac App Store's activation procedures) to an external location on your hard drive (the Application Support folder). Our regular version can then look for that validation file and thus recognise your machine as registered. So please do go ahead and download 2.3 from our site if you don't want to wait for all of these issues to be resolved. (Please note that to install it without any problems, you should delete the version of Scrivener from your Applications folder, first - you will most likely be asked for your admin password when you do this because the Mac App Store write-protects downloaded apps. If you don't do this first, you may have problems installing because of this write-protection. This won't affect any of your Scrivener projects.)

As an addendum, I am going to grumble about one aspect of sandboxing, seeing as implementing it has involved a lot of bug-reporting to Apple over the past few months. I'm not against sandboxing in principle (much), and the Apple sandboxing engineering team has worked hard to fix as many bugs as possible. But the big flaw, in my personal opinion, is that, for the first time in as long as I can remember, Apple has enforced a new technology on third-party developers without first "dog-fooding" it (as all the cool kids say). At the point when Apple announced that all apps on the App Store would have to be sandboxed, of Apple's own software, aside from some small helper apps, only TextEdit and Preview - some of the simplest apps on your Mac - were sandboxed. That's still the case on Lion (on Mountain Lion, a couple of other apps have been sandboxed, but not many). None of Apple's major, high-end programs have yet been sandboxed. Having problems with updating an app on the App Store because of sandboxing issues is thus made just that little bit extra infuriating when you know that Apple released significant, non-sandboxed updates to iPhoto, iMovie, Aperture and Final Cut after their own 1st June deadline (Apple isn't obliged to follow the rules it sets for other developers, of course, this being its own store - but its own software is just as much at risk from security threats as any other). I can't help feeling that, had Apple set out to sandbox all of its major applications before ruling that other developers must, we wouldn't be facing problems such as these, and sandboxing would have been a much smoother transition for developers and, more importantly, for users.

But hey, I'm just a grumbly developer. Fingers crossed Apple helps us iron out these issues and 2.3 (or 2.4) makes it onto the App Store soon. In the meantime, my sincere apologies to our Mac App Store users for these delays.