Apple is not going to Open Source OS X

Last week I was listening to episode 203 the MacBreak Weekly Podcast. I agree with most everything that's talked about on the episodes I've listened to and for the most part this one was much the same, except for the one member of the panel insisting that Apple should open source OS X. Let me be one to say that there is absolutely no way that OS X will ever be open sourced. There are so many reasons that this will never happen it's ridiculous. Here's a few off the top of my head:

Traditional computing devices aren't dead yet

The notebook is far from dead. The desktop maybe, but not the notebook. iOS isn't feature rich enough to replace a traditional computer for the vast majority of people. There are lots of places where the OS shines, but for general purpose computing it's not there yet. You can tell that this sentiment is echoed by Apple in the fact that iDevices are treated as satellites that have to sync up with the mothership Mac.

OS X is required for iOS development, why lose out on a sale?

Right now the only platform that you can develop iOS applications on is OS X. The only hardware that you can (legally) run OS X on is a Mac. Apple makes a sizeable amount of revenue off of Mac sales. If they were to open source the platform they'd been losing Mac sales. Why would they want to cut off that revenue stream?

OS X is encumbered

Probably the main reason that Apple will not open source OS X is that it legally can't. Unlike full open source operating systems like Linux, OS X probably contains lots of code that's been licensed from third parties. What code might that be? It could be anything. For an example of this, have a look at OS/2.

Unlike OS X, OS/2 was actually discontinued. Some years later devoted users and businesses that had bought into the operating system wanted it to be open sourced. IBM refused citing licensing problems over some of the code that makes up the OS. This is coming from a company that does a lot of open source work. If they could, I'm sure they would open source it but legally they can't. It's not worth their time or the potential for law suits.

Apple is not going to open source OS X.

Design Matters

It's summer time again, and with summer comes a change in wardrobe. I have two pairs of shorts in my small wardrobe that are extraordinarily similar. They're just about the same color and cut, and fit very well. Where they differ is in the design of their velcro-shut pockets.

Now, I'm not talking about design in the sense of expensive clothing designers. No, no. I'm talking about their general design and the thought that went into it by whoever designed them. Let me explain, as best I can, the general design of the pockets on each of these. Lets call the first pair A and the second pair B.

Pair A was designed as a standard pocket is - a slit on the fabric with a pouch sewn in. The band of velcro is attached right at the top on the inside of the pocket. Pair B on the other hand has a flap that covers the opening of the pocket when it's closed. The soft side velcro strip is attached to the inside of the flap, while the rough side is attached to the outside of the pocket.

Both of these seem fairly straight forward and obvious. But do either of these strike you as bad designs? I can guarantee you that it isn't obvious until you've used this configuration for a little while. Think about it for a minute.

Did you figure it out? That's right! Design A is the bad design. Because the the velco is on the inside of the pocket whenever you try to pull anything out of the pocket your knuckles get scraped on the rough side of the velcro. It's not so bad the first few times but after a while it can wear your knuckles raw. With design B the only the soft side of the velcro will come in contact with your hand.

Now for the $64,000 Question: Which of these designs do you think was on the more expensive pair of shorts? Yeah, that's right. Design A, the scrape-your-knuckles-raw design was more expensive, but that's not what's important. What's important is that design matters. It matters a lot.

When you're designing something you need to really think hard and sweat the details. What you don't think is a problem could turn into real, actual, physical pain for your users.

You're an Artist!

Do you know anyone that refuses to show you what they're working on until it's perfect? I know I do. What drives this person to polish and buff their creation before showing it to the world? I think they do it because they're scared of what people might think. They're scared that someone might ridicule their hard work. To this person I say: don't worry about what other people think. I know it sounds trite, but seriously. If you're too scared to show off your creation then why did you create it?

Putting something out in front of people isn't easy. Everything you create is a piece of art, it says something about you both personally and professionally. It's a very scary thing, but you have to get over that. If you never show people your half-finished project how will you ever know if you're on the right track? I heard an excellent analogy that sums this up very nicely: You can't steer your boat until it leaves the dock.

What's the worst thing that could happen if you show off your creation? If someone tells you what they think that's great! It doesn't matter if it's good feedback or bad. The fact that someone is taking the time to tell you what they think means they have some feelings about what you're doing. And isn't that what you're trying to do? As an artist you should be striving to make people feel something.

What, you don't think you're an artist? Do you know what the definition of artist is?

Artist

n. A person whose creative work shows sensitivity and imagination.

If you've created something from nothing that shows creativity. If it's something that's never existed before, that's imagination. And there's no doubt that a program shows sensitivity. Have you ever used a program or seen some code that you thought was beautiful? I'm sure we all have, that's sensitivity. So, believe it or not but you, a programmer, are an artist. So stop pretending to be something you're not. Create something that people will have feelings about and unleash it on the world.

An Elegant Design: Maildir

One of the most elegant software designs I've seen is that of Dan Bernstein's Maildir. Maildir is a way of storing email messages common to Unix platforms. It's the successor to an earlier format called MBox. These two formats differ in a very big way. To better understand the driving factors behind Maildir it's important to know the flaws that it's trying to address. The flaw in MBox, it turns out, is baked into its design and cannot be fixed without throwing out the design and starting fresh; which is exactly what Dan did with Maildir.

The MBox format keeps all messages in a single text file in the user's home directory, or anywhere for that matter. Each message is delimited by a From line, that is, a line that starts with from:. As you can imagine, this design makes is easy to count the number of messages in the file by counting the number of from lines. Another benefit of this format is that since messagse are appended on to the end of the file all of the messages are in chronological order. These benefits, are not without their drawbacks.

Since MBox stores everything in a single file that file needs to be locked anytime a process wants to write to it. That means that message delivery agents (MDAs) need to lock the file while accepting mail for a user limiting the total number of messages that can be accepted for a user at a given time to one. Locking isn't the only problem that occurs when you store lots of information in a single file. When a message user agent (MUA) wants to remove a message from the middle of the file it has to rewrite the entire file without the message that the user wants to delete. Along the same lines, in order to get a listing of the messages in the mailbox the MUA must scan the entire file from the first byte to the last -- apparently listing and deleting messages were not considered a common use case. Another problem, albeit one that only reveals itself in extreme cases, is that if a mailbox gets extremely large it could potentially run up against the operating system's file size limitations. Lastly, the file format is extremely fragile -- an errant From: line coult corrupt the mailbox and render the mailbox unreadable to MUAs. When designing systems it's important to balance speed, efficiency, robustness, and simplicity; MBox seems to be designed for simplicity at the expense of others.

Clearly the bar is set fairly low for other mailbox formats. Luckily, Maildir wasn't designed as an incremental improvement over MBox, it was designed to maximize speed, efficiency, and simplicity; and it did so in an amazingly elegant and robust way.

Enter Maildir

Maildir, as the name implies stores its data in a directory structure. This is a crutial difference between Maildir and MBox. Maildir, unlike MBox purports to be a lock-free system -- the mailbox can be read and written to at the same time. Under high load this is a really good feature to have. The most interesting piece is that Maildir maintains its lock-free design even when the mailbox is stored on a remote system and mounted using NFS (network file system).

Not only is the system lock-free it also doesn't fall victim to the other problems faced by MBox; receiving and deleting messages are O(1) operations -- meaning that they take a constant amount of time to complete, instead of an amount of time proportional to the amount of messages in the mailbox. Another advantage to storing the mailbox as a directory is that the number of messages can be found by counting the number of files in the mailbox directories -- the MUA doesn't need to open the files at all unless it wants to get information from them.

So how does it work? Maildir is a fairly simple system which has the nice side effect of making it robust. A Maildir mailbox is a directory that contains 3 subdirectories to store messages: tmp, new, and cur. Messages that are being delivered are placed into tmp while they are being received. Once messages have been received they are moved to new before being read. After a user reads a new message it's moved into cur.

So far this seems like a fairly simple system; but what are the files that contain the messages called? Each message is given a unique name. The unique name is made up of a combination of machine and process dependent information such as the host name, current time, PID of the MDA delivering the message, etc. By concatenating a few of these together a globally unique name can be created; in the event that a collision occurs Maildir specifies that the MDA wait at least one second before trying again. This timeout is meant only to ensure that enough time has elapsed that the data used to create another unique name will be different.

That's effectively all there is to the mailbox format. Certainly it's simple, but how is it elegant or robust? To get a handle on that it's important to have an understanding of the properties of the Unix file system.

Unix File Systems 101

When most people think of a file on their computer they percieve a one-to-one mapping between file names and data; that is, a file and its data are synonyms. Under Unix this isn't the case. Unix treats file names and data as two separate things. What you might think of as the data in the file is actually known to the operating system as a unique number called an inode. Files are simply references to an inode. When you create a new file the file system allocates a new inode and associates the data you're saving with it. The name you specified for the file references the inode. When you delete the file you're deleting the reference to the inode. When the file system sees that there are no longer any references to the inode it knows that the space being used by that inode can be reclaimed.

Since multiple names can reference the same inode you can save disk space and headaches by having multiple names for the same inode. The process for creating a new name for an inode is called linking and there are two APIs (application programming interfaces) exposed by POSIX (the portable operating system interface [for unix]) that manage links: link and unlink. As you can gather from the name link creates a new name while unlink removes a name.

Linking and unlinking inodes is an atomic operation -- an operation that occurs instantly without the possibility of being interupted part way through. When a new link for a file is created the file is completely accessible, there's no delay while the data is copied on the disk. Simply put linking amounts to adding some accounting information in the file system but doesn't need to shuffle data around on the disk. This property of atomicity can be used by any application that requires that readers have a consistent view of the world.

Another feature of the Unix file system, and file systems in general is that the metadata -- the data about the files -- is stored separately from the file data. The stat information includes, but is not limited to: creation, modification and access times as well as size and ownership information. Having this information at the file system level means that it's relatively inexpensive to load information about a file or directory. Loading information about a file is called stating a file based on the name of the POSIX function stat. Since this information is standard information stored by the file system and isn't buried within a data file access to this data doesn't require specialized APIs -- the ones exposed by the OS are sufficient.

How Maildir Exploits the Unix File System

The features of the Unix file system mentioned previously are exploited very well by the design of Maildir. In order to ensure currency Maildir stores mail that is being received in a temporary directory, tmp. This ensures that readers of the mailbox, who read messages from the new and cur directories will not see messages that have not been completely received. Once a message is completely received it is linked over to the new directory. At the moment that the file system is updated with the new link to the file in new, all readers of the mailbox can see the message -- and the message they see is the full message, not a partially written one. In a similar vein, the stat information makes is very easy to get the size and received date of each message without having to parse the message file.

These benefits are extremely tangible, especially when considered in concert with the notion that a user's mailbox is generally located on a remote machine mounted using a Networked File System (NFS). Accessing files over a network is an order of magnitude slower than accessing files that are local to a particular machine; however the networked file system is presented to the system as any other file system -- neatly abstracted away behind the standard POSIX calls. From a user's perspective an NFS doesn't look any different than any other file system, it's just slower. But since Maildir is optimized to use the POSIX calls for many common operations, the whole experience is greatly improved.

Consider the case where an MUA wants to get a listing of the number of messages and sort them by date. An MUA that uses MBox would need to first lock the mailbox file, then read and parse its entire contents before it could provide the listing. For large mailboxes this is an expensive operation -- and an order of magnitude more expensive when the mailbox is on a remote machine. The MUA using Maildir would need to simply stat the file system. The data returned doesn't need to be parsed since its already in a standard POSIX structure. To get email specific information from each message such as subject, sender, etc Maildir helps by reducing the amount of I/O that needs to be performed. An MUA can simply read the first few lines of any message it's interested in to parse the headers. Contract this with MBox where it must read the entire mailbox file since the only way to know that you've reached the last message is when you hit the end of the file.

An Example -- How MTAs Accept Mail for Maildir Mailboxes

As mentioned previously the Maildir directory consists of at least 3 subdirectories: tmp, new, and cur. When writing new mail to a Maildir mailbox the tmp and new directories are used.

When a new message arrives, the system constructs a (possibly) unique identifier for the message. The MTA then checks tmp directory for a file with the same name. If a file with that name exists the MTA waits for 1 second and constructs a new unique identifier and tries again until it finds a unique identifer that doesn't clash with an existing file. The 1 second delay allows enough time for any time-related information used in the unique identifier to change.

The MTA then writes the message data to the uniquely named file in the tmp diectory. When the file has been completely recieved a new link is created to the message file in the new directory. The file is then unlinked from the tmp directory leaving a single link to the file in new.

Conclusion

The design of Maildir leverages the facilities provided by POSIX to produce an elegant solution that is simple, efficient and robust. Its clever use of the Unix file system and associated POSIX APIs demonstrates that it's possible to design a robust and scalable inter-process communication (IPC) mechanism using nothing but these facilites. In general, Maildir is a great example of what can be accomplished by having an understanding of the pieces of technology that a system is built on.

The iPad, iPhone OS 4.0 and Dropbox

From the looks of the iPad it's designed to be used primarily as a content consumption device with support for content creation. It's a good position for a first release since consumption doesn't require Apple to work out all of the kinks around how to use this brand new computing device as a way to create content. However, Apple seems to be slightly schizophrenic about the device with the release of the iWork applications for the platform. From what I've been reading the problems with using the device as a content creation platform are numerous. Most of the complaints I've read fall into one of two buckets: the device is designed as a touch screen device and input is fraught with problems, and there's no built in support for syncing work between the iPad and another machine.

Obviously Apple is aware that people aren't going to get any serious work done with a touch screen keyboard, hence the iPad Keyboard Dock and support for Bluetooth keyboards. But allowing users to hook up a keyboard to the device is only a half solution; you still need to interact physically with the device to perform some operations. John Gruber of Daring Fireball sums this up nicely in his article about the iPad:

The iPad is fundamentally a touchscreen device. You absolutely do not need a hardware keyboard for it. But if you re hoping to do any amount of serious writing with it ... you re going to want one. There are a few places in the iPad UI where I really wish the keyboard was useful but it isn t. For example, Safari location field suggestions. ... On the iPad, you must use touch to select from the list. Since you re already typing if you re entering a URL, this is just begging for arrow key support. (Ditto for suggested results from the Google search field in Safari.) The Esc key does not dismiss popovers, but that s probably OK. It s only possible to invoke popovers via touch, so it seems OK that you must dismiss them via touch as well.

As an aside, it's nice that Apple's including the Bluetooth functionality for the iPhone as well. I'm not sure how much use it will be, but it's a nice little feature that other phones won't have.

The other issue that's been brought to light is that the sync story for apps on the iPad, as well as the iPhone and iPod Touch, is really weak. Apple hasn't provided an API to allow apps to sync their data with a Mac. In fact, given that the iWork apps don't sync with the Mac I'm not even convinced that the functionality exists. Sure, you could say that since the iWork apps are smaller, lighter versions of the iWork applications for the Mac they might not want to automatically sync since there isn't feature-set parity between the two suites.

The lack of any syncing support is really surprising to me given that Apple is trying to position the iPad as a light-weight computer. The fact that it wasn't available in the initial release is forgivable but when the iPhone OS 4.0 announcement came and went without any indication of it being in the pipeline I was annoyed.

The lack of a native sync story for 3rd party apps has really turned out to be a ghetto. Each application has its own way of synchronizing data between the sattelite - the iPad, iPhone, iPod Touch - and the mothership - the Mac. Developers are forced to come up with their own way of syncing data between the two devices. Obviously this isn't a good place to be in for developers or users.

From the developer's perspective they have to either write their own custom syncing protocol or use something like WebDAV, FTP or iDisk to transfer the data to location that's accessible to both the Mac and the device. Clearly this is a waste of time on the developer's behalf and it's not even that good of a solution since it has the fatal flaw that the app can only sync data if it's running on the mobile device.

As for users: it's a complete usability nightmare. Each app has to have its syncing configured separately. The app on the mobile device has to be running for the sync to work and potentially so does the sibling application on the Mac. If either isn't running the sync will fail and user's will have to figure out why. This shouldn't be. Syncing is one of those things that users just expect to work. Period.

Synchronizing data is a common enough problem that the platform really should address it in a standard way. The fact that sync support didn't make it into iPhone OS 4.0 but iAds, an advertising API did is just appalling.

So, if you read the title of this post I think you'll know where I'm headed: Dropbox. For the un-initiated, Dropbox is a service that automatically keeps a directory on multiple machines synchronized. Best of all, for small amounts of data - up to 2GB - it's free! Dropbox could provide the infrastructure needed for mobile and desktop devices to keep their data synchronized, a great place to be. The fact that Apple's dropped the ball on such an important piece of the puzzle leaves an opening for Dropbox to become a de-facto standard for synchronizing between mobile and desktop devices.

There's only one slight problem: as of this moment there is no public API available for Dropbox. This means that even if developers want to integrate with Dropbox they can't. From what I've heard an API is in the works, but it's not ready quite yet. All I have to say is that they had better get their act together and soon before Apple comes along and eats their lunch with an integrated solution.

What is Source Code?

Bryan

Hey, I've got a question for you. Can you define source code?

Kyle

Seriously? Source code is source code. What's wrong with you?

Bryan

No, seriously! I'm having a philosophical debate with myself about what source code is and so far I'm losing. I need an outside opinion.

Kyle

Let's see. Hmmmm. Well, I'd say that source code is some human readable form that tells a computer what to do.

Bryan

What do you mean by 'human readable?' Do you mean that it's made of letters and numbers and other things that humans write with?

Kyle

Yeah, human readable, like with words and numbers and stuff.

Bryan

Does it have to make sense to people in order for it to be human readable?'

Kyle

Well, I guess so. To a certain degree anyway. I mean you wouldn't expect just anybody to be able to read it and make sense of it.

Bryan

Ok, so source code is something that's human readable, in the sense that some people may be able to make sense of it, that tells a computer what to do.

Kyle

Yeah, how does that sound?

Bryan

Full of holes!

Kyle

Such as?

Bryan

Well, your definition seems to imply that the human readable form is run directly by a computer. So basically everything that most people think of as 'source code' falls outside of your definition.

Kyle

Like what?

Bryan

How about source code that's run through an interpreter, or a compiler, or a VM!?

Kyle

You know what I mean!

Bryan

Suppose I don't! That's the point of the debate!

Kyle

Alright. Source code is something that's human readable, in the sense that some people may be able to make sense of it, that is processed in some way to produce some output that then tells a computer what to do.

Bryan

That's better. But let's say hypothetically there is a computer that can understand and follow the instructions in the human readable form?

Kyle

I think you're splitting hairs here.

Bryan

Humor me.

Kyle

Fine. Source code is something that's human readable, in the sense that some people may be able to make sense of it, that may be processed in some way to produce some output that then tells a computer what to do, or is interpreted directly by the computer. Satisfied?

Bryan

Not even close. What's this computer you speak of?

Kyle

You're kidding me, right?

Bryan

Nope.

Kyle

A computer, ya know? A thing that runs the source code or the stuff output by the process that processes it.

Bryan

Interesting take on things. Ya know, I seem to recall the word 'computer' has many meanings. Before there were computing machines there were people whose job was to run computations. They were called computers. So far both the machine and the human definition of 'computer' fit your definition of 'source code'.

Kyle

Oh god, what's wrong with you?

Bryan

Pedantry. But I wouldn't say it's something that's wrong with me. Can you define computer for me?

Kyle

...

Kyle

A computer is an electronic device that performs computations.

Bryan

Humans use electrical impulses...

Kyle

ARGH! Fine, an electronic device that you plug in that performs computations.

Bryan

Well, that definition seems rather crude, but I'll accept it just to keep the ball rolling. So what about configuration files? They're human readable and tell a computer what to do. Take emacs for example, its configuration is a bunch of data structures that get interpreted by emacs, well the lisp interpreter part of emacs anyway and are run.

Kyle

What's with you and that stupid editor?

Bryan

What's with you and your stupid editor? What's it called again? 'six?'

Kyle

It's called 'vim' and if you're truely going to be pedantic, VIM in roman numerals is 6000.

Bryan

Whatever, both the fact and question remain.

Kyle

What fact?

Bryan

Your editor sucks.

Kyle

It does not suck, it's awesome. You just don't grok it.

Bryan

Look, why are you arguing with me about text editors? We're supposed to be talking about source code.

Kyle

Very well. I'm not sure that I agree with your assertion that configuration counts as source code. It doesn't tell the computer what to do, it tells a program what to do.

Bryan

That's true, it does tell the program what to do not the computer. But the configuration is still human readable, and processed, and tells the computer what to do. Maybe there's no intermediate form that's produced but I don't see that as being the sole determinant between source code and configuration. If that were the case then source code for an interpreted language would be considered configuration if the interpreter simply walked over the AST.

Kyle

Yeah but, it's configuration. I think you picked a bad example since most people think of lisp as a language, not configuration. What about simple configuration files that are just key-value pairs.

Bryan

I would argue that it could still be considered source code depending, of course, on what's reading and interpreting the configuration. I would say Turing-completeness shouldn't enter the picture at all. Simple configuration or extremely rich configuration doesn't matter. Some people consider HTML and CSS to be source code, but neither are Turing-complete. Conversely what about TeX and PostScript? Both are Turing-complete but nobody considers documents in either to be source code.

Kyle

Look, I'd love to debate with you about what is and isn't source code but I have some work to do that involves writing source code, not just thinking about it.

A Programmer's Toolbox

Every programmer needs to have a scripting language in their toolbox.  The more languages the better, but at least one is a necessity.  For the longest time I've been torn between a few different general purpose scripting languages.  Endlessly debating over the relative strengths and weaknesses of each:  "Python is a fairly ubiquitous, but the libraries aren't consistent."  "Ruby has consistent libraries but there seems to be a steep learning curve before I can be a ninja."  Forgetting, of course that in the end it doesn't matter which one I choose, just so long as it solves a particular problem.

Well, the other day I had just a problem that I needed a scripting language to solve.  I was re-re-reading one of Steve Yegge's terrific blog posts (yes, I've read them a few times).  Specifically it was the one about 10 challenging books every programmer should read.  One of the books he mentioned was Structure and Interpretation of Computer Programs by Harold Abelson and Gerald Sussman.  I'd heard of the book before, and being a fan of lisp I'd always wanted to read it but somehow never got around to it.  I'd even seen the free copy online and the lectures posted on the book's website.  Finally I decided that I'd done enough procrastinating and while I wasn't going to want to read through the book, the lectures might be a good thing to watch when I have some time.  The only problem was I didn't want to have to download each of the 20 torrents and start them up myself.  Clicking on links is for suckers, in case you didn't know.

I needed a way to automate the downloading of all of the torrent files.  I'd heard of a library for Python called Beautiful Soup, but I'd also heard good things about the late _why's Hpricot.  Since I'd already dipped my toes in the Ruby water for a small tool I started working on the other day (more on that another time) I figured I'd try Hpricot.  After about 10 minutes of playing around with the library I had a working script that would scan through the page and download all of the torrents of the .avi versions of the lectures.  Below is the code I wrote, with an explanation to follow.

require 'rubygems'
require 'hpricot'
require 'open-uri'

url = 'http://groups.csail.mit.edu/mac/classes/6.001/abelson-sussman-lectures/'
document = open(url) { |f| Hpricot(f) }

document.search('a') do |a|
  if a[:href] =~ /.avi.torrent/ then
    open(File.basename(a[:href]), 'wb') do |f|
      f.write open(url + a[:href]) { |f| f.read }
    end
  end
end

The above code makes some fairly heavy use of blocks, one of my favourite features of Ruby.  The code is fairly straight forward if you understand that functions and blocks implicitly return the result of their last expression.  Here's a play-by-play of what's going on:

• Lines 1-3: A few libraries are loaded.
• Line 6: The web page containing the links I want to read is loaded and parsed.  Since the block being passed returns the result of its last expression the document is returned.
• Line 8: The document is searched for all of its links.  For each of the elements found the passed block is executed.
• Line 9: Weed out any links that don't target the files I'm looking for.
• Line 10: Open a local file to write the contents of the file pointed to by the link.
• Line 11: Read the target of the link.  The result of the block passed will be written to the local file.

That's it.  15 lines of code and I didn't have to click a single one of the links.  Yes, I probably could have done it faster manually, but by taking the time to write a script this little task is a great saw sharpening exercise and the amount of productivity gained from that is well worth the investment of time.