• Quicksilver — first thing I install on a fresh Mac. Launch apps, browse through address book, send files from Finder…
• iTerm — version 2 adds split panes in full-screen, which is just awesome.
• zsh with oh-my-zsh — of course. You’ll need git for oh-my-zsh, so see below…
• Homebrew — the package management system of choice.
• MacPorts — a more traditional package management system for OS X, I still keep it around since not everything I need is on Homebrew.
• Command-line Tools for Xcode — If you only need Xcode to compile ports, register a developer ID with Apple and download this 171M package instead of a full-blown multi-gig Xcode.
• Sublime Text — switched to v2 from TextMate, never came back.
• Solarized — this is what I put on top of my iTerm and Sublime Text.
• Inconsolata-dz — this is what I put on top of my Solarized.
• Divvy — Even though this window manager went up 2x in price after making it to App Store, it’s still well worth it.

There’s also an extended list of stuff that I use at my i use this page.

PS I absolutely forgot about about GNU desktop environment! Unless you a hard-core FreeBSD fan, don’t leave $HOME without it. port install coreutils findutils gives you same utilities and switches you would expect from a modern Linux distro. See coreutils and findutils pages for details on what you get.

I forgot where this came from… Boundary? Anyways, if you are going to require verification, it’s a good etiquette to notify the subscriber in advance.

Meraki again.

Actually, Pivotal has a similar feature.

When creating an additional account in Pivotal Tracker, ‘Account’ changes to ‘Accounts’.

It occurs to me that the situation in the infrastructure-as-a-service space today is somewhat similar to the beginning of the microprocessor era. Processors are designed for the end user (programmer), provide a rich variety of functions and hide a ton of machinery behind invocation of simple commands (think about all of the complexity that is required to deal with binary-coded decimals, for instance). Because there’s so much going on behind the scenes, it’s hard to optimize the code that is running on the processor, and sophisticated programs require custom hardware to run efficiently.

Things changed when RISC processors entered the market. They were designed to serve compilers, not programmers and hence allowed for much more freedom of expression with less primitives as opposed to first-generation microprocessors. On one hand, it was much easier for the complier to generate optimized code. On the other hand, writing a large program for the microprocessor by hand, without any help from a higher-level compiler became next to impossible.

Advent of high-level programming languages and RISC architectures led to decoupling of the language roadmap and infrastructure roadmap, allowing innovation to happen independently on each layer of the stack. Transition from CISC to RISC architectures is sometimes cited as a pre-requisite for Moore’s law to hold.

The idea should be pretty clear by now: instead of designing APIs for the programmers, design them for the compilers. Expose internals of the platform. If your platform is successful, people will try to do weird stuff with it — stuff you can’t even imagine while designing the API. If there’s a need for a more high-level API, build it on top of the low-level, high-granular ones. (Actually, more and more programmers today actually use high-level abstraction libraries like libcloud or jclouds instead of calling your API directly.)

So the question is — if “RISCy” API to your cloud provider is so appealing, both for the provider and a consumer, why don’t we see those kind of offerings on the market?

One explanation would be that the offerings are just not mature enough to be able to plan ahead. Most of the infrastructure-as-a-service providers today came from hosting business and their APIs tend to reflect their UIs, despite that the user persona has changed dramatically. Even the flagship of infrastructure-as-a-service, Amazon AWS, was not on the market for long, and even three years ago the applications that run on AWS [were just not sophisticated enough] to take full advantage of the fluid infrastructure. However, there’s different (and more profound) explanation out there.

When I described this thought to a colleague of mine, who actually happened to witness the rise of RISC first-hand, he was quick to point out that RISC could not have happened until there was a trusted compiler from a high-level language. In a sense, there’s no sense in providing low-level APIs unless there’s no popular accepted tool to take advantage of them.

Is this a problem of chicken and egg? Probably not. A successful compiler(s) could very well target providers of today and take advantage of the next-gen APIs when they arrive. For portability sake, there might be a need for a common API that can be extended with vendor-specific features (think i386 architecture and extensions made by Intel and IBM). The two most likely contenders for this abstraction layer today could be AWS API or OpenStack API; neither, however, are low-level enough to become the foundation for the next-gen offerings.

It’s pretty clear that there are bits and pieces of the stack that are available today. In the later posts I’ll try to explore the relationship between processors, high-level language compilers and operating systems and how they map to cloud landscape — what’s there already and what’s still missing.

At Grid Dynamics, we often help our customers building products that can be broadly categorized as software- or platform-as-a-service offerings. One of the key design decisions that has to be addressed early on is how the underlying infrastructure should look like.

The mainstream solution is to use captive hardware — rent a datacenter, fill it with blades, connect blades to a SAN, install F5s on the aggregation layer — there, you have it. Straightforward, but not simple, and nowhere near being cheap.

However, with the big promises of infrastructure-as-a-service providers, it is tempting to rent cycles by the hour instead and not bother with the issues of capacity management, amortisation, and hardware failures. The economics driver behind this is huge. Even for an infrastructure powehouse like eBay, offsetting some of the infrastrcture costs to an external vendor can results in non-trivial savings. For smaller folks, it is even more relevant because of the smaller capital expenditures as opposed to invesing in own infrastructure. Naturally, one of the first things that happens in the beginning of a new project is evaluation of the infrastructure-as-a-service offerings and their applicability to the problem at hand.

When the product that is being built is relatively simple, infrastructure-as-a-service is usually a good fit. However, more sophisticated software requires more transparency and control over the infrastructrure level than the current offerings provide.

Here are a few examples of the infrastructure-level concerns that need to be addressed by a sophisticated application:

• Workload placement. More often than not, software-as-a-service products need to be multi-tenant. When the software needs to handle financial transactions, PCI regulations come into effect. One of requirements is for the financial data belonging to different customers to be processed on different host machines. Without a tool to control placement of virtual machines, there’s no way to move financial transactions outside of captive hardware.

• Network management. Similar to the one above, segregating traffic between different customers is imperative to satisfy PCI regulations. While most of the infrastructure-as-a-service offerings include tools to manipulate firewall rules, working directly with network interfaces, ports and virtual LANs is still an exotic and poorly understood feature.

• Access and usage control. As it stands today, most of the infrastructure-as-a-service offerings are just not built with application multi-tenancy in mind. It is up to the application to either sub-divide the credentials that are issued by the provider (and risk contamination of all of the customer accounts if the superuser credentials are compromised) or create and maintain a number of provider credentials, one per each customer accounts. Same dilemma has to be solved for the application to be able to track usage per customer account.

• Failure response. While it is common for the infrastructure-as-a-service offerings to provide a programmatic way to provision additional infrastructure resources, providing a programmatic way to access failure conditions is not common. Thus, the promise of self-healing software is not fulfilled: the software could provision additional resources to compensate for the failure condition, but to detect the condition it requires the level of transparency that the infrastructure offering does not provide.

• APIs. Every infrastructure-as-a-service provider has APIs, right? Unfortunately, these APIs often do not cover more than basic resource provisioning. Usage, billing, access control, monitoring are often designed to be consumed only by a human operator, usually through a browser-drven GUI. Pretty much every sophisticated application that relied on infrastructure-as-a-service resorted to parsing GUI screens or calling to undocumented URLs to support some of its critical functions.

Usually these concerns are addressed by business arrangements between the application owner and infrastructure provider. This, however, affects portability and for the smaller shops a custom arrangement may just not be a possibility.

Is there a technology solution that could address these concerns?

This is part one of the two-series installment.

Finally configured Disqus, so the posts are now commentable.

It’s been three months since the last update, but I have a two-part post on SaaS/IaaS impedance mismatch in the queue, so stay tuned.

My MBP has two hard drives: one is a small 128G SSD, and another is a much bigger HDD. Naturally, I need both for Windows that I run in the dual-boot mode.

Supporting two hard disks in Boot Camp is straightforward; Windows was able to identify my drives so I just mapped the HDD as /opt with the device manager. However, I wanted the same experience within VMWare Fusion. Adding a raw partition as a disk is not something that Fusion exposes through UI.

After tinkering and googling around a bit I came up with the following recipe:

$ diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *320.1 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:                  Apple_HFS opt                     159.9 GB   disk0s2
   3:       Microsoft Basic Data opt                     159.9 GB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *120.0 GB   disk1
   1:                        EFI                         209.7 MB   disk1s1
   2:                  Apple_HFS Vertex                  98.5 GB    disk1s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk1s3
   4:       Microsoft Basic Data 7                       20.7 GB    disk1s4

disk0s3 is obviously what I need to mount. Need to create a VMDK for that:

$ cd ~/Library/Application\ Support/VMware\ Fusion/Virtual\ Machines/Boot\ Camp/Boot\ Camp.vmwarevm
$ /Library/Application\ Support/VMware\ Fusion/vmware-rawdiskCreator create /dev/disk0 3 opt ide

And bind it to the VM by adding the following into the VMX file: $ mate Boot\ Camp.vmx

ide1:0.present = "TRUE"
ide1:0.fileName = "opt.vmdk"
ide1:0.deviceType = "rawDisk"

(In my case, I had to remove existing CD-ROM definition on ide1:0 that Fusion created for me.)

Since the IDE channel and partition ID is the same, I did not even need to remount the drive in Windows, it picked that up by itself.

I may also want to cover advanced networking at some point. It is amazing how much stuff can be done under the Fusion’s hood.

Install on Quit — This is the first time I see such an option, and it is seriously awesome.

When I talked about the big black boxes earlier, I mentioned that we learned pretty well how to sell and buy them and why selling technology is different. What about buying technology?

If we look back, business machines such as Burroughs arithmometers were literally boxes on cast iron legs. Those were probably served by a technician who was responsible for data input and output, and a support team from the vendor that was responsible for changing oil and replacing bearings. This all makes sense — after all, arithmometer is a useful box, but still just yet another business tool like a fax machine or a pencil sharpener.

Time passed; we replaced arithmometers with mainframes, minicomputers and arrays of blade servers; punch cards with magnetic tape and disk arrays; telegraph lines with copper and optic fiber. The approach, though, has hardly changed. Buy a database box and create an IT group around it that will input and output the data. Buy a web server box and create another IT group around it. Buy a build server box… rinse, repeat. After all, why would we treat those boxes in a different fashion than fax machines?

The problem is that when the boxes need to evolve to support the growing needs of business, innovation stops at the organization boundary. This is exemplified by Tom West’s experience of “seeing Digital’s organization chart in the design of the product”.

Change happens when people start to treat the technology as manufacturing equipment, rather than infrastructure plumbing. This is how technology companies see the world differently. This is when, against all business books, technology becomes a competitive differentiator.

GoGrid is arguing whether we should say that something is running in or on the cloud.

Of course it’s in the cloud. For me, it has nothing to do with grammar: when you’re in the cloud, your part of it, while when you’re on, you’re like a cupid hoping that his platform is solid enough to withstand his increasing weight.

It is interesting how the big self-contained boxes of software slowly give way to more modular, flexible, library-ish solutions.

We had monolithic servers — now we build boxes from commodity parts and pair Linux kernel with a bunch of packages. We had Apache — now we embed our web servers within the application. We had Oracle — now we handle transactions within app tier. We had hardware network devices — now we put offload cards in our servers and make them a part of the app.

Probably the two most recent examples are:

• Relying on the database less. Do I actually need ACID? Do I actually need relational semantics? Hence MapReduce, NOSQL, event sourcing, etc.
• Decentralized message routing. Do I need “once and only once” delivery? Do I need queuing? Case in point would be 0mq vs traditional message brokers.

Why is this happening?

My guess — most of this technologies came from the enterprise. This is understandable : enterprise vendors know how to sell boxes, enterprises know how to buy boxes. Much less so with the components.

Then there comes a technology company and stretches the enterprise technology and thinking in the direction it was not supposed to stretch. Monolithic concepts rip apart, releasing a plethora of knowledge and components that are then polished to the point of becoming reusable.

I am curious to see if this trend continues. Most people still think in the terms of big boxes and until we learn to sell and buy something other than boxes, there’s no incentive to otherwise.

On the other hand — isn’t that what cloud is supposed to be? An anti-box?

Over the last month I’ve been asking different people to help me with market research. We would spend an hour or two over a cup of coffee, me listening frantically taking notes.

It was a first experience of that kind for me, and I was actually pretty scared asking the guys that I did not know too well to spend time helping me with a personal thing. What surprised me is how many people gladly agreed to do so and how willing they were to talk about their needs and pains.

When I got back to my notes and analyzed the data, it suddenly came to me that during that month I got more insight into the industry that I had for maybe the whole last year combined.

Whenever you ever have a chance to talk to people, ask them what are they working on and what are their pains are. You will be amazed how much data and insight is there, stuff that is not in the books, delivered first-hand and with passion.

Why do we compare software systems to automobiles?

Cars contain thousands of details. They are produced en masse, according to the blueprints that are developed over years. Changes are expensive. It could be appropriate for the software circa 1970, but not for the modern systems — and it is definitely not the best metaphor to keep in mind while designing one.

If anything, modern software system is a car factory. It has to manage resources and data, employ external contactors, adapt to changes in the blueprint, optimize itself — all for a single goal, stamping out new shiny cars that the customers will happily drive away.

Why do we still build software as if we were building cars?

Have stumbled across this post on LtU today.

You know, I am writing this post using Markdown, a nice markup language created for the purposes of formatting plain-text writings. Using Markdown, I can create hyperlinks, lists, insert images etc. HTML can be use to achieve the same purpose, but Markdown is better for several reasons:

• It is not as bulky as HTML when you write;
• It is not as painful as HTML when you read what you’ve written;
• It favors convention over configuration,
• It can be compiled into something else then HTML — for instance, PDF.

Usually, we draw a solid line between programming languages and markup languages. However, markup is usually present in the code written in the programming language as well. Consider the symbol documentation, or comments. Not surprisingly, I do not use Markdown for that purposes. Reason? Markdown knows nothing about my code. I can not insert a meaningful link to a symbol, for instance, and hope for some tool to update the dependencies when I decide to rename that symbol later. On the other hand, Javadoc and what-do-you-call-it documentation system from MSFT both do that.

How about other documentation? I believe that keeping the docs alongside with the code is a good practice. Expose it over the web and you can use your wiki a lot less frequently. However, there are issues with that. Suppose I am creating an architecture definition document. I’d love to link from the doc to the module in the source tree. Markdown can not help me here, and markup languages akin to Javadoc do not really do a good job when you try to use it for standalone documentation. Moreover, none of the above is very extensible, so if I want to embed a umlgraph diagram into my doc… well, you better shoot me in the head right now.

So how Scribble handles that? Well, a document written in Scribble is essentially a module in your system, and it behaves as such. If you try to mention some symbol in your doc, it has to be imported from the relevant module — otherwise, you’ll get an error. You can import and use any functions you want, from your modules or from third-party libraries. (Guess you can do the reverse trick and link to documentation chapter from the module, too.) Scribble can be used as a stand-alone tool or in-place, in form of a symbol documentation or code comments.

When you compile the Scribble document, it essentially runs the program it denotes and provides you with the output. If you ever worked with TeX, you know the drill.

Where this approach can be useful besides creating system documentation? One other use case that immediately comes to my mind is test execution. One can create a set of test cases in Scribble, and the output will be a formatted execution report, always up-to-date.

So why won’t we take it and use on a daily basis? Well, I guess you could, if your project is written in PLT Scheme. I see no particular reason, however, why this approach can not be adapted to other languages. The easiest thing to do, I guess, would be to adapt it for Clojure. Its native metadata handling make the problem especially interesting.

If you are interested, I strongly suggest to read the original paper, it’s only a dozen pages long.