Ruby's Best Feature?

In my experience of building systems in Ruby over the last 7 years, I believe there is one main feature of Ruby that made these systems successful.  Before I get to that unique feature of Ruby I want to describe a few of the first systems I built in Ruby and what I learned from them.

The first system I developed in Ruby was a testing infrastructure for a distributed multi-agent system being developed and funded by DARPA.  DARPA is the research arm of the US Department of Defense.  This project's grand challenge was to build a software system that could survive "in theater".  The system had to continue functioning in a war-like situation with enemies attacking the software or the hardware the software is running on.  The main function of the software (beyond surviving) was real-time logistics planning.  Logistics includes routing supplies to people and places where its needed.  That may not sound very interesting but logistics happens to be the key function that makes large groups operate effectively.

What I built for DARPA was a testing framework that observed the events being generated by the software agents using XMPP and then sent commands to these agents using HTTP.  The core of the framework was a Domain Specific Language (DSL) which allowed us to script the multi-agent system.  Initially this scripting was limited to simple things, but that basic automation helped the program significantly.  In the first four months I wrote all the scripts.

During the next year the DSL was rewritten and then expanded to include all the forms of network and system attacks the program was tasked with creating.  What was really cool was the majority of the extensions to the DSL were written by other non-Ruby developers in the program.  The multi-agent system was written in Java, but each of the 80 people contributing to the attack-side of the program ended up learning enough Ruby to encode their attacks.  We also expanded the DSL to generate the logistics rules and from the rules the agent society definition.

This was a large deployment.  We had 300 dual core machines with 4GB of RAM and 10,000 agents running across them.  The Ruby script that was generated for a single agent society definition was over 1 million lines of code.  The events that Ruby handled per hour were well over 3 million messages of 1-10k each.

By the end of the program, Ruby was used to generate the entire agent configuration, drive the simulation, generate the attacks, record the results, generate reports and summarize them for the Program Manager.  This development taught me three very important lessons about Ruby:

1. Ruby is an excellent language to encode domain specific logic in.  Its flexible syntax makes Ruby almost disappear and what's left is domain terms.  Once the domain is encoded its very easy for domain experts to use the Ruby DSL.  These domain experts may not be able to write it, but they most definitely can read and validate it.  When people talk about a 10x developer productivity increase, I believe this is the primary reason for it, the close communication between domain experts and those building the system.
2. Programmers can quickly learn Ruby and can be effective in it, especially if they learn it in the context of something they know.  Every person on our project that wrote Ruby enjoyed it.  Their enjoyment was the main motivator to them becoming more and more proficient in Ruby.  I believe any good programmer can be a good programmer in Ruby.
3. Ruby scales.  We pushed Ruby to unbelievable levels on this program and it exceeded our expectations.  The application under test was critical which made the testing framework even more critical.  There were places in our code where we had to drop into C to perform computationally intensive operations.  The ease of doing this is another key aspect of Ruby's scalability.

The second system I developed in Ruby was a desktop-based functional prototype for a complex system that was being designed for the US Air Force.  The goal of this system was to create a unified view across the 18 different groups that coordinate the movement of military aircraft and provide mid-air refueling with tankers.

The team that hired me to do this was a Human Computer Interface research organization that had spent the prior 3 years traveling the world to interview and capture the requirements of all 18 groups.  They had written a very detailed user interface specification for the system.  They wanted to deliver this user interface specification along with a functional prototype to show the final implementors what they meant in the text & pictures of their design.

The primary challenge was time.  They had less than 2 months left to build the functional prototype after a failed 24 person-month Java effort.  I chose Ruby and Flash to build the prototype in.  I will not focus on the Flash side of what I did because the Ruby side of this, although non-visual, ended up being transformative to the project.

My first meeting with the research scientist (who had become the domain expert) occurred in Boston.  On the flight up, I had no idea where to start with the massive tome of information in the specification they had created.  After arriving in her office we sat down in front of my computer and I brought up a text editor.  I asked her to describe the the concepts in the domain.  As she described it I typed in what I knew was valid Ruby syntax.  As we encoded her domain knowledge we created a DSL on the fly.  After 5 hours of talking and typing we had produced a very elegant 400 line script that she could read as her domain and I could read as code.  On the flight back to DC, I wrote the code behind this syntax to enable the construction of the domain model objects (I called this approach syntax-driven development.)

Over the next 4 weeks we iterated on the domain model.  I wrote scripts to parse flight plans and convert them to DSL code and flushed out more of the domain.  I sent snippets of the DSL to domain experts for review.  What amazed the team was the DSL encoding of the specification exposed numerous examples where the specification was incomplete or contradictory---something that none of them had seen in their reviews of it.

The final 4 weeks were spent developing the Flash-based UI which was driven by the domain objects in Ruby.  In the end we met our goals.  The functional prototype was delivered along with the updated specification.    Although the user interface specification was main product delivered to the implementation team, the Ruby DSL was an important tool they could use to better understand the domain.

So, what did these two projects have in common?  In both projects I used Ruby to implement the system through a language customized to the project domain.  These domain specific languages became effective mediums for team communications.  In my experience, this is the single most important feature of the Ruby language---capturing the domain in a language the entire team can speak. If you believe clear communications is an essential quality of highly effective teams, this feature is something that you should strive to exploit in your own Ruby projects.

Comments

[Guy Hurst]

Thanks for sharing your experiences.

Nice to see you still plugging away with Ruby, and in such a meaningful way. :)

[Stephan Schmidt]

Ruby did scale because you did use C? Didn't C scale then and Ruby not?

I'm a little confused.

Peace
-stephan

[LeoHorie]

>> Ruby scales

>>Ruby did scale because you did use C? Didn't C scale then and Ruby not?

Languages don't scale. Applications do.

Anyways, being able to write in a domain-specific language subset is definitely nice. I think that's why libraries like Jquery are popular.

[Tien Dung]

Great post!

It's make me want to jump in Martin Fowler's Domain Specific Languages Immediately.

http://www.martinfowler.com/dslwip/

[Paul Drummond]

This is really interesting.

Any chance you could post some examples of the DSL's you created?

[4ga12nkhgvp3eutlx8e0]

http://gogle.com google

[Phiala]

People should read this.

[Understanding PPC]

I just came across your blog about and wanted to drop you a note telling you how impressed I was with the information you have posted here. I have a site Understanding PPC http://understanding-ppc.110mb.com so I know what I'm talking about when I say your site is top-notch! Keep up the great work, you are providing a great resource on the Internet here!