The Idea Ether

Striving for Civility in the Ruby and Rails Community

The Ruby community is something that I have been proud to be a part of for going well on eight years.  I am one of those people that fell in love with Ruby by reading the first PickAxe book cover to cover...in one weekend.  It was like reading a great novel where the novel ends up coming true.  I joined early in 2001 on RubyTalk and have attended every RubyConf.  I really enjoyed the engaging exchanges with people doing interesting hobbies with Ruby.  

Oh how things have changed!  

I think the change comes because of the fact that with Ruby and Rails popularity we have those that have defined themselves by what they have built with it.  This was never the case before, but it is now.  Now, when I say people have defined themselves, I literally mean that people see their own core value in what they have created, whether it be a service, a framework or a library.  There is nothing wrong is having pride in what you build or are a part of.  The wrong comes when a person actually believes that without this thing that they built, without people praising that thing, that they are nothing.  

That completely saddens me.  

Each of you...each of us...are vastly more valuable that the things we create.  A human being is not defined by what they do, please don't ever fall into that trap.  When someone descends into this way of thinking a common pattern emerges.  When another person criticizes what they have built whether directly or even indirectly they take it personally.  I mean very personally.  Since their foundation is this construct, you shake that foundation and you shake them to the core as a human being.  The criticism is rarely attempting to be an attack on the person, its usually just an intellectual difference of thought.  The next part of the pattern is the creator of said construct attacks back the criticizer in a way that that person does not even remotely understand...sometimes to the point of threat.

This is wrong!

The United States used to be a place where we could have vastly different views, but the conversations were civil.  When civility is thrown out the door, when extremes on varying sides rip apart the fabric of what the vast majority in the middle hold to, its hard to forget that at the end of the day we all need to work together to live and thrive.  I think this happens when people see their party, or an issue, as the foundation of who they are.  Like the person you builds themselves on created work, an idea can form the same trap, and people fall into the same pattern of attack/counter-attack.  Our community, like any community, cannot endure this kind of stress.  We don't want to be like the rest, we should strive to rise above it.

We should demand civility.

I propose that we demand civility in the Ruby and Rails community.  Civility does not mean conformity.  I don't expect people to believe as I do, although I will argue what I believe fervently.  Civility does not mean that we cannot disagree.  Civility is founded upon the notion that we are all people, and as people we have value.  That value is intrinsic in us as people, not on what we have done, created or are a part of.  When ideas we hold, or things we are created are criticized, we must separate ourselves from those things and defend them intellectually.  When someone in the community does attack another personally, we must defend against a lack of civility and speak with a unified voice that attacks on people are wrong.  Some people believe that in a community anything should go.  That is not a community.  Communities have values and borders.  I for one believe that the civility that was there in the beginning should be restored as a central defining quality of the Ruby and Rails community.  I would like to know if my community agrees with me.

Ruby's Best Feature?

In my experience of building systems in Ruby over the last 7 years, I believe there is one main feature of Ruby that made these systems successful.  Before I get to that unique feature of Ruby I want to describe a few of the first systems I built in Ruby and what I learned from them.

The first system I developed in Ruby was a testing infrastructure for a distributed multi-agent system being developed and funded by DARPA.  DARPA is the research arm of the US Department of Defense.  This project's grand challenge was to build a software system that could survive "in theater".  The system had to continue functioning in a war-like situation with enemies attacking the software or the hardware the software is running on.  The main function of the software (beyond surviving) was real-time logistics planning.  Logistics includes routing supplies to people and places where its needed.  That may not sound very interesting but logistics happens to be the key function that makes large groups operate effectively.

What I built for DARPA was a testing framework that observed the events being generated by the software agents using XMPP and then sent commands to these agents using HTTP.  The core of the framework was a Domain Specific Language (DSL) which allowed us to script the multi-agent system.  Initially this scripting was limited to simple things, but that basic automation helped the program significantly.  In the first four months I wrote all the scripts.

During the next year the DSL was rewritten and then expanded to include all the forms of network and system attacks the program was tasked with creating.  What was really cool was the majority of the extensions to the DSL were written by other non-Ruby developers in the program.  The multi-agent system was written in Java, but each of the 80 people contributing to the attack-side of the program ended up learning enough Ruby to encode their attacks.  We also expanded the DSL to generate the logistics rules and from the rules the agent society definition.

This was a large deployment.  We had 300 dual core machines with 4GB of RAM and 10,000 agents running across them.  The Ruby script that was generated for a single agent society definition was over 1 million lines of code.  The events that Ruby handled per hour were well over 3 million messages of 1-10k each.

By the end of the program, Ruby was used to generate the entire agent configuration, drive the simulation, generate the attacks, record the results, generate reports and summarize them for the Program Manager.  This development taught me three very important lessons about Ruby:

1. Ruby is an excellent language to encode domain specific logic in.  Its flexible syntax makes Ruby almost disappear and what's left is domain terms.  Once the domain is encoded its very easy for domain experts to use the Ruby DSL.  These domain experts may not be able to write it, but they most definitely can read and validate it.  When people talk about a 10x developer productivity increase, I believe this is the primary reason for it, the close communication between domain experts and those building the system.
2. Programmers can quickly learn Ruby and can be effective in it, especially if they learn it in the context of something they know.  Every person on our project that wrote Ruby enjoyed it.  Their enjoyment was the main motivator to them becoming more and more proficient in Ruby.  I believe any good programmer can be a good programmer in Ruby.
3. Ruby scales.  We pushed Ruby to unbelievable levels on this program and it exceeded our expectations.  The application under test was critical which made the testing framework even more critical.  There were places in our code where we had to drop into C to perform computationally intensive operations.  The ease of doing this is another key aspect of Ruby's scalability.

The second system I developed in Ruby was a desktop-based functional prototype for a complex system that was being designed for the US Air Force.  The goal of this system was to create a unified view across the 18 different groups that coordinate the movement of military aircraft and provide mid-air refueling with tankers.

The team that hired me to do this was a Human Computer Interface research organization that had spent the prior 3 years traveling the world to interview and capture the requirements of all 18 groups.  They had written a very detailed user interface specification for the system.  They wanted to deliver this user interface specification along with a functional prototype to show the final implementors what they meant in the text & pictures of their design.

The primary challenge was time.  They had less than 2 months left to build the functional prototype after a failed 24 person-month Java effort.  I chose Ruby and Flash to build the prototype in.  I will not focus on the Flash side of what I did because the Ruby side of this, although non-visual, ended up being transformative to the project.

My first meeting with the research scientist (who had become the domain expert) occurred in Boston.  On the flight up, I had no idea where to start with the massive tome of information in the specification they had created.  After arriving in her office we sat down in front of my computer and I brought up a text editor.  I asked her to describe the the concepts in the domain.  As she described it I typed in what I knew was valid Ruby syntax.  As we encoded her domain knowledge we created a DSL on the fly.  After 5 hours of talking and typing we had produced a very elegant 400 line script that she could read as her domain and I could read as code.  On the flight back to DC, I wrote the code behind this syntax to enable the construction of the domain model objects (I called this approach syntax-driven development.)

Over the next 4 weeks we iterated on the domain model.  I wrote scripts to parse flight plans and convert them to DSL code and flushed out more of the domain.  I sent snippets of the DSL to domain experts for review.  What amazed the team was the DSL encoding of the specification exposed numerous examples where the specification was incomplete or contradictory---something that none of them had seen in their reviews of it.

The final 4 weeks were spent developing the Flash-based UI which was driven by the domain objects in Ruby.  In the end we met our goals.  The functional prototype was delivered along with the updated specification.    Although the user interface specification was main product delivered to the implementation team, the Ruby DSL was an important tool they could use to better understand the domain.

So, what did these two projects have in common?  In both projects I used Ruby to implement the system through a language customized to the project domain.  These domain specific languages became effective mediums for team communications.  In my experience, this is the single most important feature of the Ruby language---capturing the domain in a language the entire team can speak. If you believe clear communications is an essential quality of highly effective teams, this feature is something that you should strive to exploit in your own Ruby projects.

First Facebook App (but not the last)

We have been working with Voxant over the last 6 months or so to move them over to a new infrastructure for their Web service TheNewsRoom. TheNewsRoom is a pretty cool service that allows you to find and embed 70+ news sources' news stories into your own Web site/blog and you get a cut of the ad revenue generated. They have LOTS of news sources too in video, text or image formats. The stories are streamed down in Flash.

TheNewsRoom has been resurrected in Ruby/Rails (of course) from is prior short and painful life in Java. Tom, Ryan and Dave have been working mostly on the Ruby/Rails stuff and I have been focusing on creating a very flexible media streaming framework in Flash/Actionscript for them. I would prefer the Ruby coding, but hacking Actionscript is where I am needed right now.

That is until about two weeks ago when Chad (our new CTO) and I decided that it would be cool to get TheNewsRoom to run inside of Facebook using their new app API. Chad and I did a quick weekend development effort and then showed the Voxant folks. They were quite thrilled to see their content streaming inside of a network of 30 million potential users (and growing). They gave me the go-ahead to develop the app in full and last Friday we launched it (and I got to build it in Ruby...and Rails...yeah!). After a weekend of waiting for the Facebook folks to approve the application its now in the directory. Since Facebook does not really have news outside of what your friends are up to, this is a pretty solid news app. Right now its focused on video news but we surface 13 different categories including Business, Entertainment, Health, Life and Leisure, Politics, Science and Tech, Sports, US, World, and a few others including "Odd" (which has some weird stories in it). When you watch a news story you can comment on it and your comment will post on your profile (so your friends will see it in their feed if they have TheNewsRoom installed). There is an additional category of just news items that you and your friends have commented on (News with Comments). Its simple to use and fun to watch and comment together on what's happening in the world. Give it a spin if you are on Facebook. If you are logged in, navigate to this URL: http://www.facebook.com/apps/application.php?id=2406724706. Let me know what you think.

Update: It seems like if you have a Facebook app that you want to link to, you can just use this URL pattern: http://apps.facebook.com/thenewsroom and it will take you either to the app or let you install it. So, go install TheNewsRoom app!

indi Beta 3 Profiles

With Beta 3 of indi and indi Channels we introduced the ability for you to create and share profiles. Your profiles are what you push to others on your Direct and Group channels. Your profiles they are never seen by anyone else (i.e. we cannot see them on our servers because they are always encrypted).

In indi, the profiles that you publish can be edited in the Profile Editor utility. If you select one of these profiles and press the Edit button you will see two multi-line edit boxes. The top box contains your profile template, the bottom box contains the CSS style for that template. Flash allows us to use a limited number of css styles and we added a few of our own. The template side allows basic html tags (again limited by Flash to: a, b, br, font, img, i, li, p, span, textformat and u). The template also includes template entries. Those entries are surrounded by square brackets such as [firstName]. There are currently 30 template entries that you can choose from:

[prefix]	The prefix (Mr., Ms., etc.)
[firstName]	First name
[lastName]	Last name or surname
[suffix]	The suffix (Jr., Sr., etc.)
[workEmail]	Work email address
[homeEmail]	Home or personal email address
[mobilePhone]	Mobile telephone number
[indiAddress]	The indi address
[imAddress]	Fully qualified IM address
[workCompany]	Company name
[workJobTitle]	Work job title or role
[workAddressStreet]	Street number of work address
[workAddressStreet2]	Second line of street number of work address
[workAddressCity]	City name of work address
[workAddressState]	State or province of work address
[workAddressZip]	Zip or postal code of work address
[workAddressCountry]	Country name of work address
[workPhone]	Work telephone number
[workFax]	Work facsimile number
[workWebpage]	The web page (URL) for work
[homeAddressStreet]	Street number of home address
[homeAddressStreet2]	Second line of street number of home address
[homeAddressCity]	City name of home address
[homeAddressState]	State or province of home address
[homeAddressZip]	Zip or postal code of home address
[homeAddressCountry]	Country name of home address
[homePhone]	Home telephone number
[homeFax]	Home facsimile number
[homeWebpage]	The web page (URL) for work
[note]	Note about person

Template entries are merged in from your contact record which you can edit in the Address Book or right inside the Profile Editor (Contact Form button in the upper right). Doing this merge allows you to freeform type in whatever you want in a profile, while your contact data is kept in sync with your contact record. We will be adding more capability to this templating engine in the next beta release scheduled for the Spring. Also, we will be adding the ability to automatically extract these template entries from other's profiles to automatically fill in their contact record. So, in addition to all this sharing madness indi will provide a way to keep others up to date with your latest contact information!

indi Beta 3 is out!

We released indi Beta 3 yesterday.

Whew, its been a huge effort to build a distributed sharing solution (using Amazon's S3) enabling real privacy (using encryption) for consumers (yes, your Mom can use it) that runs across OS X and Windows (using Ruby and Flash) and soon Linux (Adobe released the Flash 9 player today).

indi is free to use and has NO ADS.

The new features in this release are many, and I will follow up with some detail on what we have added. The key thing for now is this: indi is the simplest way (drag & drop) to share with people on the Net. If you are tired of SPAM, get indi. Please provide us feedback...we value that most of all.

Now to get those screencasts recorded...

The Web 3.0 parade has begun

There has been this flurry of posts on Web 3.0 now that Web 2.0 (whatever that was) has mainstreamed. Its always amused me to apply version numbers to something that is changing all the time with new services and sites appearing daily, but that's marketing for ya!

What is really interesting to me in all of this "let's jump to Web 3.0" is that people just assume its going to be a simple jump in versions. In reality, semantics and ontologies are not easy...not easy at all. If you want to catch a glimpse of how the W3C defines the semantic web look at the OWL Web Ontology Language Reference.

Relational databases can be hard for folks to get, but descriptive logics...yeah. Using ontologies can be pretty straightforward but I will say that defining them can be a pain in the rear end. Programmers think that its about class heirarchies (like in programming) but its really not. Individuals (objects), via inference, can enter and exit class membership based on the facts you know about them. Classes have union, compliments and disjoint definitions.

Properties are fully namespaced and those properties have...properties. Think of this...you define a namespace#hasChild property which is the inverse of a namespace#hasParent property. Pretty easy to figure that out. Now is namespace#hasHusband the inverse of #namespace#hasWife? Damn...a gray area (in many people's minds).

This brings me to the main issue that people are going to run into when trying to work with ontologies and semantics on the Web and that is gray areas. We have many of them and the current crop of "social" Web sites allow the human to deal with that gray area (using the nuance of human language). In the world of ontologies systems have to do that, and its very, very hard.

All that said, there is real power to ontologies and semantics captured and used by systems. The key to these systems IMHO is that they are constructed on some real ontological and semantically rich foundations (not just a relational data model), and all that crap is completely hidden from humans. Humans build up and leverage stuff on that rich foundation without ever knowing about it. They may feel its there, but don't have to understand it to use it. An environment that allows this for normal folks (on a continual basis) is one of our goals with indi. I really don't like to jump on bandwaggons but it seems the wagons are coming to us. Its nice when that happens, welcome to the parade.

Web 2.0 and back again

Something had been nagging at the back of my mind for some time now. As I read all these blogs about what Web 2.0 was, I kept remembering I had heard some quote somewhere that describes this...far back in the past. I did some "Google digging" this morning and found what I had remembered. It was this quote

"The Web is moving from being a place where people access information to a place where people access other people in an information-rich environment" - Paul Saffo

Now granted this was quoted from a piece where Paul was being asked about the survival of AOL, but still, what a great quote. What is great is this was captured in a wired article in December of 1996. This is probably the most succinct definition for Web 2.0 that I have seen, and it was written in the early successes Web 1.0. Futurists are great when they are right.

Fingerprint every US worker? NOT!

Slashdot just posted a piece that Bloomberg wants to build a database of every valid US worker to include DNA or fingerprints. This is completely insane, and more importantly its completely unnecessary.

This is insane because centralization of data is a bad idea. Just look at what recently happened with the analyst at the VA who was robbed with his laptop containing 26 million veterans records. Imagine a ~150 million worker database containing biometric data. Crazy insane. But again, its not necessary to build such a database.

I am not against the idea of unique identification via a biometric, but just don't centralize the data. If you want to make sure a person is a valid worker, build a mag card that holds two things:

• Biometric 'signature' of a person's fingerprint stored as a small data file
• Digital signature w/certificate of the biometric signature generated by a 'signature authority' with a valid certificate chain.

That's it. What this would do is create a card that can be self-authenticated without any network access. You have a terminal that takes a fingerprint, produces a signature, then compares it to the one on the card. The digital signature would be validated against the stored biometric data to ensure it has not changed since the authority 'signed' it. This terminal could be completely disconnected from any network and still produce a valid result. No log would have to be generated. No loss of privacy would result. If someone had the card it would be useless without your finger to validate it with. Its just like your signature, just one someone cannot copy without compromising a signature authority. This is not novel, its been done before although there they were trying for much more. The above idea is a simple, cheap way to ensure identity without loss of individual control or privacy and more importantly without the need for centralization.

indi Beta 2 Arrives

We have been busy. Digital Independence is getting closer. We have reached Beta 2 of indi and its been released into the wild. We have decided to invite everyone who signed up on the waiting list to get as broad a feedback as possible, so you all will be getting your invitations over the next few days. Remember, each of you can invite others. So lets spread the word (get indi baby!).

We have lots of new things added in this beta. In Beta 1 we had tagging of files in the File Manager, but that was it. Based on many good comments we decided to promote the tagging feature to be something indi-wide. You can now tag any object in indi including contacts, events, files or notes. You can tag items in the store that may interest you (if you view their details). When you are chatting with someone with indi's built-in IM utility you can even tag the conversations themselves. We will be doing some very cool things in the next release of indi with tagging and presence as well as group sharing using tags. Stay tuned for more on that.

Importing and exporting files now use the OS native file open/save dialog boxes which simplifies the process from Beta 1. Additionally, when you open files now from within indi they are unencrypted for your use on the USB drive, but before you exit indi it cleans the files up, importing them back in if they have changed and cleaning off the drive. This ensures if you lose the drive your data will not be compromised. Speaking of encryption, now all your IM traffic is encrypted point-to-point, so your privacy is maintained (even for multi-player games...we would not want your "Hearts" moves snagged by Carnivore ;-)

That brings us to another feature addition, indi database backups. Now when you select the option to sync with a computer, you can additionally select to back up to that computer. When you enable this indi backs up your user database automatically upon exit. If you lose your USB drive just get another one, re-download and install indi (from our site), and start it up from the computer you last backed up to. The indi will ask if you want to restore the database from the backup. Seven copies of your database are maintained to help in case database corruption occurs and you want to skip one.

The store still has lots of items in it which we plan to build, future plug-ins that will make the indi branch out in all kinds of directions. In Beta 2 you can still vote on these items and we ask you do so because those votes help us focus on what to build first. The store is also now active. We have a few plug-ins in the queue that we hope to release very soon which will be able to be "Purchased and Installed". Of course, you will not be charged in the beta but it will help us flush out this feature. We will be adding some new games, and hopefully some nifty new plugins (photos anyone?) that will be available in the store for download.

We added a patching service in Beta 2. If we identify bugs in this beta that can be resolved quickly we will deliver them as a patch. These patches will be encrypted for security reasons, and you will have to confirm to install them, but their download will be automatic. One of the nice things that "Web sites as software" deliver is continuous fixes as problems or features are identified. We want to leverage patches in the same way for the indi installed and executing from your USB drive.

The indi Helper has been rewritten too for both Windows and OS X. The indi Helper is the utility that you can (optionally) install on a host PC which helps autorun an indi-enabled USB drive when you plug it in. This is a handy tool and prevents you from having to browse down into the file system to launch indi. (If you are a Beta 1 user please remove the indi Helper that you already have installed. This one is way better. See our support page for how to uninstall it.)

In Beta 2 we have added sounds. They are custom and we think they are cool, but please let us know what you think. Lastly, the sync speed for iCal, Address Book and Outlook have very much improved with Beta 2. This makes indi a really great mobile personal information device that seamlessly works with what you have now.

Building cross-platform applications is difficult work, but the three main technologies that make indi possible (in addition to our coding efforts) are Berkeley DB, Flash and Ruby. Thank you Sleepycat (aka Oracle), thank you Macromedia (aka Adobe) and thank you Matz and the Rails team too! And thank everyone who has or will use indi. We hope you enjoy the experience, and let's make it better together. For those that are not in the Beta, ask around for an invitation, or peruse the Getting Started Guide from our support page. If you really want in convince us by joining the waiting list.

A "Free" Market Economy

I was reading a recent post by Fred Wilson on his A VC blog about his favorite business model for Internet companies. Basically his summary was to give a service away for free then charge for a premium or value added service to a subset of those who use the free service. After asking his readers for a name for this model he settled on Freemium.

What struck me by this post was that I have always been leery of the view that a small subset of a customer base actually pay for the rest...but the post brought me to a small epiphany (it may be obvious to everyone else, but it was not to me until now). Let's start with a question:

Q: How do you find and market to a customer base in the abyss of the Internet?

I call the Internet an abyss for obvious reasons. The Internet is a vast space with a massive number of people and a massive number of services and from this space how do you distill your market? What I realized was the free service offering was not there to acquire customers per se but to build a market for your real offering, and a market that you completely dominate. Follow this reasoning.

• You come up with a great idea for a product or service you know some demographic of customer will pay for.
• Instead of traditional demographic profiling, targeted advertising on search engines, etc you come up with another service which would be a draw to a superset of your intended market.
• Put that free service online and through social mechanisms (word of mouth, etc) build up the potential customer pool (as Fred indicated in his post).
• Offer your paid service to the free service pool, knowing that within your free service demographic is your paid service customer base.

So this free service is effectively a mechanism to create an singular advertisement medium where you know (a) your target customers will see the ad for your paid service and (b) you alone advertise in the medium. Rather than look at the free service users as your customers perhaps you should look at them as your potential customer base. The ones that will actually pay you for your product or service offering are your customers. This does not mitigate the cost of having to service those who use the free service. That is critical because your future paying customers are in there! Some may see this only working for services like Flickr where there is a basic (free) then pro (fee) service offering, but it could go way farther than that. Traditional product or service vendors could spend a fraction of their massive advertising budgets to build teams that create these types of free services which would draw in their intended customer demographic creating a medium to reach paying customers, while at the same time offering something that has high value to everyone who uses it. We could name it a free market economy but I think that term is already in use.