Networking & DevOps: Getting Back to Our Roots !

I spent last week at the Open Networking Summit and Open Networking Foundation Member Workday and came home with a head full of ideas to write about.   Here's my first set....

At ONS I heard the term DevOps mentioned many times.  In fact, Stacy Higginbotham from GigaOM (who btw was an excellent moderator for the panel I participated in) wrote a post about it as well (See "Does networking need a devops movement").  Perhaps it's because I live in the corn fields of Indiana or because I really live in my own little world most of the time, but I actually had to Wikipedia the term DevOps (Can you use Wikipedia as a verb ?).   In any case, when I read the Wikipedia article I thought, DUH OF COURSE !!  

You see, this is how we've been developing network management software at the GlobalNOC at Indiana University for the past 12 years.  For those of you who don't know us, the GlobalNOC partners with universities and non-profits to help them manage their networks by providing NOC and network engineering support.  Very early on, as we grew from working with 1 network to 3 or 4 networks,  it became painfully obvious that we would need very good, highly customized set of software to help us manage these very diverse networks.

This started out with a few network engineers (like myself) who had CS backgrounds hacking together some open-source software with way too much homegrown Perl scripting for our own good.  We wrote the software, we supported the software and we used the software to manage networks !  Over time, as the team grew and the software became much more sophisticated, it was necessary to split the developer/sysadmin team from the network engineering team, but they are still very closely linked.

I'm no longer directly involved with the developer/sysadmin team (SYSENG as we call them), but the parts of the development process described in the DevOps article on Wikipedia, such as reduced release scope with smaller changes more often and automation of deployment, sounds very familiar :)  I'm told we rolled out something like 40 software releases into production last year.  In addition, the "users" of the software, ie the network operators and engineers, have a very close link to the developers/sysadmins making it easy to exchange ideas for new features and to track down bugs.

One of the big reasons I've been excited about SDN is the possibility that it could be used to apply these same DevOps principles to the development of control-plane software for networks.  If I'm not mistaken, this is how IP networking started in the first place (or so I'm told) !  I'm certainly not old enough to have been around during the early days of the Internet.  However, I was very fortunate to start my career with a company called ANS, which operated the NSFnet, and had the privilege of working with many of the people who were.

From what I've heard, the early IP networks were run by computer scientists who both operated the network and wrote the software that powered them.  The process of getting new features into the software didn't involve hundreds (or thousands) of network engineers submitting ideas for new features to sales reps, product managers trying to distill these requests into the actual features that will go into the software, developers building the software, QA teams testing the software, etc - only to have the QA teams at the customer sites do their own extensive QA tests to verify the product works properly in their unique environments - before the feature could be put into production.

Now, I don't know for sure, but I suspect the process "in the good ole days" led to a few more bumps in the road then would be acceptable in today's production networks.  However, I think SDN at least (re)opens the possibility of a tighter "loop" between the people who manage networks and the people who build the software the powers them and IMO that would be a good thing !  

At the Open Networking Summit last week, the Indiana University team demo'd our first example of SDN software developed using this methodology and it's already deployed on a nationwide, production backbone network called NDDI.  We'll be demo'ing our next piece of SDN software at GEC12 next week which solves a specific issue on campus LAN.  Our plan is to have it running in production in the Indiana University network by the end of November.  From the internal demo session this morning, it looks like were very much on track to make that happen.

In the GigaOM article I referenced earlier, the question was posed as to "where in the networking world these developers would come from".   Well, Stacy, we're growing them today, right here in the corn fields of Indiana !!