Cloudy with a Chance of Data

The over-use, misuse and hijacking of the term cloud computing has been with us for some time now.  It has become the buzzword-du-jour of the technology industry, with ever more specialized variants emerging. With increasing frequency, we hear related terms such as software-as-a-service (SaaS), infrastructure-as-a-service (IaaS) and platform-as-a-service (PaaS).
The emerging data-as-a-service (DaaS) is, perhaps, the most interesting.  The barriers to entry for a startup or new player in this field are, potentially, the lowest of any of the as-a-service sub-genres.  Firstly, unlike IaaS or PaaS, there is no implied customer hardware leasing with this type of service.  Secondly, unlike its elder and more celebrated cousin SaaS, DaaS does not necessarily require providing customer access to software packages (in this case that might be structured or unstructured data management software such as relational database management systems (RDBMS)).  Instead, a compelling DaaS solution need only provide a logical and robust API to a set of data -- potentially stored elsewhere and managed by a different third-party -- served up in a variety of formats for the contemporary web.  Such formats would currently include variants of XML, JSON, and CSV.
When the dust settles, I can conceive that the principles behind DaaS provide the infrastructure for maintaining data synchronization between our devices.  Currently, such synchronization is hidden behind proprietary and opaque processes designed and maintained exclusively by hardware and software vendors such as Apple, Google and Microsoft.
I am mildly optimistic that the industry standards committees will forge a path to empower such innovation. With the expected continued dominance of mobile devices and the relative immaturity of both DaaS and data synchronization, I think I see a big, juicy plum.

So Much Data Yet So Little Time

Today, we are subjected to vast amounts of data. Nowhere is that truer than on the web. So, it is absolutely imperative that, for that fleeting moment while we have our viewer's attention, we present in an informative, appealing and interactive way. The emerging field known as data visualization aims to provide both the software tools and best practices to achieve those goals.
On the lighter side, taking the tried-and-trusted "Six Degrees..." concept, the Oracle of Bacon combines an extremely simple user-interface with the invitation to find two people in Hollywood who are more than 3 "degrees" apart. It is addictive and the simplicity of the website masks the complex algorithms based on the concept of breadth-first search (BFS) and the volume of data, courtesy of IMDB.
Top-notch data visualization is equally effective in presenting serious and sobering information. The University of Victoria, Canada website includes an interactive graphic presentation of catastrophic earthquakes. A quick study of this excellent page helped to identify my own misconceptions (killer earthquakes were relatively equally spread throughout the continents and they occurred with increasing frequency as the decades passed).

The (Hidden) Dangers of Functional Specialization

Like so many things in modern life, software development has become an exercise in absolute functional specialization. That is a single-minded goal of increasing levels of specialization. Economic specialization, whereby an individual worker has a single function, is at the heart of today's global capitalist system. This phenomenon is described in Adam Smith's seminal book The Wealth of Nations.

Economic specialization encourages each contributor to focus on a single goal or very limited set of goals. The payoff is efficiency. Each of us becomes an expert in a single endeavor, enabling a "group of highly optimized individuals" to team together. The proverbial whole is greater than the sum of its parts. There are, of course, darker aspects to this specialization fixation, many of which affect the individual contributors themselves (monotony, lack of motivation, reduced artistic expression, dependency upon others, etc), but it is difficult to argue with the theoretical gains in efficiency of such a system.

Those same efficiencies and "economies of scale" that are a result of specialization must transfer to software development, right?

It has long been argued, in software development circles, that two of the foundational premises upon which absolute functional specialization is based are bogus. The first is that there is some sort of median productivity against which to measure each individual. The second is that by "adding more bodies" we will always reduce the effort of each individual. Each of these assertions has been largely discredited in works such as the Mythical Man Month. This book claims, instead, that the range of productivity amongst individuals is so vast, depending upon expertise and experience, that it is meaningless to try to establish placeholder estimates. Secondly, through a series of case studies at IBM, it shows that the expected "reduction of effort" by continuing to add resources to a team not only fails to materialize but actually has the opposite effect. In other words, there is an optimal size for a team...and it is not infinity! Although written way back in the 1970s, I believe that this book remains highly relevant in today's Web 2.0 world.

As with any engineering project, multiple disciplines must be mastered to ensure quality. For software development, those are typically defined as:

1. Analysis
2. Architecture/Design
3. Development
4. Testing
5. Maintenance
   

Conventional absolute functional specialization wisdom would have us believe that we should employ sub-teams of experts within each of these disciplines. It would also suggest that we decompose those sub-teams further. Perhaps we would create a "Black Box Testing Team" or a "User Experience Design Team"? Further, we might be tempted to create a "Back-End Development Team" or a "Java Development Team", perhaps even before we have defined the architecture? These tendencies toward absolute functional specialization, although logical in the abstract, reveal a subtle difference between creating software and creating something tangible. There is only one "raw material" used by each and every one of the software development disciplines: "words" (either written or verbal).

Think about the raw materials needed to build a house. Here are just a few examples:

• Lumber
• Brick
• Concrete
• Sheet metal
• Stucco
• Paint
  

Given the array of raw materials needed to build a house, it is little wonder that such a project results in teams of sub-contractors (foundation pourers, framers, carpenters, electricians, plumbers, etc). However, software development, in my opinion, is not nearly so complicated as we have made it...and there is just one raw material! Many of the aforementioned software development disciplines are the result of artificial distinctions. Why would someone who can develop software be unable to test it? Is it unreasonable to expect the business analyst to implement his "words" in today's software languages? Why should there be separate "Test Specifications" or "Test Plans" when each is really only a different lens through which to view the Analyst's requirements?

Since the publication of the Mythical Man Month, other factors have emerged (some simply as a result of the maturation of software development industry). These new factors further discredit the notion of absolute functional specialization:
- Democratization/technology mainstreaming: a much broader section of the population is exposed to and comfortable with technology
- Increased demand for software: including "Internet time" releases
- System fragmentation: open-source and other factors have led to a much wider set of tools, technologies and languages that contribute to each "software system solution"

In conclusion, I believe that there is such a thing as "functional over-specialization" in software development.

A better approach, I argue, is to empower a single team to perform all of the necessary disciplines. It is also important to stay current with the disciplines best-practices, allowing you to limit the toolset up-front. (Although common-sense, the "limited toolset" restriction is oftentimes loosened as a compromise that is felt necessary because a sub-team, for example, "only does testing".) Finally, it is critical to enable efficiencies not by defining further functional specialization but by instead enforcing use of a team-wide content management system.

Tune-Up Your Win2K Pro PC

MS Word Document
*** Click link above to view blog entry in MS Word format ***

Securing Your Broadband PC

MS Word Document
*** Click link above to view blog entry in MS Word format ***