Lightsail - My Journey from GoDaddy to AWS

About 9 months ago, I decided to migrate my 3 domains and 2 websites from GoDaddy to AWS.  I had been a loyal customer of GoDaddy for 20+ years, but as more and more of my projects and customers migrated to the cloud, it seemed logical for me to do the same.

I first attended an AWS Summit event about 6
years ago.  I appreciated the seriousness with which each speaker took his/her presentation and the uniform recognition of the scope of Amazon's mission to simplify and improve software delivery.  Having previously been responsible for maintenance of departmental hardware in my role at Sun Microsystems, I immediately recognized the value proposition that cloud computing offered.

Since then, the vast majority of the software projects that I have managed were either predominantly or completely hosted in the cloud, mostly AWS but also Azure and GCP.

For my simple needs, and to control budget, I determined that I would use the following AWS services:

• Lightsail ("Virtual private server for a low, predictable price")
• Route 53 ("A reliable and cost-effective way to route end users to Internet applications")

The Migration

Here is a guide to the migration, with a few comments along the way!

Provision the Lightsail Instance

Login to your AWS account and provision the Lightsail "instance" here:

• https://lightsail.aws.amazon.com/ls/webapp/home/instances

This was relatively simple, especially given my somewhat outdated and incomplete understanding of infrastructure provisioning.  Note that I also had to account for multiple domains.  This video was helpful:

• https://www.youtube.com/watch?v=upZOhKhefAs

After navigating the Lightsail Create Instance wizard, which takes a matter of a couple of minutes at most, you simply input the Public IP (provided for each instance) into the web browser address field and it renders your "base website".  

I particularly appreciated the following:

• The AWS Lightsail web UI provided an SSH console "out-of-the box"
• I did not have to pfutz with encryption keys

Migrate Content

I used the Aptana SFTP client to migrate, en-masse, my htdocs content, as follows:

• Configure SFTP connection from Aptana client to the Public IP for the instance provided through the Lightsail web UI
• You will need the default username and password generated by Lightsail Create Instance wizard (see the previously noted video for details)

I also used this as an opportunity to clean up my legacy web content.  There was (and still is) "tech debt" that needs to be repaid to minimize future maintenance overhead.  Here are some integrations, specific to my content:

• Github's public API: Worked without issue
• Google Maps integration: Worked without issue
• Google Blogger blog feed: Worked without issue

I performed a side-by-side comparison of the functionality of the existing site (GoDaddy) and the migrated site (AWS Lightsail).  This helped identify 2 minor regression issues, that I have noted and still have to troubleshoot in more detail.

Domain Registration

Refer to this guide that is specific to GoDaddy -> Route 53 domain registration migration:

https://blog.shikisoft.com/transferring-domain-registrations-from-godaddy-to-amazon-route-53/

Key Takeaways

Benefits

In my most recent roles, I am less "hands-on-keyboards" than I used to be.  

However, I still find great value and satisfaction in having a basic understanding of the technologies and services at play.  Performing this migration helped me to better understand the AWS ecosystem and, more generally, the operational mental model and nuances of cloud infrastructure when compared to on-premise.  

It will also provide me with a platform should I need to perform first-hand exploration of other emerging technologies and services.

Complications

I achieved "secure http" -- or https -- as part of the migration to AWS Lightsail.  However, this required a little more work than I was expecting.  These additional instructions helped me navigate this:

• https://docs.bitnami.com/aws/how-to/generate-install-lets-encrypt-ssl/

My legacy content was dependent on a MySQL database.  Therefore, I selected the AWS Lightsail Bitnami LAMP stack image.  I encountered some confusion in the database setup, although this helped:

• https://docs.bitnami.com/aws/infrastructure/lamp/get-started/understand-config/
  

I was also migrating a WordPress-based site to AWS Lightsail.  The following instructions helped with this part of the migration (and eliminated the need for me to respond to a series of GoDaddy alerts to perform WordPress updates!):

• https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-using-lets-encrypt-certificates-with-wordpress

Cost Management

Lightsail promises a "low, predictable price".  Amazon also provides this pricing estimator.  For my situation, I also ensured cancellation of my GoDaddy license and disabled auto-renewal.

Of course, I have heard stories about runaway costs and budget challenges given the AWS price structure and, more generally, cloud metered pricing.  As a result, over the months since my migration, I have paid close attention to my charges from either AWS or GoDaddy.  

I am very happy to report, there has been no double-billing or unexpected charges and, moreover, my net monthly hosting expense has been reduced!

Frictionless Software Leadership

Frictionless helps us to make sense of the complexity

Delivering quality software is complex.  This has never been truer than it is today.  Over a long career, I have regarded it my mission to help simplify this process, no matter the role.  While there isn't a one-size-fits-all for every team, product, project and initiative, there are a collection of best practices that have consistently served me well.  I call it Frictionless Software Leadership that draws on the best of Agile, DevOps, Lean, Scrum, Six Sigma, Total Quality Management and even Waterfall.  

Frictionless Software Leadership is guided by principles of integrity, transparency and simplicity.

Please note that I use the expression "delivering quality software" and not "developing quality software".  This distinction helps to underscore the importance of DevOps (infrastructure and deployment), SecOps (security and compliance), UX (usability and experience), QA (quality and testing) and Client Services (tech support and professional services).  Treat each of these as first-class disciplines in teams and projects to help dispel the misguided notion that their considerations can be bolted on after development.

What is Frictionless Software Leadership? 

Frictionless Software Leadership allows us to provide efficient and effective guidance to teams working on software projects, products, and other initiatives.  It helps provide answers to the "Why?  What?  How?  Who?  When?" questions, while ensuring that leadership is not unnecessarily in the critical path or becoming a bottleneck.  

Equally important, Frictionless Software Leadership puts the client (or customer) first.  As W. Edwards Deming asserted: "The consumer is the most important point on the production-line."

There are 4 pillars of Frictionless Software Leadership:

1. Appreciate the (Technical) Art-of-the-Possible
2. Understand the Constraints of the Initiative
3. Drive Planning with Data (Not Documents)
4. Enable Stakeholder Self-Service

I regard these pillars as equally critical and complimentary to one another.  I will dive deeper into what I mean by each of these in a moment.  However, I first want to mention that much of Frictionless Software Leadership has become second nature to me.  That is to say, I do these things almost subconsciously.  This blog post has been a helpful exercise, allowing me to unpack what I have tried in the past and objectively reflect on and document what has worked.  I have also used it as an opportunity to corroborate my first-hand experiences with industry methodologies such as Agile and Lean.

Appreciate the (Technical) Art-of-the-Possible

It is imperative that technical leaders regularly dive deep into the state-of-the-art in tools, frameworks and infrastructure, re-calibrating their leadership approach accordingly.  Recent advancements in our industry -- such as APIs and microservices, containers and orchestration, and, of course, cloud computing -- have fundamentally altered the way that software is architected, deployed, managed, and monitored.  In addition, advancements in no-code, low-code, and BPM/BPA significantly affect the composition of project teams, timelines, and budgets.  It is unthinkable that these fundamental shifts would not have an impact on the way we lead and manage technical teams and projects.

This re-alignment, between our leadership approach and contemporary technology, helps us to:

1. Gain the trust of clients and customers
2. Maintain respect within the technical team
3. More deeply appreciate the business problem(s) being addressed
4. Genuinely appreciate the skill of the technical team, while providing the platform to intelligently probe when necessary
5. Assist the delivery team with prioritization, design, estimation, development, testing, and other hands-on tasks
6. Inquire about vendors' functionality and integration claims
7. Anticipate industry trends*

*For example, 6 years ago, as I was digging deep into the pros and cons of SaaS and PaaS, I identified an at-the-time unmet opportunity to "productize data management".  That notion of DaaS (Data-as-a-Service) -- broadly speaking at least -- is now embodied in products such as Snowflake, Splunk, Mulesoft, New Relic and facilitated by the de-facto industry standard REST APIs.

Incidentally, due to the ubiquity of REST APIs, I would recommend all technical leaders and managers develop a simple web or mobile app to familiarize themselves with the details of a call construct and the returned JSON payload.  At an absolute minimum, familiarity with a tool such as Postman to execute a variety of API calls, provides us with much-needed context and insight into this core architectural style.

Understand the Constraints of the Initiative

Every software project, product or initiative comes with constraints.  In addition to easily quantifiable and understood limitations (for example, the project budget for the year is $200,000), it is important to recognize and document more subtle boundaries before planning begins. 







These constraints loosely fall into 3 buckets (with some examples):

• Organizational 
• Top-down, command-and-control reporting structure?  May not be compatible with some of the team composition ideals noted above.
• Small(er) team size?  Usually preferred.
• All stakeholders identified?  
• Motivations understood?
• Who controls the PM iron triangle parameters?
• Budget?
• Schedule?
• Scope?
• Cultural 
• Predefined thresholds for metrics and standard definitions for Red, Yellow, Green flags?
• Definition of "Done"?  In the context of a feature, sprint, or release, it is critical to get consensus on this "metric".
• Project Begin and Project End?  These are not always as obvious as we might think.
• Client (or customer) engagement?  It is important  to clearly identify ownership and responsibilities around engagement and be sure to properly distinguish project (or initiative or product version) tracking from client (or customer) tracking.
• Are "heroics" encouraged?  Ideally not, since this leads to burnout and is also often an indicator of a single point of failure or knowledge/power hoarding.
• Is product/project secrecy critical?
• Encouragement to "eat our own dog food" and use tools developed in-house?  Ideally yes, since this shortens our internal quality feedback loop. 
• Technical 
• Greenfield** or Brownfield?
• On-Prem or Cloud or Hybrid?
• Integration with legacy or home-grown CRM, ERP or other application?
• Corporate/enterprise mandated tools?

**Counter-intuitively, even so-called Greenfield projects or products will have constraints (mostly Cultural or Organizational).

From my experience, project outcomes are positively influenced by transparent conversations about goals, budgets, schedules/deadlines, and roles/responsibilities before any (meaningful) work begins.  Naturally, it's important for us to regularly review these constraints (and assumptions) to ensure alignment and accuracy.

Drive Planning with Data (Not Documents)

As software managers and leaders, we sometimes mistakenly equate raw effort with progress.  In particular, I have seen the count of project artifacts held up as a meaningful measurement of forward movement.  Instead, I recommend a data-first strategy.  Replace the effort involved in authoring and updating project artifacts with investment in a Project Portfolio Management (PPM) system.  There are some excellent PPM products on the market -- some of which are free -- or we can extend our project management tool (Smartsheet, MS Project, etc) or issue tracker (JIRA, Github, Zendesk, etc) or even develop from scratch.  The solution does not have to be exotic or elaborate; the key is to remove as much of the manual overhead and drudgery associated with project plans, charters, stakeholder analysis, requirements and other project artifacts.  Equally important is ensuring that this data is accessible to all interested parties (see Enable Stakeholder Self-Service, the 4th pillar of Frictionless Software Leadership). 

Depending on policy, there are likely certain project artifacts -- for example, SOWs, contracts, legal papers -- that must be standalone documents.  Make those the exceptions.  From my experience, the efficiency gains in making the transformation to (primarily) data-first are considerable, especially as our number of projects, products and initiatives increase.  This transformation to a data-first strategy should include a clear understanding of our current and future system(s)-of-record and an extensive use of data visualizations, wherever possible.  I have had good success with the following digital visualizations:

1. Calendars***
2. Maps***
3. Timelines
4. Kanban boards
5. Gantts
6. Milestones
7. Value streams and flows (work-, data-, process-)
8. Architecture diagrams
9. Checklists****
10. Cheat-sheets****

***The archetypal visualizations!
****Not visualizations but extremely useful nonetheless.

Enable Stakeholder Self-Service

As technical managers and leaders, we can help bring order to the chaos by establishing a one-stop-shop project portal with optional authentication and authorization, accessible through a single URL via any web browser from any channel.  This will consolidate all the project data and artifacts for convenient access by all stakeholders, ideally through a drill-down dashboard-style interface.  Occasionally, there will be exceptions, but universal access to everything should be the starting goal.  Transparency and simplicity are keys to eliminating project ambiguity and confusion. 

Since we have already made the majority of our project information data-driven, the portal will be real-time.  This helps to eliminate confusion around document revisions and versioning.  However, where there are exceptions and static documents are referenced from the portal, we must be vigilant about concurrency.

It is our job, as leaders, to direct stakeholders to the portal and to encourage input on suggestions for content and format change.  Depending upon the portal's complexity and thoroughness, it also offers us a forum to regularly deep dive into a particular aspect of project tracking.  This helps the team-members to better understand and appreciate disciplines and features in which they might only tangentially be involved.  It also provides an opportunity for stakeholders to showcase their work and achievements or even highlight an area of concern.

Benefits of Frictionless Software Leadership

In summary, Frictionless Software Leadership helps teams to:

• Be more efficient, by...

• Favoring smaller, more focused teams
• Favoring a consolidated toolset
• Understanding communication channels
• Clarifying goals and definitions
• Focusing on DevOps' The First Way ("work should only flow in one direction")
• Empowering adaptation to situational changes (for example, remote collaboration and WFH)
• Be more trusting and transparent, by...
• Accessing a single source-of-truth ("project portal")
• Sharing accomplishments and tips (and frustrations!)
• Collaborating regularly
• Be more content, by...
• Encouraging "pride of workmanship" (one of Deming's bedrocks)
• Reducing frustration (when searching for project relevant information)
• Enabling leaders and managers to spend more time addressing team blockers (instead of attending to busy work updating marginally useful documents)

This blog post documents the ideals and, of course, we recognize that concessions will usually be required.  However, using this approach and applying our much-harder-to-quantify-but-equally-important soft skills, we can have success delivering quality software.

Technical Knockout: How COVID-19 Put Society and Big Tech on the Canvas

It is probably safe to say that few of us anticipated the impact of COVID-19.

The virus has affected every facet of our lives.  Aside from the terrible and tragic human toll, societal norms have been turned upside down.  Social distancing and shelter-in-place have become mandated across the globe.  Board games and jigsaw puzzles, seeming relics of a past era, have enjoyed a renaissance and in some cases are selling in greater numbers than during Christmas.  Sports events, ranging from the Summer Olympics to the NBA to Wimbledon, have been postponed or cancelled entirely.  Schools and universities have been shuttered resulting in teachers, parents and students alike undergoing a trial-by-fire as they struggle with distance learning and homeschooling.  More distressingly, the economy has been harpooned as 10 years of US job gains have been wiped out in the space of just 4 weeks.

Yet, five months on from its initial identification, we still have so many basic questions about this invisible foe:

1. What was the virus's origin?  A "wet market" in Wuhan, China or something more sinister?
2. Exactly how contagious and deadly is this virus?
3. Can we develop immunity to the virus?
4. How different is this year's global and regional mortality rate from the average* and from other key milestone years?

And how has Big Tech, perhaps unreasonably heralded as society's knight in shining armor, fared during this once-in-a-generation crisis?

+ Positives

On the plus side, the sprawling, highly distributed network that we call the internet, has enabled significant work to be performed from homes around the world.  Through Facebook, FaceTime, Zoom, RingCentral, Skype and other videoconferencing and collaboration tools, the internet has also enabled us to connect remotely with loved ones.  Despite the huge additional data loads, we have not -- to my knowledge -- suffered an internet meltdown of any meaningful magnitude.  The free transfer of information provided by the internet, has helped expose and highlight the lunacy of entities such as Harvard receiving a bailout.  Also, data collection and dissemination technology have enabled innumerable dashboards and data visualizations to convey, in excruciating near-real-time, the horror of the disease.  (My particular go-to dashboard has been this one.  Apparently, it was developed and is maintained by a high-school student named Avi Schiffman.)

The fact that we also have a robust physical delivery infrastructure -- with Amazon, UPS and FedEx at the forefront -- has helped to offset the imposed limitations around personal movement and the fear of contagion associated with in-person shopping.  There have been other positive contributions, including monetary donations, job creation, masks from 3D printers and more.

In addition, Apple and Google have partnered to develop a contact tracing API on their respective mobile operating systems.

- Negatives

However, on the other side of the ledger, Silicon Valley's technical might has not, to this point, contributed to the much-needed medical solution of a vaccine.

Furthermore, the fragility of those whose livelihood depends on the oft-trumpeted gig economy has been brutally exposed as Uber and Lyft fleets have largely ground to a halt and AirBnB -- only months removed from being considered the hottest IPO on the planet -- is suffering its own apocalypse.

The pandemic has also exposed, once more, the huge digital divide.

Even the previously mentioned positives are often two-edged swords.

The vast amounts of COVID-19 data available on the internet can leave our heads spinning and are sometimes seemingly contradictory.  Even when the data source is unquestioned -- including my go-to dashboard mentioned above that cites the CDC and WHO among its sources -- other troubling questions emerge such as "what are the precise definitions of the terms ACTIVE, RECOVERED and CRITICAL?"  This can lead to increased FUD (fear, uncertainty and doubt).

There have been significant supply chain and delivery issues that have required arcane "hacks" to get grocery delivery scheduled.

In addition, the contact tracing technology noted in the above article has its critics, mostly based on privacy concerns, and a significant number of devices will not be compatible.

Needless to say, society and big tech are facing a defining moment.  I believe and hope that we can all rise to the occasion by making sustained empathy a cornerstone of our recovery.  This trial has surely illustrated the value of each and every person, irrespective of wealth or status.

* NOTE: It took some sleuthing to address the frequent and troubling objection that the 2020 mortality rate is fairly typical and the COVID-19 numbers are simply manifesting as a "different cause of death".  However, by comparing the CDC's Average Daily Number of Deaths of 7,708 from 2017 (latest data I could find) to a recent 3-day rolling average of US COVID-19 deaths of 2,034 , we can reasonably conclude that 2020 is definitely anomalous.  Furthermore, in the same sample year of 2017 -- influenza and pneumonia which would be the counterparts to COVID-19 in a typical year -- account for only 2% of annual mortality.

My Apple Smartphone is Kinda Dumb

The iPhone is still considered, by most, to be the gold-standard in smartphone usability.  It has been around for 12+ years and my family has owned 4 different models, the latest being the iPhone 8.  I must concede that, aesthetically at least, the device continues to "wow".

However, when it comes to content management (file management, data synchronization and backup), I regard its capabilities as quite primitive, leading me to question how smart we should actually consider the device.  I will discuss specific shortcomings later, but I first want to speculate on why this apparent blind-spot from a company usually so visionary.  I hope you recognize the interdependent nature of these observations:

1. An unnecessarily tight coupling with Apple's own iTunes and iCloud software, each of which has been criticized for its own usability issues.  
2. Apple does not appear to have seriously invested in iPhone file management, data synchronization and back-up strategies beyond its own iCloud service.  This means, effectively, that the company is up-selling its customers additional iCloud storage as the "free tier quota" is quickly consumed. 
3. Since the roll-out of the original iPhone, Apple's primary focus appears to have been on consumer-oriented features such as increased camera/photo resolution, video recording, voice control (Siri), touch ID and improved battery life.  Each of these enhancements play well in advertising campaigns as they are relatively easy to quantify.
4. Although data synchronization as a concept is simple for most of us to grasp, it is deceptively nuanced and complex to implement.  There are offline and online trade-offs around caching for performance and real-time access to systems-of-record that have to be considered as the device switches between WiFi and cellular networks.  Furthermore, each application has its own notion of data attributes, think an iPhone Contacts app record versus a Microsoft Outlook Contact record.
5. Apple corporation's notorious tendency to internally silo products and teams, likely make this type of enterprise-wide effort more difficult to anticipate and execute, even assuming the appropriate corporate stakeholder prioritization is in place.
6. The smartphone class of devices has, in this regard, become a victim of its own success and ubiquity, as it increasingly becomes the hub of our digital lives.  Ever more data management responsibilities -- mostly around our social, entertainment and financial well-being -- are being piled onto a device that was originally conceived as a simpler computer for the masses.

The first two observations are manifestations of the (in my opinion) misguided vision of an Apple-only ecosystem.

While I deeply admire Apple and even many aspects of the iPhone itself, I am underwhelmed and disappointed with some of the iPhone's content management (file/synchronization/backup) capabilities.  Here are some specific problems:

• Inconsistent photo and video synchronization behavior.  I have observed, across iPhone models running different iOS versions, the inability of the device to locally delete uploaded images and videos, even when explicitly instructed to do so.  As a result, the device becomes "gummed up" with copies of images that I considered archived, requiring serious detective work to resolve.
• App organization.  While it has improved over time, the device's ability to organize and locate apps remains quite crude.
• No unique device identifier appended to files.  This could help in a variety of ways during synchronization, upload, backup and organization.  At a minimum, it would help eliminate the possibility of file namespace collisions when multiple devices are archived to the same backup storage.
• Limitations of Apple ID model.  Authentication through an Apple ID governs the features and data available to an iPhone user.  For example, it controls access to data stored in iCloud and purchases made through iTunes.  However, higher level data abstractions, such as an Apple customer's family and device inventory do not appear to be properly accounted for in the underlying architecture.  This results in Apple's customers having to employ unnecessary and clumsy workarounds to share content.

I challenge Apple to think different again.  Below are suggestions on how to make an even smarter phone:

1. Offer an all Apple-ecosystem architecture and content management workflow that you (Apple) are able to simplify and optimize.  However, you must also offer a hybrid ecosystem architecture and content management workflow.  Most of us, in either or both our personal and professional lives, must straddle the Apple and non-Apple digital worlds.
2. Continue (and complete) your evolution away from client-installed applications (e.g., iTunes and iCloud for Windows) towards a fully cloud-based architecture.
3. Revisit the design that regards the Apple ID as the highest level data abstraction.  Consider the notion of a household entity that would allow multiple Apple IDs and devices to be associated, smoothing the opportunities to exchange and synchronize content.   
4. Make your pricing model, for both the Apple-ecosystem architecture and the hybrid ecosystem architecture, simple and transparent.
5. Provide the option for the device to "auto-organize" apps by categorizing using App Store attributes or user-defined tags or properties.

Taming the Business Software Development Life-cycle

In this age of seemingly weekly JavaScript framework releases, countless web APIs, and cloud-based everything, it seems that business software development has lost its pragmatism.  The pace of change – propelled by hyperbole, first-to-market pressures, endless updates to underlying software stacks and insatiable consumer and corporate demands for the “latest and greatest” – is dizzying.    

That this Gold Rush mentality has emerged in an industry whose primary goal is often equated with efficiency, is ironic to say the least.  In this New Wild West, business software line managers and project managers sometimes feel like planning is a luxury that they must do without!

How Did We Get Here?

In its infancy, in the 1980's, software development heavily leveraged the processes and tools that had been used for decades in civil engineering projects.  These were adapted ever-so-slightly for software development to become the Waterfall Methodology.  This methodology (or model) adhered to the basic notions of a budget, a team of resources, a scope and explicitly or implicitly stated quality, and it followed a simple, sequential flow from requirements gathering to maintenance.  The Waterfall (or Traditional) Methodology provided executives and management with assurances that planning and accountability were baked into increasingly greater capital intensive (and hence scrutinized) software development projects.  

What has become clear over the past couple of decades, however, is that in software development (when compared to traditional engineering disciplines) there are "moving parts" that are significantly harder to measure.  Therefore, delivering a project on time and under budget -- the historical yardstick by which project success is measured -- becomes orders of magnitude more difficult when the project involves software development.

The initial response to this realization was for software development teams to increase time spent on analysis and estimation up-front.  Although seemingly logical, this adjustment did not address the unique challenges associated with software development projects.  In particular, it failed to recognize the following:

1. A given software development resource could be orders-of-magnitude more (or less) productive than his/her counterpart.  While there is a broad spectrum of ability in any human endeavor, it is particularly the case in software development.  Fred Brooks's book The Mythical Man Month does an excellent job of making this (and other) critical points related to software engineering.
2. By their very nature -- industry and company specific processes, for example -- business software development projects are unique in the extreme.  This means that it is virtually impossible to even define (Requirements box in the Waterfall model diagram above), let alone build (Implementation box), the optimal solution without multiple iterations.
3. Software development quality, when compared to many other traditional engineering disciplines, is notoriously difficult to measure.

Naturally, without the ability to accurately estimate a resource's productivity (point #1 above), define up-front and in detail the scope of the project (#2), and ensure its quality (#3), it becomes almost futile to try to develop a schedule and budget.

Along Comes Agile

This lack of software development project management success has, of course, been well documented.  As a counterweight to the Waterfall Model a new approach emerged in 2001.  That approach, termed Agile -- backed by its very own Manifesto -- essentially proposed breaking down the silos between the disciplines within software development and repeatedly asking and refining this question: Are we building the right thing?

This philosophical break from the rigidity of the Waterfall Model, significantly reduces the pressure on the up-front phases (Requirements and Design) which, as we have already seen, are almost certain to "miss the target".  It compensates by recognizing that, collectively, the "team" (both business and technical) will iterate towards a solution that will be revealed during regular proof-of-concepts, demos and pilots.  These iterations are typically organized as short "sprints", oftentimes just a week in length.  (To further distance itself from "tradition", Agile introduces its own vocabulary, including "sprint".)

Agile has become so influential that some of its principles are being applied outside of software development, including so-called "Business Agile".

More recently, others have gone even further in proposing the no projects movement.  The movement argues that the project paradigm and its associated vocabulary is oftentimes unrepresentative and misleading and results in harmful practices (for example, the creation of temporary/cross-functional project teams).  The movement asks some interesting questions.

Looking to Industry Trends

In order to simplify and improve the quality of software development, and by extension the management of such work, I believe that we can look to the following industry trends that are moving us in the right direction:

1. Migration to the Cloud.  For all of the hyperbole surrounding it, the en-masse migration to the cloud is a net positive for most business software development teams.  Once the new tools and cost structures are well understood, "the cloud" helps to eliminate many of the challenges of managing and tuning complex IT infrastructure, allowing better focus on solving the at-hand business problem.
2. Service Oriented Architectures and REST API.  The movement away from monolithic applications and suites to "a la carte services" can be challenging, since it introduces "seams" in the software stack that must be "sealed" by the development team.  However, as the approach matures and coalesces around the REST API, it increasingly seems to embody just the right levels of flexibility and robustness.  And, of course, REST is built on, perhaps, the most resilient and well-tested software protocol (HTTP) ever!
3. Open-Source.  The rise of open-source has democratized and leveled the playing field for entrepreneurial technologists and helped to provide a wave of low-cost, high-quality tools, facilitating improved team collaboration, source code management, deployment, system administration and testing.  
4. Browser as Lowest Common Denominator User Interface.  The trend towards the browser as the de-facto UI has been something of a mixed blessing, although it ultimately simplifies software development.  It reduces the number of user "platforms" that must be considered for deployment, testing, etc, and abstracts away many of the details required in designing usability.
5. Improved Integrated Development Environments (IDEs).  Features such as code-completion and linters are becoming standard in IDEs,  helping to reduce many of the syntax and library reference errors that were common in prior generations of these tools. 

What's a Technical Manager to Do?

As we have seen, there is undeniably, and understandably, frustration with the way that software development projects have been managed.  This frustration might be experienced by any of the stakeholders, on either the business or technical side.  Moreover, with the current “developer power mentality" coursing through the veins of tech companies big and small, there is an increasingly vocal line of argument that business process, project tracking and even managers just get in the way

However, it should also be clear that something as complex as software development, warrants and requires significant planning.

Those of us who have been involved in software project management for a while, have probably observed two competing but equally true phenomena:

• No project is exactly the same as another
• All projects have a large degree of commonality

How do we make sense of this apparent contradiction and tame the software development process?

I have come to embrace many of the important philosophical adjustments that Agile encourages.  In particular, frequent and open demos to key stakeholders to level set and calibrate the direction of the development work with the business need.  Simple ticketing systems, such as Jira, and the use of User Stories are also valuable tools.  Short "sprints" (development cycles) and "retrospectives" (process reviews) help to keep the lines of communication open between the technical and business teams.  Perhaps most important of all, Agile helps to break down silos, most notably between the business and technical sub-teams.  Almost as critical, Agile helps to reinforce DevOps and QA as first-class responsibilities of the project, thankfully moving us away from a "throw it over the wall" mentality.

At the same time, Waterfall probably better recognizes some of the realities of business software development projects, such as deadlines and multiple stakeholder communities (both upstream and downstream of the "core customer").  In addition, Waterfall incorporates the Gantt Chart -- still one of the most popular project management "visualizations" available -- and other artifacts that can, with a little creativity, be allied to a generally Agile approach.

Collectively, we might consider this approach to be Hybrid or Pragmatic Agile.

As a supplement to this Pragmatic Agile approach, here is my checklist for technical project and line managers:

[X] Enlist
[X] Communicate
[X] Organize
[X] Simplify
[X] Visualize

Enlist

As Technical Managers, a fundamental first step is building and maintaining a high-performing team.  In general, we should err on the side of smaller (rather than larger) teams and, of course, ensure that roles and responsibilities are clearly established.

Although we don't always have full control over project personnel, we should be on the lookout for the following critical characteristics:

1. Project motivation and interest, on both the technical and business sides
2. Team-orientation and cross-disciplinary interest (planning, development, testing, documenting, etc)
3. Willingness to learn ("agility") and to stay current with industry technologies (see previous paragraph)

These traits apply equally to us as managers, helping to maintain our respect and credibility.

A great reference for building high performing teams is a book titled Beautiful Teams.  

Communicate

It is largely accepted that at least 75% of a project manager's time is spent communicating.  As technical managers, we deliver this communication in a variety of written forms (text messages, emails, documents, presentations, diagrams, etc) and verbally (by phone, in person one-on-one, in meetings and other group situations, etc).  We should take the time to understand the preferred communication format and frequency of each team-member and, where appropriate, adapt accordingly.

When communicating with the team, we should emphasize the importance of the project (to the client).  For example, we might quantify the expected revenue, number of transactions, unique visitors or other metrics associated with the solution.

It probably does not come as a surprise that many software professionals view meetings and writing documentation with skepticism.  Offering options is a way for us to reduce the cynicism, get engagement and facilitate improved quality.  As technical project managers, we may offer the option of remote meeting attendance.  We  may also offer the enticement of a technical deep dive as a segment of every staff or status meeting with a round-robin opportunity for each team-member to both choose a topic and/or present.

Instead of a traditional user guide to support our software development project, we might recognize more enthusiasm within the team for directing and/or starring in a video that "documents" the application's usability.  As a bonus, this might also provide a more unique and superior user experience.

Above all, we should be transparent in our communication, both internally to the team and externally to the client and other project stakeholders.  We should encourage participation in the direction-setting.  As has always been the case, the keys to managing software are communication, planning, and more communication.  There is such a thing as over-communicating, but rarely have I experienced it. 

Organize

As technical managers, we can facilitate much of the communication through logical and consistent organization of project artifacts.  Documents, issue tickets, project charters, team rosters and shared credentials should be easily accessible.

We should strive to review, understand and document, as necessary, the related business processes, working to remove any potential roadblocks that might interfere with smooth and unencumbered project progress.  This includes translation of the business vocabulary into the required technical parlance (development, quality control, operations, support, etc).  At the same time, we need vigilance to ensure that the processes and supporting documentation truly serve only the needs of the project itself, and do not become a means to an end.

By reviewing the project artifacts, we often find duplication of content.  Our goal is to have a single source of truth through consolidation.  From my experience, these redundancies are a very common and time consuming occurrence on software development projects.  They are overlooked or excused because they are targeted specifically for a given project constituency (end user, business analyst, developer, project manager, etc), using that specialization's nomenclature.  As an example, we must clearly determine the level of specificity for each business requirement, and whether we should record them directly as user stories (or similar) in a ticketing system or out-reference tickets to a "specification document".  The key is not to maintain both!

Simplify

Simplification is another key principle that we, as technical managers, need to apply and re-calibrate on a frequent basis.  We can work with sponsors to keep the project goal in focus and reduce the likelihood of scope creep.

We can also simplify by ensuring alignment, as much as is reasonably possible, around development, testing and other tools.  The proliferation of programming languages, dialects, integrated development environments, test frameworks and more is a blessing only when the final tool selections are identified, clearly communicated and enforced throughout the team.  Without that, our teams will be fighting productivity-sapping compatibility and setup issues.

I have recently been working on large eCommerce projects using the Intershop platform.  One of the strengths of this platform is its flexibility and its ability to interface with many other enterprise vendors (such as Microsoft, SAP, Oracle, Salesforce, etc).  However, that ability to customize must be used judiciously.  Fortunately, the Intershop platform also offers a wealth of out-of-the-box functionality that can be implemented with simple configuration changes only.  This, of course, means that you are leveraging tested and resilient functionality.

Bottom-line, promote low code (and similar low customization) approaches within your software development teams, reserving custom coding for the truly critical and differentiating features of your software development project.

Visualize

As technical managers, we also have the opportunity to help stakeholders across the entire spectrum truly understand the project's status.  Every project includes countless informative metrics ranging from budget to resource allocation to estimates and actuals.  We must ensure the data integrity of these metrics and, with tools such as Jira, Trello, Smartsheet and Microsoft Project, we have the capability to track, aggregate and visualize.

We can assemble these metrics into reports and data visualizations, perhaps starting with simple but critical calendar-based visualizations that communicate project milestones, resource availability (personal vacations, company holiday/workshops/team events), meetings, etc.  We can also include clean and simple diagrams, such as this recent one from Amazon, to help convey complex concepts such as application architecture. 

Eventually, we might progress to interactive visualizations provided by business intelligence tools such as Tableau, Qlik, Microsoft Power BI or D3, enabling drill-down dashboards across the entire project portfolio.

A picture really might be worth a thousand words, especially if the team does not have to hear them in another project meeting!

Developing Software Simply

Creating software is, without question, a complicated process.  Daily, I am reminded of this fact even after 25 years in the industry.  Despite having worked in high caliber teams and with many incredibly talented programmers, none was able to produce perfect software.

I still search for ways to improve as a technical manager and when developing personal projects.  Agile Software Development is the latest high profile methodology that describes the processes and principles upon which to develop software.  After reading this interesting article on Agile and its wide-ranging comments, it seems that the jury is still out on the methodology's relative merits.  Reflecting upon my direct exposure to Agile, to varying degrees depending upon the project and my role, there are aspects of it that I appreciate:

• It provides important framework, context, and vocabulary (as, of course, does any methodology) 
• It offers a "clean slate/less-is-more" mentality that has proven to be helpful in many human endeavors (writing, music, art, etc)
• It encourages team empowerment and the flattening and distribution of decision-making

Perhaps above all, Agile emphasizes iterations.  I view this as a two-edged sword.  On the one-hand, it is a recognition of the "perfect software challenge" that we already noted.  On the other hand, it may become a self-fulfilling prophecy that limits quality.

Agile and, indeed, each software development methodology (waterfall, RAD, XP, and RUP) that I have employed -- once again, to varying degrees -- seems to provide an incremental quality uptick.  However, as many pointed out in the comments on the aforementioned Agile article, the relative success of a project remains mostly the result of the skill and experience of the team.  Another big success factor that is mostly unmentioned in software development methodologies, are stakeholder "soft skills", such as collaboration, likability, and motivational expertise.

Furthermore, Agile does not overtly call out many of the root causes of what I call unnecessary seams.  These unnecessary seams are, I believe, the source of additional complexity that we layer onto an already difficult problem.  Seams in the physical world, for example where the shoe upper is sewn to the sole, require extra work and yet typically remain the most vulnerable part of the object.

After learning the hard way, I follow these rules-of-thumb to avoid unnecessary seams:

• Avoid Multiple Overlapping Technologies  The technology in question might be a platform, language, tool, or library.  I heard recently of a project team that used both Git and CVS for source code version control.  Why?  Each of those tools is complicated enough!  Make a decision on one tool and convert the other repository!
• Don't Be Trendy  As in many spheres, humans are drawn to shiny new toys.  Although it is imperative to stay current in our very fast moving industry, we shouldn't feel compelled to always employ the latest platform, language, tool or library.  Stick to the tried and trusted!
• Buy It, Don't Build It  Balance this consideration against the previous two but, in general, use someone else's software when you can.  The benefit of using a pre-existing/pre-tested application, plugin, add-on or service will almost always be more efficient.  Don't code it yourself unless you really have to!
• Assign Appropriate Technical Arbitrator  Ensure that the most broadly knowledgeable individual on the team is the technical arbitrator.  The current trend towards "developer power" sometimes means the wrong team-member calls all the technical shots.  Think big picture when defining your technology stack!
• Recognize Automation Opportunities  Each manual step is an unnecessary seam.  Introduce automation into all of your processes, including quality assurance, builds, deployments, etc.  Automate your entire flow!
• Single Source of Truth  Ensure that you have a clearly communicated single source of truth for each of your project artifacts and data.  Too often, I see milestone dates for the same project managed in a calendar application, in a spreadsheet, and in a database.  Pick one, communicate that choice, and enforce its use!
• Make Artifacts Self-Documenting  Ensure that the names of files, properties, and other "public" project artifacts are self-documenting.  For example, if your application requires a set of 3 roles, don't name them Role 1, Role 2, and Role 3.  Instead, name them something like Administrator, Project Manager, and Developer.  There will be many occasions when stakeholders will need to refer to these roles.  Don't require your stakeholders to perform mental mapping!
• Keep Teams Lean  It is tempting to believe that additional personnel and further specialization on a project will always result in a net gain in productivity.  This has been largely debunked in favor of right-sizing the team to eliminate unnecessary lines of communication, to reduce the consensus building effort, and for more direct ownership (see my previous blog post for more details).  Reduce the number of stakeholders to a minimum!
• Inspire Ownership  Enthusiasm and pride for a product (or project) is mostly the result of a stakeholder's perception of his or her ownership.  The product's quality typically parallels the team's collective enthusiasm.  The signatures of each of the vaunted Apple Mac team were included on the interior of the initial 128K Macintosh case because "artists sign their work".  Encourage creativity with software Easter eggs and the like!