The Tension of Modern Software Development - Fast Feature Delivery versus Privacy-Forward Design
In our first article, we discussed the importance of Privacy by Design and making privacy a first-class consideration during our product development efforts. In order to give us the context to explore this further, we first need to look at how software development practices have evolved and the challenges and opportunities these practices present to us as we look to create privacy-forward products.
There was a time when a software development effort was a meticulously planned effort that stretched over 12-18 months with distinct phases such as requirements, design, coding, and testing. This is commonly known as the waterfall development methodology. One important aspect is that the requirements were locked in place up front in the requirements phase, then the design completely put together based on the fixed requirements, and so on. The thought was that we could systematically identify what needed to be done, then schedule those tasks in a sequential and predictable manner.
Some of the defining characteristics of software development efforts of that time were cost and schedule overruns, death marches in the latter half of projects to make up for inaccurate planning, and a tendency for the end-product delivered to not achieve the desired goals for users or the sponsors of the project. The key weaknesses of this approach were planning at the point we know the least about what we are working on (at the beginning of a project), delaying tasks that provide critical feedback until the very last steps of the project (think testing, load testing, scalability testing, legal compliance, etc.), and holding fast to a long-lasting plan even in the face of updated information and new knowledge.
In 2001, the formal formulation of a new approach to software development was made at the Snowbird Ski Resort in Utah by 17 software developers who had found better ways to develop software than the traditional waterfall method. This group crafted a group of preferences through the Manifesto for Agile Software Development and its associated principles that form the conceptual foundation for the many flavors of agile development approaches now practiced.
Agile development teams are cross-functional and highly collaborative, focused on small units of functionality and value, delivered in short increments of time. Cross functional teams are used to avoid the perils of handoffs, lack of customer context, and to increase the adaptability of teams to pivot as new information is learned. Minimizing the amount of work not done is another key concept - agile teams only go to the level of detail on requirements, designs, and analysis that is commensurate with how likely or imminent it is that the team will work on implementing the feature in question. Something that is off in the horizon a bit might only warrant a one-line description of the feature whereas something that the team will be working on in the next iteration will be broken down into small user stories with enough detail as the team determines is necessary for them to successfully complete the story within an iteration. This enables the team to work and deliver value quickly and also creates efficiency compared to the waterfall approach by eliminating the overhead of having to specify detailed requirements and designs for features that may ultimately not be included in the end product or deemed unimportant after continued discussion with customers.
The quick and continuous delivery of small chunks of functionality as working software is a hallmark of agile software development. While this allows for a high level of adaptability and efficiency, this tends to result in a heavy focus on delivering functionality, with large time commitments to coding and testing of the feature within the current iteration (for Scrum and similar time-boxed frameworks) or within the current work in progress set (for Kanban and other non time-boxed work processes). Feature factories can result where the only focus is on delivering more features all the time.
While producing a constant stream of valuable features is a good thing, there are a few important activities that can be displaced in the pursuit of ever-higher levels of feature delivery efficiency.
Technical Architecture - When creating new systems from scratch or when incorporating a major new feature set or leading edge technology platform, a critical activity is thinking about how the system should be organized or refactored to support things like scalability, performance, interoperability, integration, security, etc. In the waterfall methodology, this would take place during the Technical Design phase prior to coding and would be detailed in a technical design document that the team could refer to during the coding phase of the project. However, using an agile approach where we work to deliver smaller chunks of functionality and minimize upfront technical design, there is ambiguity and uncertainty about what is the right time and place to design the system architecture. This calls for different approaches to evolve a system architecture over time rather than statically designing one up front.
Product Discovery / User Experience - Traditionally, most of the work for determining the requirements for a system and the design of the user interface were done up front before coding began. With the adoption of agile methodologies, some of these activities are now done one or two sprints ahead of time (including backlog refinement). While this allows incremental delivery of functionality, it is often done without thinking about the new information learned each iteration, in essence working off a fixed roadmap without consistently revisiting the general customer problem we are trying to solve or our proposed solution in the face of new knowledge gained through incremental delivery of working software, an analysis of customer behavior, and validated or invalidated assumptions critical to the success of our proposed solution. Newer attempts to do both discovery and delivery simultaneously and continuously with each providing feedback to the other is an attempt to expand the effectiveness of our product development investments and efforts.
Scalability / Security / Performance / Data Privacy - These “non-functional” requirements are often not addressed in the initial development of application features which relegates them to being addressed in the latter stages of development. Often viewed as something to do later once we have created the basic functionality these items essentially become a scale-up activity at the end. This is problematic as it delays feedback on these critical dimensions of our application until most of the work has been done, making it expensive and time-consuming to address any required changes or upgrades. Within the Scrum framework, there is the concept of Definition of Done where these could be included as things that have to be incorporated into each chunk that is delivered, but this rarely happens because it seemingly makes it harder to fit delivery of value into an iteration. It also may be seen to violate the principle of maximizing the amount of work not done, as we may learn quickly that customers aren’t using the feature, at which point things like performance and scalability efforts would not actually be needed. This is the trade-off that we’ll explore more later in this series.
Now that we have set the context for how modern software development practices have evolved, we will next explore the privacy regulatory landscape in the United States and how these evolving laws can affect our approach to product development.