UP4edu, Part 1
All of the modern Agile literature I have read is written around the assumption of software production in a business model not completely compatible with the way I work. UP is particularly, though unnecessarily, grounded in the values of big business. My job is IT support and software development for an educational institution, and the factors driving development are significantly different in subtle ways.
The types of projects I work on conflict with Unified Process approaches in several ways. Inception can't always be a drawn out process. Despite the claims that OpenUp is appropriate for "small teams" it is unwieldy compared to Scrum. Part of the issue lies in the particular way UP nests iterations. OpenUp has a certain prescription with certain minimums that lack flexibility.
A major qualm I have with Agile processes in general is the obsessive fixation on iteration. Must one do everything repeatedly when a single time is truly sufficient? This is akin to XP's prescription of always only programming in pairs. The practice of Pair Programming is often useful, but not ideal. It is a good idea to double-check one's work, and for important documents making multiple drafts is prudent. Some tasks however are casual and mundane enough that no second thought is required. There is a fine line between the notion of "trivial work" and falling into the Waterfall trap. One main flaw in the Waterfall Method is that it relies on creating a chain of contingencies. The point of Iteration is to encapsulate the activities of each such that one Iteration does not hinge on another in ways that are hard to orchestrate, or inflexible to deviation.
Defining a Project, Interation One
One important, though often under emphasized, concept from Unified Process is the project configuration. Beyond the way UP defines "configuration" there is a more broad application for the same idea. UP4edu (1.0) is a first attempt to generalize in a meaningful way the UP process to Educational Institution Software Projects, without unnecessarily limiting application to institutions that are educational in nature or projects that develop software.
Thinking metaphorically about how my work environment is defined, consider a set of Objects modeling a Solution Development problem domain:
A Project is a single Product over the course of one or more Deliverables (a deliverable Iteration of the Product). The project has one or more Customers. These customers are Agents that have Needs. Agents may be People or Systems. The success of the project is measured in how well the product meets the needs of the customers. The needs change over time, and to keep pace the product changes over time as well.
An iteration is a period of time where the product of a project is changed to better meet needs. Iterations may be deliverable, or not, variably. Any iteration may be made deliverable if the needs that it satisfies warrant the cost of delivery. Similarly any iteration's delivery may be canceled in the face of changing needs. A project may be comprised of only deliverable iterations, or many undelivered iterations leading to a single deliverable, or any combinations of undelivered and deliverable Iterations. Every iteration results in a different Version of the product, deliverable or otherwise.
A new project addresses an initial set of needs, each new iteration addresses new or changing needs. The project has a Vision to make the overall goal across iterations clear. This vision may evolve, but is generally fairly static and long term.
A project is managed by exactly one Project Lead, a Manager. It is also managed by exactly one Iteration Lead, a Developer. Projects have one or more developers with time to devote to making the needed changes to the product each Iteration. In the most trivial case, the project lead and the iteration lead may be the same person, and they may be the only developer on the project.
The project manager's responsibility is to mitigate Internal and External Risk that may affect the iteration's successful completion (not the level of success in implementing effective solutions). The project manager also formulates and evolves the vision when needed through interactions with customers. This process happens across iterations, and is viewed at a level of detail called the Product Life Cycle. The customer is primarily aware of activities at this scope.
The iteration manager's responsibility is to ensure that the current iteration addresses as many of the highest potential needs possible while staying on schedule. Efforts on a project in an iteration are limited by time constraints on the developers. The resulting new product at the end of the iteration will implement new Solutions. Depending on the project Configuration, some solutions may be optional to complete the iteration, or some solutions may have cheaper implementations that can be selected to keep schedule.
During each iteration, the iteration manager interacts with customers to adjust the needs that will be addressed by the Iteration in the most cost/return effective manner and to ensure that solutions implemented in the iteration yield high value. Human customers (people) are consulted to generate Primary Needs Knowledge (such as Use Cases) and from those requirements additional Secondary Needs Knowledge may be developed between the GUI and underlying Service Layer.
Within an iteration, developers incrementally create and improve solutions that satisfy customer needs. Needs knowledge gathered from customers indicate needs and Potential Solutions. The potential solutions are prioritized based on how critical the needs they address are and based on any interdependencies (which should be kept to a minimum within an iteration). Each day developers work on a single Increment of one or more solutions. Increments of a solution that work better in at least one respect and as well if not better in all respects may be Committed to the iteration's new product. A single solution may have more than one increment in an iteration, but each must either be successive, or carefully Merged if two happen in parallel. Only the final set of solutions is included in the finished product for the iteration.
External risk is generated by agents outside the scope of the Iteration, including customers. Internal risk is generated by agents inside the iteration scope, including the project lead, iteration lead, developers, and any agent members of the Project Configuration. Risks represent forces that drive changing needs. When risks are ignored or unaddressed they can reduce or completely negate the value of solutions when applied to the needs they are designed to satisfy.
Solutions are procedural functions that satisfy agent needs. Solutions have a Quality rating, which is variably dependent on the agent applying it to a need and the configuration of the need over time.
Configuration is a declarative aggregation of information (facts) and solutions. Agents, projects, products, and needs have configuration. While agents Apply solutions recursively or in sequence, configuration can affect agent solution application through Reflection. Communication between agents can also initiate or affect solution application.
Project configuration includes information and solutions (such as development tools and processes) that help the developers produce solutions for the product.
The product's configuration is composed of information and Internal (private) solutions that may be used by the External (public) solutions exposed to agents using the product. A product's configuration may be altered during delivery as part of the delivery cost to make that product's solutions better suited to the agents using the product (an Instantiation of that version).
Additionally there are several special cases worth noting:
Developers on a iteration may be affiliated with one or more Organizations which donate time of the developers in the expectation that the resulting product will better meet some of the organization's needs. Such organizations are agents which are also a collection of (one or more) people, and are considered Patrons for the project.
Patrons may have direct or indirect influence on the vision, but care must be taken that this influence is constructive from the customer viewpoint, and not destructive. When patrons are also customers, their patronage may dominate and skew the vision (an external risk). When patrons are not customers, their patronage may complicate or cause inconsistency in the vision (an external risk).
Working directly with customers is not always possible, and it is not always ideal. Proxy Customers (customer advocates) and Customer Models (personas) are two substitutes for working with real customers.
All people, while agents, have a particular component of their configuration commonly called Free Will. Free Will has two primary consequences that must be considered as risks. First, since free will is part of a person's configuration, if can affect how they apply solutions. People that are agents applying a developed solution cannot be expected to apply it as intended or instructed the way system agents can. Second, people that are customers cannot be expected to apply solutions included as part of the software development process as intended or instructed. This is also a concern for developers, but customers have less incentive to follow the chosen software development process.
Project configuration includes software tools (including hardware), non-software tools, development processes, standards, working environment, staff that support the developers in their efforts (dedicated or otherwise), and other factors enabling development.
Projects in general are a special case of products, a sort of meta-product.
In addition to needs knowledge, which indicate how the product's solutions should address needs, there is a need for insight into potential external risk from the customer. The iteration lead should also collect Solution Risk Knowledge (Abuse Cases), risks that the product should specifically address. These are not project risks, and as such are not handled by the project lead. These are product risks, and are rarely an exhaustive list. As customers use the product additional solution risk may be discovered and can be prioritized for addressing in a future Iteration.
Solution risk includes bugs implicitly, but bugs are often poorly classified. Defects include problems with a solution that are both strictly internal to the product and represent a failure to mitigate solution risks that were identified to be addressed within the iteration. Risks can change, and risks can be misunderstood. Deviations include problems with a solution that are a result of a mismatch between the Current Risks and solution risk knowledge. A solution with no defects is attainable, but solutions almost always have deviations.
Defining a Project, Iteration Two
A second look at the problem domain of solution development reveals a more generic model. There is a danger in distilling a model too far: it looses semantic meaning. None the less, I feel this deeper level of abstraction is useful when considering the two very hard to reconcile domains of Computer Science and Social Science. Computer Science is an applied Math. I feel many professionals try to bleed an Engineering discipline out of Computer Science, and they feel by squeezing it with enough Social Science such a discipline can be attained.
Rather than trying to bleed a turnip, I prefer to follow a different popular methodology: separation of concerns. Social Science can tell us a lot about the process of programming, but the actual result of programming will always be Math. There needs to be a clean division between these two disciplines if either is to be applied effectively and appropriately.
Modern approaches like Object Oriented programming abstract the problem domain of programming computer hardware to a level much easier to understand for humans. Object Oriented programming in particular has a very weak basis in Math where as approaches like the Relational Model also abstract solutions while keeping a firm mathematical basis. Both approaches have benefits and weaknesses. Abstraction in general can weaken or limit a potential solution, but the potential trade-off is a set solution that are easier to understand, develop, and maintain.
A mathematical view of computer science supports many concepts needed to model solution development.
At the basic level, programs are comprised of Values and Functions. Through the lens of Social Science, these are essentially Things and Events respectively. Without leaving the mathematical basis, values are static and Immutable. They are simply facts that exist, and are identified by the value itself. A Function is a Dependence between values, and it can be represented in a number of ways. Functions too are also immutable, and identified by the dependence they represent. These two simple concepts create many computational possibilities, but they lack certain catalysts. The first issue is that something must apply values to functions actively. Doing this, one may line up functions end-to-end like dominoes to create limitless interesting results from a single initial value, but the product remains a static line.
The notion of Identity is important. For values and functions as defined above, the identity is the value or function itself. In order to make something Mutable, something changeable is needed. Variables are symbols. They may be a symbol for any value or for any function, and may even be a symbol for another variable. Variables are identified by the symbol, but the symbol only defines how they are referenced, not what they are.
Sets and Tuples are another needed concept from Math, these create Structure. A set is an unordered group of any number of unique values. A tuple on the other hand is an ordered group of a specific number of values, and may contain the same value more than once. Each of these can be considered a form of association both between the identity of the set or tuple and its components, and among its components. From these mathematical constructs, the notions of Type, Atoms, and Corpus are possible. A type is simply a set of values and/or functions. An atom is a typed variable, meaning that it has a symbolic link with a value, but that value must be a value in set of a specific type. A corpus is a set of atoms.
It is important to consider that values can be a set, tuple, or a Trivial value. Trivial values (also called Scalars and indicated as * in the diagram above) have exactly one distinct components. Sets and tuples have zero or more distinct components. Tuples (also called Arrays), specifically Pairs are important because they allow for order and branching which is needed to establish procedure. A pair is a tuple with exactly two values. Functions can be defined as a pair who's values are tuples of the same cardinality (size). The Nth value in the second tuple can represent the value dependent on the Nth Value in the first tuple. This definition reveals that functions are merely a specific type of atom.
While other cardinalities of tuples are useful, it is often best to use the simplest components possible. Pairs make it possible to represent information as a Tree of values. Just as it is possible to actively chain functions linearly, a tree can be actively traversed in a particular order: the first value in the pair, then the second. If the first value in the pair is another tree, its first value is traversed, then the second and then the second value of the original tree.
A Process is a tree which has an implicit linear structure, has a specific input type, and has an output atom. A process is like a function, but it has the flexibility to Branch and a Constraint on the input value and output value.
Looking back at the first definition for the process of solution development, everything defined can be modeled using these few components.
Forces are a medium for change. They have a Target corpus and are created (owned) by an Agent. The target, or any component of that corpus may be in input of the process. Entities are a form of corpus that have an Interface allowing a limited degree of access to any components of the corpus that would otherwise be unavailable due to Closure (encapsulation). Agents are simply entities that may create forces in addition to providing an interface.
Patterns, Templates, Process, and Configuration
Configuration is a specific thing and has definition, even though most of the configuration for a project is intangible and subject to change, from moment to moment it is a specific structure of values. Part of a project's configuration is the people working the project, and part is the technology they are using to get the work done.
UP4edu does not deal too much with decisions on either of these two layers of configuration. It is assumed that the layer dealing with people is relatively fixed. There are some things your project configuration can influence when hiring new team members and assigning roles, but is is assumed that that project configuration is not the primary concern when undertaking these activities. Similarly, project configuration can influence technology choices as it directly impacts the process of developing solutions, but not as it impacts the product itself.
All aspects of the project, including people and technology related
issues are reflected by the configuration. It is important to
understand the difference between the reflection of these facts in the
configuration, and the factors deciding them. Process configuration does define what people in various roles are supposed to do in the scope of their work on the project.
How teams arrive at a process varies, but in Agile development the process should be adaptive. This is one divergence point on various prescriptions of Agile process. Some XP and Scrum books assert the two processes are not negotiable and reconfigurable, others indicate a certain level of flexibility. A process that cannot be changed is simply a process, one that can be tweaked in specific ways is a form of Template. I am personally a fan of Patterns, and so as much as possible UP4edu defines any prescriptions in such terms.
Regardless of the level of adaptability, who makes decisions regarding adaptations is reflected by, but not decided by, configuration.
One reason patterns are so useful, and applicable to the model UP4edu proposes to use is their definition is framed in terms of forces and solutions. Patterns are flexible in their application. One pattern does not imply the use of another, though using patterns in tandem may have reproducible benefits. Rather, application of patterns is dictated by the nature of the problem, the forces that need to be resolved. Templates lack this flexibility.
UP4edu is not strictly a Pattern Language, but rather a model of the solution development process that includes a pattern language. In the second part of this post I'll define UP4edu, what it values, and how to apply it.