Technology and Organizations

I’m getting ready to update my syllabus for my Executive MBA course on Managing Innovation and Change. We cover the same material as in the regular MBA program, but with a greater focus on issues for the experienced manager. In my course we focus our attention on the creation and management of innovation, motivation/compensation, negotiation/change management, and the integration of technology tools with management practice. For each major course concept there is a written assignment where students explain the basics of the concept and provide citations, apply the concept to the student’s work or other related organization, and design an evaluation method to assess the ROI of the application. This last section is a major roadblock for most students, and I’m trying to understand why. I’m planning on changing the final project such that students must select, implement, and evaluate one of these concepts – not just write about how they might do these things.

Given the trouble that students have with even just writing about this process, how successful will they be at full implementation? That said, if they can’t implement what they are learning, and show value, then what’s the benefit to the organization? Even if you have good ideas about intertwining technology and organizational practice — if you can’t implement and evaluate, how will you be able to manage the situation as it unfolds?

(Pure speculation begins here.) Is this difficulty because the earlier portions of the Executive MBA program – accounting, operations management, finance, marketing, international management – imply that only certain kinds of objective outcomes have value? Do they imply that their measurement techniques are only valid in their particular disciplines? These students have been in the program for over 14 months when I meet them. They have gained the majority of the skills available in the program. They (and MBA students world-wide) have measurement and evaluation skills based on accounting, operations, finance, marketing – but why aren’t we teaching them measurement and evaluation skills for general management and organizational change, or at least showing them how these other, more traditionally empirical, disciplines may help them evaluate their management/technology practice?

I’ve mentioned evidence based management before/, but Tracy Allison Altman’s recent post on the loneliness of the evidence based manager made me look at my industry to see if we were doing our students a disservice by not formally teaching research methods appropriate to management. I have added a couple of class sections on the topic, but the underlying skills (survey design, experimental design, etc.) are generally given a full course in industrial psychology degree programs. Measuring more subjective outcomes is difficult and requires a specific skill set (bad surveys, for example, being far worse than no survey at all). An extensive search turned up very few appropriate readings (see below for links). Harvard Business School Press lists 997 cases and articles on negotiation. Eighteen cases and articles come up when you search on evidence-based management – and eleven of these are by the same two authors (Jeffrey Pfeffer & Robert Sutton) largely focused on their excellent “Hard Facts,” ideas, but you’d hope that this critical topic were addressed by more management scholars (again, see my list below for what is available). We don’t have room to add whole new courses in most MBA programs, so we better find a way to teach this material more efficiently.

The students have good intuition. They realize that their decisions may impact only a few and so fancy statistical assessments may not work with such small sample sizes. They also understand that a control group would be of value for being able to discern differences – but they don’t see how they can manage both an “experimental” and a control group in the real world. I am surprised by how rarely before and after measures are considered. I believe this has something to do with the limited culture of measurement, let alone evidence, in many of their organizations (i.e., they don’t have the “before” data). This harkens back to my earlier visualization posts. How can we know that what we implement is working if we didn’t know from where we started?

I’d appreciate thoughts on how you or your firms are working to build a culture of evidence. Both technology focused and general management practice can be measured, and as a result, adapted as need be. Evidence can also provide the background to make changes your intuition alone cannot support. On the other hand…

Background material:

• Pfeffer & Sutton’s website/blog. Includes teaching materials and case examples.
• Thomke’s Enlightened Experimentation: The New Imperative for Innovation
• Perla’s Do your Metrics Measure Up
• Lawler’s Why HR Practices are Not Evidence-Based
• Roussea & McCarthy’s Educating Managers from an Evidence-Based Perspective
• World Bank Selecting an Evaluation Design

I had the pleasure of working with folks from SRI, the US Army, PARC, and other interesting places yesterday. Our topic was Social Information Production. Much of the discussion focused on Communities of Practice (CoPs) & After Action Reviews (AAR).

CoPs are a set of people bound together through common interest and language with the goals of open communication, and exchange and retention of pertinent knowledge. They don’t have to have technology support, but John Sawyer and I have found that CoP technology tools can especially support the development of explicit knowledge related to the CoPs’ area. As I’ve discussed before, it’s the combination of technology and organizational practices that is often most powerful.

AARs have a simple set of questions to guide the review:
1. What did we set out to do?
2. What happened?
3. Why did it happen?
4. What are we going to do about it?

LTC Nate Allen, one of the two founders of the Army’s well-received CompanyCommander and PlatoonLeader CoPs (see his and LTC Tony Bugess’ book here), gave me some updates on their AAR process:

They are finding good value in looking at “close calls” versus bad outcomes. Close calls don’t get people as defensive as they would with a disaster — better learning environment.

They also now include another agenda item: What did we get away with? For example, the outcome was ok, but in hindsight it could have been a problem and should be considered in the review. I’m going to add this one to my class presentation on AARs. Overall, I like the idea of doing AARs as a matter of course – as part of a milestone process. Emotions may be less likely to cloud the process if it’s a standard tool and the approach is familiar.

I came away from our discussion with a lot to think about.

1. Key to the ideas of CoPs and AARs was that AARs have to be seen as a process, not an outcome. You don’t do an AAR to generate content (e.g., lessons learned) – though that can be a great outcome – you do an AAR to carefully reflect on what could be done better to help you plan for the future.

2. To the extent that part of the CoPs’ approach for learning is to integrate with AARs, the CoP can only be more powerful. The CoP can be a platform for the process and the outcomes. Facebook’s recent amazing valuation may as much about its growth as a platform, rather than its current capabilities as a “social watering hole.”
3. To the extent that we can use technology to support the AAR, it is more likely to be a living process (e.g., by tracking the workflow in the first place so the AAR can be evidence-based or by capturing the outcomes of the AAR so that the determined changes can be tracked and themselves evaluated).

Both CoPs and AARs are powerful in organizations and well worth the effort. The added power of one supporting the other is also well worth the effort of integration.