Is Management without Evidence like Building on Sand? Does a Culture and Tool Kit of Evidence Provide us with a Better Foundation?
I’m getting ready to update my syllabus for my Executive MBA course on Managing Innovation and Change. We cover the same material as in the regular MBA program, but with a greater focus on issues for the experienced manager. In my course we focus our attention on the creation and management of innovation, motivation/compensation, negotiation/change management, and the integration of technology tools with management practice. For each major course concept there is a written assignment where students explain the basics of the concept and provide citations, apply the concept to the student’s work or other related organization, and design an evaluation method to assess the ROI of the application. This last section is a major roadblock for most students, and I’m trying to understand why. I’m planning on changing the final project such that students must select, implement, and evaluate one of these concepts – not just write about how they might do these things.
Given the trouble that students have with even just writing about this process, how successful will they be at full implementation? That said, if they can’t implement what they are learning, and show value, then what’s the benefit to the organization? Even if you have good ideas about intertwining technology and organizational practice — if you can’t implement and evaluate, how will you be able to manage the situation as it unfolds?
(Pure speculation begins here.) Is this difficulty because the earlier portions of the Executive MBA program – accounting, operations management, finance, marketing, international management – imply that only certain kinds of objective outcomes have value? Do they imply that their measurement techniques are only valid in their particular disciplines? These students have been in the program for over 14 months when I meet them. They have gained the majority of the skills available in the program. They (and MBA students world-wide) have measurement and evaluation skills based on accounting, operations, finance, marketing – but why aren’t we teaching them measurement and evaluation skills for general management and organizational change, or at least showing them how these other, more traditionally empirical, disciplines may help them evaluate their management/technology practice?
I’ve mentioned evidence based management before/, but Tracy Allison Altman’s recent post on the loneliness of the evidence based manager made me look at my industry to see if we were doing our students a disservice by not formally teaching research methods appropriate to management. I have added a couple of class sections on the topic, but the underlying skills (survey design, experimental design, etc.) are generally given a full course in industrial psychology degree programs. Measuring more subjective outcomes is difficult and requires a specific skill set (bad surveys, for example, being far worse than no survey at all). An extensive search turned up very few appropriate readings (see below for links). Harvard Business School Press lists 997 cases and articles on negotiation. Eighteen cases and articles come up when you search on evidence-based management – and eleven of these are by the same two authors (Jeffrey Pfeffer & Robert Sutton) largely focused on their excellent “Hard Facts,” ideas, but you’d hope that this critical topic were addressed by more management scholars (again, see my list below for what is available). We don’t have room to add whole new courses in most MBA programs, so we better find a way to teach this material more efficiently.
The students have good intuition. They realize that their decisions may impact only a few and so fancy statistical assessments may not work with such small sample sizes. They also understand that a control group would be of value for being able to discern differences – but they don’t see how they can manage both an “experimental” and a control group in the real world. I am surprised by how rarely before and after measures are considered. I believe this has something to do with the limited culture of measurement, let alone evidence, in many of their organizations (i.e., they don’t have the “before” data). This harkens back to my earlier visualization posts. How can we know that what we implement is working if we didn’t know from where we started?
I’d appreciate thoughts on how you or your firms are working to build a culture of evidence. Both technology focused and general management practice can be measured, and as a result, adapted as need be. Evidence can also provide the background to make changes your intuition alone cannot support. On the other hand…
Background material:
- Pfeffer & Sutton’s website/blog. Includes teaching materials and case examples.
- Thomke’s Enlightened Experimentation: The New Imperative for Innovation
- Perla’s Do your Metrics Measure Up
- Lawler’s Why HR Practices are Not Evidence-Based
- Roussea & McCarthy’s Educating Managers from an Evidence-Based Perspective
- World Bank Selecting an Evaluation Design
4 Responses
June 4th, 2008 at 8:48 am
Terri, You are asking lots of good questions — including “why aren’t we teaching them measurement and evaluation skills for general management and organizational change, or at least showing them how these other, more traditionally empirical, disciplines may help them evaluate their management/technology practice?” On Evidence Soup, I write about some ongoing efforts to bring better evidence to managers. But in my opinion, MBA programs aren’t providing enough hands-on experience and toolkits to students, so they can better know how to develop and evaluate evidence (and I still see some professors act — perhaps unintentionally — as if scholarly journals contain the only real “evidence”… I’ll save that rant for another day).
I think it’s great that you want to change your syllabus so students actually implement something, rather than “just write about how they might do these things,” as you said — that’s a better way for them to become engaged with something they can apply in their careers. Since we don’t want to simply hit students with a firehose /smattering of topics, I’d like to see more MBA schools let students select a specific implementation and evaluation area (e.g., tactical marketing and outcomes measurement), and really dig into that, rather than learn a little of everything. (Gasp! This might mean messing around with the curriculum. As a practical matter, a school might develop, say, 4 or 5 subject matter areas or tracks that students could pursue after completing a base of fundamental courses on HR management, finance, etc.) I know lots of folks who say they went through their MBA programs picking up just enough to be a mile wide and 1/2″ deep. This is at odds with trends in the workplace, where — whether we like it or not — specialization is the thing. Allowing each student to focus on what interests him or her, and play to their strengths, would (again, IMO) make the degree program more attractive and marketable — think of it as an opportunity a B-school to offer mass education customization. It also would help your school contribute more evidence-based managers to the business world.
June 4th, 2008 at 9:10 am
Great ideas Tracy! Santa Clara does offer the regular MBA students the ability to “concentrate” on areas (I’m the concentration adviser for Managing Technology and Innovation
http://www.scu.edu/business
/graduates/academics
/managing-tech-innovation.cfm
… but the EMBA program is more lock step. However, we, and many other schools, are revamping our EMBA to be more hands-on.
Next step is to act on the ideas of giving them better evidence tool kits — and models for thinking about their practice that way.
Thanks for the comments and your blog.
June 4th, 2008 at 11:46 am
Terri, Thanks for setting me straight re: concentration areas. What I meant to say was that perhaps students should be able to customize their research, whether they’re in a lock-step cohort or enrolled in a concentration area within a standard MBA program. I wrote more about this at Evidence Soup in How B-schools could offer Mass Education Customization and teach evidence-based management at the same time. What I have in mind would be highly individualized — sort of like “sub-concentrations”. A student would choose a narrowly defined topic and investigate that common thread during each course in their program, to develop more cumulative learning and better data maturity.
June 4th, 2008 at 12:11 pm
Very interesting ideas — much more of an individualized approach than programs I know of. Stanford may have moved towards the experiential portion with their shorter modules and labs. I wonder whether the biggest value would be for the EMBA vs MBA students (who often, at least at the beginning of the program, don’t know where they want to focus. For EMBAs this seems like a slam dunk if you can manage the cost-efficiency issues you raise in Evidence Soup.