Technology and Organizations

I’m getting ready to update my syllabus for my Executive MBA course on Managing Innovation and Change. We cover the same material as in the regular MBA program, but with a greater focus on issues for the experienced manager. In my course we focus our attention on the creation and management of innovation, motivation/compensation, negotiation/change management, and the integration of technology tools with management practice. For each major course concept there is a written assignment where students explain the basics of the concept and provide citations, apply the concept to the student’s work or other related organization, and design an evaluation method to assess the ROI of the application. This last section is a major roadblock for most students, and I’m trying to understand why. I’m planning on changing the final project such that students must select, implement, and evaluate one of these concepts – not just write about how they might do these things.

Given the trouble that students have with even just writing about this process, how successful will they be at full implementation? That said, if they can’t implement what they are learning, and show value, then what’s the benefit to the organization? Even if you have good ideas about intertwining technology and organizational practice — if you can’t implement and evaluate, how will you be able to manage the situation as it unfolds?

(Pure speculation begins here.) Is this difficulty because the earlier portions of the Executive MBA program – accounting, operations management, finance, marketing, international management – imply that only certain kinds of objective outcomes have value? Do they imply that their measurement techniques are only valid in their particular disciplines? These students have been in the program for over 14 months when I meet them. They have gained the majority of the skills available in the program. They (and MBA students world-wide) have measurement and evaluation skills based on accounting, operations, finance, marketing – but why aren’t we teaching them measurement and evaluation skills for general management and organizational change, or at least showing them how these other, more traditionally empirical, disciplines may help them evaluate their management/technology practice?

I’ve mentioned evidence based management before/, but Tracy Allison Altman’s recent post on the loneliness of the evidence based manager made me look at my industry to see if we were doing our students a disservice by not formally teaching research methods appropriate to management. I have added a couple of class sections on the topic, but the underlying skills (survey design, experimental design, etc.) are generally given a full course in industrial psychology degree programs. Measuring more subjective outcomes is difficult and requires a specific skill set (bad surveys, for example, being far worse than no survey at all). An extensive search turned up very few appropriate readings (see below for links). Harvard Business School Press lists 997 cases and articles on negotiation. Eighteen cases and articles come up when you search on evidence-based management – and eleven of these are by the same two authors (Jeffrey Pfeffer & Robert Sutton) largely focused on their excellent “Hard Facts,” ideas, but you’d hope that this critical topic were addressed by more management scholars (again, see my list below for what is available). We don’t have room to add whole new courses in most MBA programs, so we better find a way to teach this material more efficiently.

The students have good intuition. They realize that their decisions may impact only a few and so fancy statistical assessments may not work with such small sample sizes. They also understand that a control group would be of value for being able to discern differences – but they don’t see how they can manage both an “experimental” and a control group in the real world. I am surprised by how rarely before and after measures are considered. I believe this has something to do with the limited culture of measurement, let alone evidence, in many of their organizations (i.e., they don’t have the “before” data). This harkens back to my earlier visualization posts. How can we know that what we implement is working if we didn’t know from where we started?

I’d appreciate thoughts on how you or your firms are working to build a culture of evidence. Both technology focused and general management practice can be measured, and as a result, adapted as need be. Evidence can also provide the background to make changes your intuition alone cannot support. On the other hand…

Background material:

• Pfeffer & Sutton’s website/blog. Includes teaching materials and case examples.
• Thomke’s Enlightened Experimentation: The New Imperative for Innovation
• Perla’s Do your Metrics Measure Up
• Lawler’s Why HR Practices are Not Evidence-Based
• Roussea & McCarthy’s Educating Managers from an Evidence-Based Perspective
• World Bank Selecting an Evaluation Design

“..43 Folders, Merlin Mann’s family of websites about stuff like personal productivity, life hacks, and simple ways to make your life a little better.” Lifehacker “features tips, shortcuts, and downloads that help you get things done smarter and more efficiently.” Both of these blogs provide great tips on a variety of topics – often stretching how you might think about using the technology you have and use everyday (i.e., duct tape and twine).

In this post I apply the same perspective to helping groups get things done just a little bit easier – without having to learn or buy new tools. Recently I posted about the kinds of things groups might gain from visualization of their processes. The more I thought about it, the more I thought there might be lower hanging fruit – actions that are a touch easier to apply and still may provide great benefit in your teams.

Scenario One. A friend of mine runs a weekly conference call managing a medium sized project. She sends out an email including the agenda and an email attachment of the prior meeting’s minutes. She runs the call and takes careful notes as the meeting progresses. After the call she spends at least 90 minutes creating the post meeting notes (minutes), formatting, and the like.

Good for her and the team that they have an agenda for the call. Basic team meeting 101 – otherwise how can people have access to the material they will need to contribute, when they need to contribute it? Where I think there is low hanging fruit is in the minutes: If the team rotated who took notes (perhaps using the agenda as the base for the notes) – and more importantly, took notes in a way that all could see them as they were being taken, they would gain at least four benefits. First, the real time nature of the notes provides visualization of what one member thinks is being said. The rest of the team can then chime in with corrections, elaborations, agreement to action items and the like. There is stronger engagement as the team is going to be held to their immediate agreement about the outcomes of the meeting. Second, once the meeting is over, value added work can begin, rather than spending time typing up notes from work that has now past. Third, since the team lead isn’t trying to take notes and run the meeting, both activities should be of higher quality. Fourth, the minutes are a living document versus an attachment that seems to just get archived and not looked at again.

Free tools that might make this work: Any wiki (depending on your need for privacy, you might pay a monthly fee – see WikiMatrix for a list of many options), or Google Sites.

Certainly there are caveats: For example, how comfortable will people be at having their initial typos and typing speed observed? In this last sentence I had to backspace for corrections twice (coffee hasn’t kicked in yet…) For this issue I do think there will need to be some social adjustments as well as the technical and practice ones. See my comment on “alpha drafts” – short version is that perhaps we can adjust our expectations for real time versus finished product.

Scenario Two. My colleagues and I were running a four-hour workshop at this year’s HICSS conference. We had about 50 people in the audience and were hoping for an interactive event. We’d also put together a wiki (with multiple pages so multiple people could contribute at the same time) with the idea that people would add to the wiki as the discussion progressed. It became clear that this wiki mode wasn’t working (I have successfully used this “live” wiki approach at another conference, but it didn’t work this time – another possible post topic). Instead of forcing the wiki, I started taking notes on the presentation PowerPoint slides. We had created a simple set of slides to lead the discussion so it was easy to just add the notes from the computer running the presentation – and then all the room could see them and elaborate. It was then a simple task to cut and paste those notes into the wiki after the session ended.

Three take away points and a question:

1. Simplify the meeting process by doing note documentation during the meeting rather than taking up time after the fact – yes it may make the meeting a bit longer, but I think the benefits will outweigh the cost.

2. Engaging participants in the live creation of a single set of notes enhances engagement, and provides an opportunity for elaboration, action item creation, and error correction.

3. When the notes are a live work product they are more likely to be recycled – the whole group created them (generating greater commitment and understanding) and will be more likely to effectively search for the content when needed.

Question: Given the ubiquitousness of powerpoint and projection in large meetings, why ever use a flipchart or whiteboard? Flipchart sheets, for example, are then either transcribed (adding effort), or thrown away. While they provide the value of the “at the moment” representation of the group’s thinking, their value sharply decreases after the meeting. Electronic notes can work face to face or in a virtual setting, can be searched and archived without additional effort. (For smaller meetings or cases where you don’t have access to projection, a variety of products let you easily – and in some cases for free – take a picture of the notes on the whiteboard, chalkboard, flipchart, marble tablet that you were using and through OCR have that material transformed into searchable notes. I’m just beginning to use Evernote (free) for this purpose.)

Thank you to Michael Griffith (my brother and Director, Application Development Group for the U of A College of Medicine) for comments on an earlier draft.

Google and many others are looking to a world where the Internet is your computer. Your data storage, word processing, presentations, everything can run remotely if you’re willing to have it live somewhere besides your own hardware. (Security and speed are two commonly mentioned downsides.)

Whether Cloud (more technically, Grid: Ian Foster’s grid computing blog) computing is the way of the future is not my area of expertise. I’m more interested in how people come to understand their computing environment – and whether this understanding matters. One side argues that you just want your appliances to work – you pick up the phone and make a call, you don’t need to know how the electrons translate into audio. The other side is that you can’t take full advantage of your systems if you don’t understand their features.

Clearly there is a range. I don’t actually know how my cell phone works, but I do know enough that if I’m getting a bad signal going to a window may help the antenna do its job. I don’t know how the underlying PHP code works that runs my wiki, but I do know how I can work around some of its limitations to get a particular job done. Similarly, most of us don’t know how email systems talk to one another, but we can make adequate decisions about when to use email versus when to pick up the phone and call someone.

Our problem solving ability and decisions are based on our “mental models” of how the system works. As long as these mental models are moderately accurate we can make decent decisions about how to use our technology.

Two points: (1) mental models do make a difference and (2) mental models are the result of the technology itself and the technology’s implementation.

Accurate models of “cloud computing” will include a knowledge of where the data is stored (e.g., on your local PC, inside the corporate firewall, on a commercial secure server); how often the data is backed-up; whether syncing between the local PC and the “cloud” version is automatic; where the application is running (e.g., will you be able to use the application when on a plane, or at a vendor’s site where you either will or won’t have Internet access); do you need to worry about updates to the software (not if the application is on the “cloud”); etc.

The difficulty with abstract concepts such as cloud computing is that they don’t give us much on which to base our sensemaking. Ancient technologies had a concreteness that hinted as to their use and functioning – the “affordances” were clearer. The more abstract the technology, the more implementation needs to focus on helping users develop their mental models. This may mean increased implementation time and budget. Not paying this price may mean users make uninformed decisions about how they manage their computing environment – an expensive outcome if that data is lost or compromised.

In a future post I’ll consider how different mental models of collaboration tools might help teams work better (e.g., who “owns” the team wiki?).

Card, Mackinlay, and Shneiderman describe visualization as “The use of computer-supported, interactive, visual representations of data to amplify cognition.”

Long long ago I was an undergraduate research assistant at UC Berkeley. Because of my job and ability to scrounge, I had a few “accounts” (at the time, computing wasn’t a free good) that let me access one of the mainframes used to run statistics packages. Other RAs, some of them my good friends, also had accounts. These were the days of terminal rooms and we were often working in different buildings in the middle of the night.
We would use the “who” command to see who else was working (and thus who could either help with a problem or go for coffee). This was a primitive form of creating situational awareness in a virtual work setting. Note that this required an action – issuing the command and then looking for the arcane login of your friend. The only representation of the data was what could be gleaned from the particular processes they were running at the time.

More modern approaches are to watch team members’ IM status (immediate feedback, can include what music they are listening to), blog updates (see description of how SocialText does this — generally longer cycle time), facebook or myspace postings (cycle time varies), Twitters (see this real estate example), and the like. Some of these may require action on the part of the tracker, trackee, or both, depending on how the systems are configured. More to the point, they require some kind of effort, and are not specifically designed to support team situational awareness through visualization.

Certainly technology provides us with the ability to create visualizations of teamwork, but what visualizations are useful?

The following are the first entries in my notes on this topic. Briefly, explicit and implicit coordination of communication and tasks may be the most mature areas of study and examples. Both have the ability to support coordination based on either direct or anticipated actions and needs of teammates and task demands. Visualization around mutual awareness, accountability, social dynamics, and work patterns (e.g., editing patterns in Wikipedia) appear fruitful, but also more complex and idiosyncratic in terms of what would be valuable for team performance.

Comments appreciated on other sources for extending this review and/or on examples of how your teams use visualization for fun or profit.

—

Unavailability and Rhythm Awareness – current and future availability. Begole and Tang describe the use of “Lilsys” and “Awarenex” within a research group at Sun Labs. Awarenex showed an augmented IM contact list which provided location, keyboard idle time, whether engaged in instant messaging, phone conversation, or scheduled for an appointment in the calendar. “The design of Awarenex transmitted awareness cues so that people could interpret the information to infer whether it would be a good time to interrupt…” (p. 12). (They do discuss the tradeoffs regarding privacy issues.) Lilsys used a sensor-based system (motion, sounds,phone, door, computer) to link to Awarenex and add machine interpretation of whether or not the person was likely to be receptive to communication. Later stages of the research used logs of this data to create visualizations of (for example) day-of-week rhythm patterns of activity. They note that this information could support contact coordination.

Coordination Requirements – “Who must coordinate with whom to get work done.” Cataldo, Wagstom, Herbsleb, & Carley report on their efforts to design collaboration and awareness tools. They developed a technique to measure task dependencies. Among other things, they created a coordination requirements matrix based on software modification request reports from a software development project within a large data storage company. They offer that a tool using their approach could provide stakeholders with visualizations to trigger facilitation of appropriate flows of communication. TUKAN and Palantir are mentioned as tools for supporting collaboration and awareness in software development – and that these tools could be augmented with better understanding of congruence between coordination requirements and coordination activities.

Team Implicit Coordination Processes: A Team Knowledge-Based Approach – Rico, Sanchez-Manzanares, Gil, and Gibson describe implicit coordination as “when team members anticipate the actions and needs of their colleagues and task demands and dynamically adjust their own behavior accordingly, without having to communicate directly with each other or plan the activity” (p. 164). Team situation models “are dynamic, context-driven mental models concerning key areas of the team’s work, such as the objectives or roles of colleagues.” The “sharedness and accuracy” of the team situation model feeds into implicit coordination (with a feedback loop), which is expected to support team performance.

Social Translucence — IBM Watson Research Center’s Thomas Erickson, Christine Halverson, Wendy A. Kellogg, Mark Laff, and Tracee Wolf note that “humans are remarkably skilled at using subtle cues about the presence and activities of others to govern their interactions.” They describe how people make decisions based on their being able to see “traces” of others’ activities. They propose digital systems that support mutual awareness and accountability. Social Translucence (not transparence) – visualization that people are doing something (e.g., participation in synchronous or asynchronous conversations, interaction in a lecture), but not exactly what they are doing/saying.

Augmented Social Cognition (blog) — Ed Chi and Peter Pirolli of PARC describe ASC as “the enhancement via technical systems of the ability of a group of people to remember, think and reason, acquire and use knowledge.” They have created a tool called wikidashboard that they hope will “surface social dynamics and editing patterns that might otherwise be difficult to find and analyze in Wikipedia. We are also interested in applying this tool to Enterprise Wikis.”

Only if it’s an entry point to effective decision making and action… The point of visualization is to help us make sense of complex data and relationships. The iVistra example in the prior post illustrates how one firm links sensors to organizational metrics to support peoples’ ability to know what’s going on, and how to make better decisions as a result. Thinking about these issues led me to wonder about the best forms for this visualization.

The following are notes from my initial foray into the topic. The wind picture above is a personal visualization favorite (fat red arrows are +25 knots of wind).

Cognos’ (an IBM company) whitepaper “Picturing Performance: Dashboards and Scorecards with Cognos” was a great start. They summarize Wayne Eckerson’s work noting that there are three dashboard applications (monitoring, analysis, & management), three layers of information (graphical, abstracted – to focus on key performance metrics; summarized dimensional data — for addressing root causes; and detailed operational data – that help to identify resolutions), and three types of dashboards (operational, tactical, and strategic). So, the best forms of visualization will match the particular goal being sought.

Rachel Bellamy and her colleagues at IBM write that “a visualization should be thought of as a user interface to a control task, not as a report.” I think that’s the biggest issue – the pretty picture isn’t the end, the pretty picture is the welcoming entrance into a relationship with the underlying data. And, like all relationships, need to evolve to survive.

Bellamy et al. draw from Card, Mackinlay, & Shneiderman’s book, “Readings in Information Visualization: Using Vision to Think” to suggest that visualizations should increase people’s visual capabilities and amplify cognition. They note that Card et al. outline six visualization benefits:

1. Increasing the users’ memory and processing resources
2. Reducing the search for information
3. Enhancing pattern detection
4. Enabling perceptual inference
5. Using perceptual attention mechanisms for monitoring
6. Encoding information in a malleable medium

I don’t yet have a copy of the Card book (it’s coming via Interlibrary Loan), but I do expect that they will talk about how even the best visualization tools provide only possibilities, and that these are turned into results when effectively integrated into the work process. No silver bullets.

These six benefits suggest that the “form” question is not one of pie chart versus line graph, or stoplight versus gauge (this point is made by most of the authors in my review, many harkening back to Edward Tufte’s work). Rather the issue is matching the visualization to how people will perceive the situation and be able to act on the information. Guidance on this matching has a broad foundation. We have fine-grained research published psychology journals (for example on “change blindness” – people’s difficulty with noticing change); user interface research like that published in human-computer interaction journals and practitioner focused work such as described in Stephen Few’s blog. All of these sources suggest that there is no easy approach, but that thoughtful application may provide significant benefit.

In Part III I’ll return to one of my own areas of research and review recent work on visualization for team collaboration.