Metrics for Design Creativity

March 6, 2010

Metrics are important to any company.  When used appropriately, business metrics enable organizations to take account of where they are and to plan strategically for the future.  Nowadays companies are also looking for design metrics, ways of measuring the contribution of the design of new products to their bottom line.

I recently read Deborah Mrazek’s essay in the March 2010 edition of the DMI News & Views. As Design Practice Manager, Corporate Marketing, HP, Ms. Mrazek is well aware of the need for appropriate metrics for communicating design effectiveness.  In her essay, she summarizes a paper written by John R. Hauser and Gerald M. Katz in 1998 that still holds good advice.

I found both this review and the earlier paper useful and interesting.  My own experience with design metrics leads me to stress some similar points.

It is important that metrics be used strategically — that the metrics reflect the strategy and values of the organization. If they do, the metrics will help the company move forward profitably. But because the metrics selected will affect actions and decisions, it is very important to look carefully at what is proposed to be measured, and at how that measurement affects the strategic direction of the organization.

Hauser and Katz stated that “The firm becomes what it measures.”  Because of the focus on the metric, managers and others begin to pay more attention to the thing being measured, and that aspect will become maximized.  Since the quality measured becomes important through the focused attention it received, selecting relevant and effective metrics is critical.

Again, this bears a circular relationship with an organization’s strategic position and goals.  The metrics must both reflect the strategic path of the organization, and point the way to actualizing the strategy.  If the metric can help move the company along that path, it is an effective metric;  if it does not, or becomes a goal in itself, if fails.

In selecting metrics for judging design effectiveness look for qualities that can be measured today, but that may also help to predict future outcomes.  There is a temptation to pick as a measure what is easy to measure, but that choice might not be a relevant marker of success in the product.  Also, look for metrics that are within the control of those whose performance is being measured by it.

This is especially important with designers and engineers, the impact of whose contribution to the bottom line may be clearly manifest only months or years farther down the line.  Because of this, it is important to measure what matters to the customer. There are often different kinds and levels of customers, from end users, to distributors and vendors, to stakeholders within the company.  It is also necessary to understand the needs of the product manager, and the design team members, in addition to the needs of upper level management.

Hauser and Katz also pointed out the problem of focusing on one easily measurable characteristic, out of proportion to its importance to the customer.  For example, focusing intently on metrics for high standards for durability, engineers and designers can be discouraged from attending to other aspects of the design that might be more important to the consumer.   I suggest looking at a broader range of criteria based on varied dimensions of product creativity.

Hauser and Katz recommend looking for new ideas outside the organization as well as inside.  A “not-invented-here” culture emerges from only rewarding ideas conceived internally, rather than from profitably adapting and applying ideas from inside and outside the organization.  This can limit the creative capacity of a design team.

In selecting metrics, precise measures seem intuitively to be desirable, but this can be difficult when measuring aesthetic qualities.  Don’t get too caught up in maximizing the precision of the metric.  A metric is important in measuring movement and direction, and too much focus on the specific number yielded by the metric can deflect focus from the direction. When measuring illusive qualities like aesthetics and taste, scientists need not be reluctant to use what may appear to be subjective measures.  If these metrics help move the organization along its strategic path, they are useful and valuable.

Finally, choose metrics that don’t take so much time and money to implement that they cost more to use than they yield in productivity. Remember that the metric is to be a tool or guidepost along the way, not an end in itself.

George Kembel

August 16, 2009

I’ve finally found time to post some comments about the talk that George Kembel made on Friday morning at Chautauqua, entitled “Awakening Creativity.” I was blown away by it. I may just become the first-ever-George-Kemble-senior-citizen groupie. His presentation was just about perfect in my estimation. It was like a feast of ideas, all of which support and nurture creativity.

Kembel capably serves the role of liaison between his university program and the public at large. He handled himself with confidence, and with an obvious respect for the Chautauqua audience. The d-school design thinking method is built on empathy, and Kembel showed it for the audience. Empathy writ large. Hmmm… We’ve heard a lot during Week 7 about empathy and compassion.

Another feature of design thinking is story-telling. Again, Kembel soared. His whole talk was a series of stories of the challenges and successes of … the school?? … no, the successes of his students at providing life-enhancing contributions to our larger society, in locations as different from each other as a NY City radio news station and a remote village in the mountains of Nepal.

His first story began after we had a chance to demonstrate that perfect pitch is pretty rare in an English speaking population. Some in the audience did have it, but that wasn’t the point. He said that this special talent is found in only about 1 in 10,000 people of an average population of English speakers. He cited studies by Diana Deutsch, who showed that speakers of Mandarin (a tonal language) had much higher levels of this ability to identify particular tones than are found among most English speakers. Why? Because they need this special skill to manage their communication in an environment where the meanings of three different concepts may be represented by a nuanced verbal differentiation based on one’s tonal levels in speaking.

He asked and answered three questions about awakening creativity. I’ll paraphrase them: Is creativity normally distributed? ie. Can everyone be creative? He says, and we agree, that much creativity lies latent because we misunderstand what it is, educate our children out of it, and misunderstand its nature, not believing that “creativity thrives on constraints.”

His second question was to wonder whether or not this latent creativity can be awakened when it is nurtured? and the answer was “yes.” Here he stressed the value of the design thinking process and noted that it is important to use the process on as many real projects as possible so that the learner can begin to trust the process, while tolerating the times of deep uncertainty and ambiguity.

The third question Kembel explored was about how an individual’s transformation to more creative behavior can scale to the larger level of the work-team or the organization. He stated that it is possible to extend this improved creative performance, but that it is necessary to train for the skills of group process and to find like-minded individuals to work with in an organization that may not seem initially to offer many creative opportunities.

Throughout the talk Kembel frequently showed and discussed his concept of reiterative “low resolution” prototypes. These “quick and dirty” prototypes are used to help think through the problem, and to communicate one’s understanding of it at any given time. Although these low-res prototypes are not intended to be successful on the first go-around, he commented about how much courage it takes to put out your design for testing because we are trained to believe that unfinished work reflects poorly on our ability.

The truth is that it will take many design iterations, and much empathetic testing to thoroughly solve the problem. Here he stated that “the crummier the prototype, the better” because the stakeholders will feel so sorry for you that they’ll want to help you improve it through their own empathetic evaluation.

Because they allowed for this vulnerability, and other reasons, I was so pleased with the work of Martha and Robin who, in one short week, explored their product idea, put a low-res prototype together, tested it, and got feedback on it. This week’s work was only an approximation of design thinking, but it clearly demonstrated the principles that we heard discussed in the Amphitheater on Friday morning.

If you missed it, or would like to review it, go to FORA.tv for the archived video.

Imagine that …

"low res" Prototype: Fold-up Cookbook Stand

August 13, 2009



Robin used coat hangers to help her express her idea for a portable fold-up cookbook stand. Then we rated her design with the CPSS.