The question brings up the basic issue of validity and reliability in testing, a topic about which many books have been written.  Both are important issues in creativity testing, as well as any other kind of testing.  In creativity testing, they are especially slippery.

First, the definitions of creativity seem to be as numerous as there are writers on the subject.  If researchers cannot agree on what creativity is, then when using a test of creativity, test users will have a problem being sure that what they find or don’t find is actually creativity, or creative ability. Maybe the test maker’s definition of creativity is different from the user’s definition.

This is the issue of validity.  Does the test measure what it purports to measure?  It may measure some elements of the characteristic or behavior, or ability, but does it “work” (by identifying creativity) for all types of people (those of different ages, races, or genders)?

The validity of any measure is never perfectly resolved.  It is a goal toward which test developers work.  When they start to test a psychometric measure, they can make claims of validity only to the degree that the test has been used and found satisfactory with a particular group of users. As the testing continues, they may expand the claim of validity as the test makers use different groups, different situations, and still find that the measure is identifying the desired quality.

Another problem with tests of creative ability is that they do not measure other correlates of successful creative performance.  These include motivation, perseverance, attention to detail, and risk-taking.  It is sometimes possible to “psych out” or figure out what the test is looking for and fake the test by selecting “far out” answers, or by simply generating a large number of answers.

If you share my point of view that “the proof is in the pudding,” you’ll look not for creative ability (because every human being has the ability to be creative), but look instead at the products that the person has created, and assess these.

Reliability, how certain you are that if a person takes the test today, his score will be similar to his score in the future;  or that if a group of raters look at a particular creative response they will have substantial agreement about the degree of creativity shown. Reliability is more easily tested for and assured.  Usually reliability is stated statistically, and assumed to be reliable within certain range of tolerance.

There are some resources that you can consult to find reviews of educational tests, including those for creativity.  Buros Mental Measurements Yearbook compiles published reviews of many tests, and their online database offers these reviews for a fee.  The ETS website offers some actual tests, not reviews.  Even when using Buros, one must remember that the reviewers may have their own biases.  You’ll still have to decide for yourself.

There are many old, “tried and true” creativity tests out there.  Some of them are used even today to help select job applicants for the next step in their selection process. This can be a problem when you realize that these tests may have become familiar to prospective employees, giving them a “test-wise” bias.  Also, realize that the work skills that were sought by these tests may have changed over the decades since they were first introduced.

In an effort to avoid these “war horses,” some organizations have made up their own creativity tests.  These “quick and dirty” tests may be useful in identifying prospects, but their results may not prove valuable. They may fail to identify the qualities being sought, or they may be unreliable for general use in the company.  Look instead for a measure that has stood the test of time, yet has been updated with improvements.

If there is anything that the “genius studies” of Lewis Terman clearly demonstrated, it is that potential for greatness is often mistaken as evidence that the future will bring achievement and success.  Terman spent decades at Stanford working the Stanford-Binet  IQ test and on testing that thought would identify “geniuses” and track their success over the years. Terman hoped to be able to follow, test, and cultivate a generation of geniuses to show how to nurture special talent in children.  His conception of giftedness was closely linked to academic talent, and he expected that the youngsters he identified would be successful in both school and life. His research was colored by his biased attitudes about the supposedly limited abilities of girls, many immigrants, and persons of any race but his own.

Over the decades of his studies, many of the students he followed did make impressive achievements.  But, it must be admitted that many others, identified as having great potential, often did not manifest this talent through their achievements.  What’s more, other children – not included in his studies — who didn’t appear to have that much going for them were able to achieve greatness, including for example, winning Nobel Prizes (which none of the “Termites” had achieved.) This gives hope to all of the rest of us “non-geniuses.”

If you want to find a creative person, look at what she has made, done, or built.  If one has been creatively productive in the past, he or she will be likely to do it again.  Don’t make the mistake of thinking that creative potential will imply creative productivity in the future.

Creativity requires the kind of relaxed attention that allows one to be “in the flow.” Sometimes that comes quickly, but it often takes time to tease out new ideas from the many mundane ideas that pass through our awareness.

Consider the new TV reality show: Work of Art: The Next Great Artist, where aspiring artists are pitted against one another in timed competition to create works of art of a particular type in 12 to 14 hours. This brings the shocking juxtaposition of our on-demand lifestyle to the world of culture.

While some professional art fields (industrial and graphic design, for example) require production on a schedule, the constraints of the TV show’s production schedule make the artistic products more like art school test projects than actual works of art. It must be frustrating for the artists on the show to have to set their artistic goals to what can be achieved in the time available.

On creativity tests, as well, the fact that you’re being timed can be an inhibition to creative production. Certainly, one cannot wait forever for a creative idea, but putting pressure on test takers is not conducive to the most creative solutions. If you have to limit the time that is allowed to solve a problem, make the amount of time provided ample so that test-takers are not rushed.

Creative thinking makes ample use of divergent thinking, so it makes good sense to consider it when trying to assess creativity.  When creativity tests were first devised, test designers were looking for ways to measure the ability to think “far out” as a way to counter the effects of the convergent thinking training we get in school.  Fluent divergent thinkers can come up with lots of new ideas.  They may not all be good ideas, but they can think up dozens.

When a creativity test asks for as many uses for a paper clip as you can think of, it is calling for divergent thinking.  But, for an idea to be truly creative, it must be novel and useful, while attractive enough to encourage adoption.  Few pencil and paper tests of creativity actually consider all three dimensions of creative production.  A test that merely counts the number of new ideas (as some do) is not a good way of judging real life creativity.

Here’s a problem and pitfall to be aware of:

1. Assessing creativity with a tool that looks for the one right answer.

There are many tests for creativity on the Internet.  Some may make you feel good about your abilities, while others can be depressing.  Some common creativity tests, like the famous 9 dot problem, have discouraged test-takers for years.  This problem is a quick little parlor trick that used to be  seen in magazine articles, purporting to test your creativity, but now has made its way to the Internet.  These tests look for a clever answer to the problem through insight, the ability to quickly see a pathway to the answer.  Even some scientifically based tests of creativity use “insight problems.”

The problem with these tests is that, just like on a traditional arithmetic quiz, there is only one correct answer for the problem. This is the kind of convergent reasoning that we are taught in school.  But creativity, by its nature, looks for multiple opportunities and many ways to “skin a cat.”  In real world creativity, one is asked to broadly imagine divergent possibilities, try a few, and come up with one or more that will work.

These insight problems test convergent thinking, while claiming to help in locating divergent thinkers.  Think twice before using these for anything but the clever amusements that they are.

In business decision after decision, Jobs has strategically valued design along with other important factors of product success.  Not only are their products designed in well-crafted and elegant ways, even their commercials and their retail stores express the same values. Whereas Microsoft products have become commoditized, becoming ubiquitous and indistinctive, Apple products have built their capitalization by going for a smaller market share but high profit margins.  They have consistently married technological advances with design advances stressing a savvy user-interface to create products that are unique trend-setters. And users have been willing to pay for the cache.

When Steve Ballmer discussed the future of PCs at the D8 conference last week, he showed that Microsoft still doesn’t get it.  He talked about PCs having various new “form factors” in the future, but concluded that PCs will still be PCs no matter what they look like.  Design isn’t just what the product looks like from the outside.  It starts inside, even conceptually, to create a device that is at its heart, truly something new.  Without that realization, Ballmer is probably correct: Microsoft’s idea of PCs will continue to be as it is today.

People will pay that extra premium for the product that feels good, looks good, and helps them express their own values and sense of themselves.  Good design truly is good business.

They’re both written by journalists who excel at making scientific information accessible to the general reader.  They both treat the subject of personal achievement in society and how it emerges.

• Both David Schenk’s The Genius in All of Us and Malcolm Gladwell’s Outliers advocate the virtues of practice, perseverance, and steadfastness.  Both debunk the myth of the gifted loner who by her special gifts is easily able to surpass all others. And both stress the importance of societal supports for creating a level playing field for achievement.
• Both also devote significant attention to the Louis Terman “genius studies,” their results (disappointing) and their methodological shortcomings (among them the selection of participants, and defining giftedness as intelligence.)
• Another similarity is in the fact that both push back on two different types of “predestination” theories that attempt to explain success and failure.  Schenk’s book is an extended essay debunking the notion that genes alone (or even genes and experience) combine statically to determine ability.  He maintains (and documents his point of view with a section of backup resources that amounts to about half of the book) that it is the dynamic interaction between the two: nature and nurture, plus the efforts of the individual through practice and self improvement to overcome weaknesses and strengthen abilities.  Gladwell is so interested in the social and cultural aspects of the nurturing of talent that he nearly suggests that the poor and under stimulated have very little chance of success in modern life.  This point of view is nearly as discouraging (or excusing) as the pure genetic explanation of success and achievement.

Outliers emphasizes how idiosyncratic, simply lucky, events can affect future success. In an interesting section he makes a case for the importance of date of birth in the emergence of the talent of hockey players. Gladwell puts an emphasis on all of the aspects of the environment around the achiever to make his point that it isn’t intelligence alone, or even pluck and perseverance, but having ability, being at the right place at the right time, and having mentors or others to support and encourage you that make the difference.

Schenk, likewise, points out the importance of having familial, cultural, and societal support for those persons able to create outstanding work. He gives tips for parents, teachers, and for individuals with a passion, for strengthen and developing that ability.

After reading both books, I developed a few takeaways from the two that seem important to me:

• Whatever your level of ability, trying hard is important.  Both books stress how important is the passionate practice of your favored activity.  Especially for people for whom many pursuits are easy, the notion of trying hard on something is not immediately apparent.  But for truly outstanding results, many long hours of practice and trying hard are essential.
• Giftedness as an explanation for achievement has outworn its usefulness.  There are too many people whose promise of giftedness has not been manifest in their creative products or their life achievements for those termed “gifted” to rest on their laurels, or for those not so classified to be discouraged or give up.  The proof of one’s giftedness is not in one’s abilities or scores on an exam, but rather in one’s products: what one actually makes, does, or achieves.  But, at the same time, until all children are given the support to help them pursue and develop their passions, attention to gifted education is important.
• There is a persistent and increasing value accorded by the world in general to both the concept of creativity and the value of its practice – in child-rearing, education, management, and invention.  Though we may not all be termed “gifted,” we may all make of our lives creative expressions of our values and our passions. We may even become gifted through our own efforts.

The Olivator

I recently received a terrific little gift.  It is the perfect thing for the person who has everything, or for somebody who likes to play with her food.  This product is a good example of the added value that you can achieve with excellent design.  The Style dimension of this product really sells it!

This little gadget  lets you insert your own stuffing into pitted olives.  (It doesn’t pit the olives; just stuffs them).  It comes with a little instruction sheet, but you can tell just by looking, how to use this gadget.  The leaflet did suggest a filling that I wouldn’t have thought possible: cocktail onions.

The few minutes of spare time I spent experimenting with various stuffings was entertaining, and gave me ideas for some other things I’m going to try.   Various cheeses work well.  Even peanuts were tasty and different when inserted in the olives. I plan to try some meat pates and maybe different nuts.

I rated the Olivator using the CPSS, and got the following results.

Surprising: I was surprised by several elements of the Olivator, reflected in the high score of 6.0.  The first was the fact that a little cleaning brush was hidden under the red cap of the plunger.  Another was the sharpness of the cutting tube.  It is possible to take samples from harder or tougher foods than I expected.

Original: Although there are other olive stuffers available, this one has been given more attention to detail, and seems much cleverer.  It’s score on this facet was 6.25.

Logical: At a score of 6.5, it is easy to use, intuitive, and meets expectations for how to use it.

Useful: This is the strongest aspect of the Olivator, rated at a perfect 7.  It solves the problem of stuffing olives in a fun and easy way.

Valuable: At a score of 4.6, this is the lowest rating for this product.  It is just a kitchen gadget, after all, but for products in that category it is a real winner.

Understandable: With a score of 6.2, this is an easy to understand product.  The physical design of the product tells you what it is for.

Organic: A score of 6.6 is a very high one.  The Olivator’s parts all function well and smoothly together and this gives an integrated, harmonious appearance.

Well-Crafted: Another very high score, at 6.4.  The Olivator has so many clever details, as well as a well-crafted appearance.  There is something solid and sturdy about the gadget that gives the impression of quality.

Elegant: At 5.8, this score is a little lower than I would have expected.  It is a high score, for a good product.  Elegance refers to a simplicity that is an indicator of refinement.  Perhaps the added details of its design made the score a little lower.

I recently read Deborah Mrazek’s essay in the March 2010 edition of the DMI News & Views. As Design Practice Manager, Corporate Marketing, HP, Ms. Mrazek is well aware of the need for appropriate metrics for communicating design effectiveness.  In her essay, she summarizes a paper written by John R. Hauser and Gerald M. Katz in 1998 that still holds good advice.

I found both this review and the earlier paper useful and interesting.  My own experience with design metrics leads me to stress some similar points.

It is important that metrics be used strategically — that the metrics reflect the strategy and values of the organization. If they do, the metrics will help the company move forward profitably. But because the metrics selected will affect actions and decisions, it is very important to look carefully at what is proposed to be measured, and at how that measurement affects the strategic direction of the organization.

Hauser and Katz stated that “The firm becomes what it measures.”  Because of the focus on the metric, managers and others begin to pay more attention to the thing being measured, and that aspect will become maximized.  Since the quality measured becomes important through the focused attention it received, selecting relevant and effective metrics is critical.

Again, this bears a circular relationship with an organization’s strategic position and goals.  The metrics must both reflect the strategic path of the organization, and point the way to actualizing the strategy.  If the metric can help move the company along that path, it is an effective metric;  if it does not, or becomes a goal in itself, if fails.

In selecting metrics for judging design effectiveness look for qualities that can be measured today, but that may also help to predict future outcomes.  There is a temptation to pick as a measure what is easy to measure, but that choice might not be a relevant marker of success in the product.  Also, look for metrics that are within the control of those whose performance is being measured by it.

This is especially important with designers and engineers, the impact of whose contribution to the bottom line may be clearly manifest only months or years farther down the line.  Because of this, it is important to measure what matters to the customer. There are often different kinds and levels of customers, from end users, to distributors and vendors, to stakeholders within the company.  It is also necessary to understand the needs of the product manager, and the design team members, in addition to the needs of upper level management.

Hauser and Katz also pointed out the problem of focusing on one easily measurable characteristic, out of proportion to its importance to the customer.  For example, focusing intently on metrics for high standards for durability, engineers and designers can be discouraged from attending to other aspects of the design that might be more important to the consumer.   I suggest looking at a broader range of criteria based on varied dimensions of product creativity.

Hauser and Katz recommend looking for new ideas outside the organization as well as inside.  A “not-invented-here” culture emerges from only rewarding ideas conceived internally, rather than from profitably adapting and applying ideas from inside and outside the organization.  This can limit the creative capacity of a design team.

In selecting metrics, precise measures seem intuitively to be desirable, but this can be difficult when measuring aesthetic qualities.  Don’t get too caught up in maximizing the precision of the metric.  A metric is important in measuring movement and direction, and too much focus on the specific number yielded by the metric can deflect focus from the direction. When measuring illusive qualities like aesthetics and taste, scientists need not be reluctant to use what may appear to be subjective measures.  If these metrics help move the organization along its strategic path, they are useful and valuable.

Finally, choose metrics that don’t take so much time and money to implement that they cost more to use than they yield in productivity. Remember that the metric is to be a tool or guidepost along the way, not an end in itself.