July 2, 2010
Last week a reader asked “Where could one find information about reliable and updated creativity tests?” What a great question! Not an easy one to answer, but an important one.
The question brings up the basic issue of validity and reliability in testing, a topic about which many books have been written. Both are important issues in creativity testing, as well as any other kind of testing. In creativity testing, they are especially slippery.
First, the definitions of creativity seem to be as numerous as there are writers on the subject. If researchers cannot agree on what creativity is, then when using a test of creativity, test users will have a problem being sure that what they find or don’t find is actually creativity, or creative ability. Maybe the test maker’s definition of creativity is different from the user’s definition.
This is the issue of validity. Does the test measure what it purports to measure? It may measure some elements of the characteristic or behavior, or ability, but does it “work” (by identifying creativity) for all types of people (those of different ages, races, or genders)?
The validity of any measure is never perfectly resolved. It is a goal toward which test developers work. When they start to test a psychometric measure, they can make claims of validity only to the degree that the test has been used and found satisfactory with a particular group of users. As the testing continues, they may expand the claim of validity as the test makers use different groups, different situations, and still find that the measure is identifying the desired quality. Read more
5 Pitfalls to Avoid When Judging Creativity: Going with an outdated or untested measure of creativity
June 25, 2010
5. Going with an outdated or untested measure of creativity.
There are many old, “tried and true” creativity tests out there. Some of them are used even today to help select job applicants for the next step in their selection process. This can be a problem when you realize that these tests may have become familiar to prospective employees, giving them a “test-wise” bias. Also, realize that the work skills that were sought by these tests may have changed over the decades since they were first introduced.
In an effort to avoid these “war horses,” some organizations have made up their own creativity tests. These “quick and dirty” tests may be useful in identifying prospects, but their results may not prove valuable. They may fail to identify the qualities being sought, or they may be unreliable for general use in the company. Look instead for a measure that has stood the test of time, yet has been updated with improvements.
June 24, 2010
4. Looking for potential, rather than performance.
If there is anything that the “genius studies” of Lewis Terman clearly demonstrated, it is that potential for greatness is often mistaken as evidence that the future will bring achievement and success. Terman spent decades at Stanford working the Stanford-Binet IQ test and on testing that thought would identify “geniuses” and track their success over the years. Terman hoped to be able to follow, test, and cultivate a generation of geniuses to show how to nurture special talent in children. His conception of giftedness was closely linked to academic talent, and he expected that the youngsters he identified would be successful in both school and life. His research was colored by his biased attitudes about the supposedly limited abilities of girls, many immigrants, and persons of any race but his own.
Over the decades of his studies, many of the students he followed did make impressive achievements. But, it must be admitted that many others, identified as having great potential, often did not manifest this talent through their achievements. What’s more, other children – not included in his studies — who didn’t appear to have that much going for them were able to achieve greatness, including for example, winning Nobel Prizes (which none of the “Termites” had achieved.) This gives hope to all of the rest of us “non-geniuses.”
If you want to find a creative person, look at what she has made, done, or built. If one has been creatively productive in the past, he or she will be likely to do it again. Don’t make the mistake of thinking that creative potential will imply creative productivity in the future.
June 23, 2010
3. Timing the test.
Creativity requires the kind of relaxed attention that allows one to be “in the flow.” Sometimes that comes quickly, but it often takes time to tease out new ideas from the many mundane ideas that pass through our awareness.
Consider the new TV reality show: Work of Art: The Next Great Artist, where aspiring artists are pitted against one another in timed competition to create works of art of a particular type in 12 to 14 hours. This brings the shocking juxtaposition of our on-demand lifestyle to the world of culture.
While some professional art fields (industrial and graphic design, for example) require production on a schedule, the constraints of the TV show’s production schedule make the artistic products more like art school test projects than actual works of art. It must be frustrating for the artists on the show to have to set their artistic goals to what can be achieved in the time available.
On creativity tests, as well, the fact that you’re being timed can be an inhibition to creative production. Certainly, one cannot wait forever for a creative idea, but putting pressure on test takers is not conducive to the most creative solutions. If you have to limit the time that is allowed to solve a problem, make the amount of time provided ample so that test-takers are not rushed.
5 Pitfalls to Avoid When Judging Creativity: Assessing creativity with a tool that looks for the one right answer.
June 21, 2010
This week, we’ll be exploring some problems with creativity tests. Each day, I’ll post one of 5 pitfalls I find with many tests of creative ability. Please offer your opinions to keep the conversation flowing.
Here’s a problem and pitfall to be aware of:
1. Assessing creativity with a tool that looks for the one right answer.
There are many tests for creativity on the Internet. Some may make you feel good about your abilities, while others can be depressing. Some common creativity tests, like the famous 9 dot problem, have discouraged test-takers for years. This problem is a quick little parlor trick that used to be seen in magazine articles, purporting to test your creativity, but now has made its way to the Internet. These tests look for a clever answer to the problem through insight, the ability to quickly see a pathway to the answer. Even some scientifically based tests of creativity use “insight problems.”
The problem with these tests is that, just like on a traditional arithmetic quiz, there is only one correct answer for the problem. This is the kind of convergent reasoning that we are taught in school. But creativity, by its nature, looks for multiple opportunities and many ways to “skin a cat.” In real world creativity, one is asked to broadly imagine divergent possibilities, try a few, and come up with one or more that will work.
These insight problems test convergent thinking, while claiming to help in locating divergent thinkers. Think twice before using these for anything but the clever amusements that they are.
September 19, 2009
You know those brain teaser questions that you see in popular magazines or newspapers under the heading, “Are You Creative?” These include the ever-popular nine dot problem, the word problems with surprising solutions, and mathematical problems like ones you hear Will Shortz offer up on the Sunday Puzzle on NPR. They may be fun, but when posed as tests of creativity, these little games have always irritated me, somehow, and just yesterday I figured out why.
I was reading an article by Probst, Steward, Gruys, and Tierney about how job insecurity can have adverse effects on creativity, yet increase productivity to a degree. It is an interesting article and I don’t doubt their results, but I do have a problem with how they measured the creativity of their participants.
They did two studies testing their hypothesis that, first, in a lab setting, participants who were threatened with losing their jobs would score lower on a creativity test than those who were not threatened. Second,in another study conducted in the workplace, a survey asked participants to respond to questions about how they felt about the security of their jobs, and then a creativity test (among other measures) was completed.
In the first study they used Duncker’s old insight problem about the candle, matches, and the thumbtacks. Coincidentally, I had recently heard Daniel Pink talk about that one at Chautauqua, and it bothered me then, as well. But I couldn’t put my finger on why at the time.
In the second study they used the RAT for measuring the creativity of the participants. Both of these measures are old and well-established metrics for identifying creative behavior or creative potential. But now I know why I always have felt somehow unsatisfied at the prospect of measuring creativity in this way.
Whenever I’m confronted with the 9 dot problem, for example, I have to really strain to remember the right answer. I’ve seen it so many times that I find it really boring, but in spite of this, I always forget how to solve it. I like doing RAT type problems more, since I am more verbal than figural, but still it is annoying if I miss getting that right answer.
Do you notice anything in the previous paragraph that flies in the face what we know about creative behavior? Right! Finding the “right” answer! This is a convergent task that pretends to measure creative ability. We all know that creativity is about considering multiple options, many of which might work together in a great combination of pointers that will let us solve a problem. It isn’t about being able to do well on Puzzler questions.
In the real world, we are not put in a room and told to solve a tricky problem as fast as possible, and by the way the timer is running. Even in jobs like being an emergency room doc, or a pilot in a dangerous environment, where performance requires quick thinking and the stakes are high, people are in a context where they have accumulated much domain skill that allows them to quickly access the many options that they have and select the one or ones that they will use. They can do this in an unpremeditated, almost intuitive way, but they are at some level scanning their options and selecting among them.
The obvious creativity of Captain “Sully” Sullenberger, the airline pilot that landed his plane in the Hudson River made a quick decision to avoid crashing into a populated area, involved his using an uncannily rapid scan of his behavioral repertoire and choosing the best alternative. He deserves much credit, and he is undoubtedly a highly creative person. I’m not sure, though, that that means he can quickly solve the 9 dot problem or the candle and tacks problem (unless he had seen them before and cared enough to remember them.) It seems a trick, somehow, to ask people to do that.
If we want to see how people are creative, we should ask them to show us what they have done. We could consider a portfolio of writing or art or design. We could ask them to bring in something cool that they’ve made, and ask them to tell us about it. We could ask them to show us some of the ads they’ve created, or the patents that they’re most proud of. Not to count them up, but to be witness to the products of their creativity. With creativity, the proof is in the pudding. Not in inventories of accomplishments or personal achievements or in solving on-the-spot parlor tricks.