Colorful Abacus Representing The Measurement Involved In Diversity And Inclusion Self-assessments

Understanding Self-Assessments: Part One

September 28, 2017
Diversity & Inclusion Measurement

Diversity and Inclusion Self-Assessments: What do you need to know?

Many are unclear on the importance of validity and reliability in a self-assessment. Is a dedicated self-assessment important when a few questions could simply be tacked on to an employee engagement survey?

In a world in which diversity and inclusion are gaining a sense of urgency, our hope is that we provide our readers with knowledge and understanding that can help advance organizations, managers and employees on a path toward a diverse and inclusive culture in which respect and inclusion are consistently demonstrated.

DEFINITIONS

Self-assessment: assessment or evaluation of oneself or one’s actions, attitudes, skills and beliefs. Most self-assessments are computerized or online, although some paper and pencil versions exist.
Individual profile: reporting of self-assessment results and using it to tell a story, or give an overview of self, and self in comparison to others.
Questionnaire: a group or sequence of questions designed to elicit information upon a subject, or sequence of subjects, from an informant.
Survey: an investigation about the characteristics of a given population by means of collecting data from a sample of that population and estimating their characteristics through the systematic use of statistical methodology.
Needs assessment: a process used by organizations to determine priorities, make organizational improvements, or allocate resources. It involves determining the needs, or gaps, between where the organization envisions itself in the future and the organization’s current state. Steps typically include: exploration and identification; data gathering and analysis; utilization (using the data to create a plan); evaluation (did actions taken close the gap).
Employee engagement survey: a tool for gathering feedback and enhancing workplace communication. Results are categorized according to the goals/values/ strategic objectives of each company. Typical engagement categories include: satisfaction; advocacy; retention; pride.
Validity: a term that psychologists use to describe the relationship between an answer and some measure of the true score. “The goal of the psychometrician and the survey methodologist is to make sure the error term as small as possible, so the answers mainly reflect the true score.”1
Reliability: the degree to which an assessment tool produces stable and consistent results. Test-retest reliability is a measure of reliability obtained by administering the same test twice over a period to a group of individuals.

Considerations

Note that these definitions occasionally overlap in their practical application. For example, both a self-assessment and a profile can be valid and reliable, although some are neither reliable nor valid. Although most assessment tools rely upon the answers from individual employees, one group of self-assessments are inward looking (individual profiles) and seek to understand and perhaps help the individual make behavioral changes. The other group of self-assessments look outward to the organization results or goals and seek to change the organization through the individual.

Clear and meaningful questions can help spot trends, inconsistencies, data patterns and all sorts of useful information about an organization, a team, or an individual. Questions that can be interpreted multiple ways, that use unfamiliar words, or that are ambiguous in some manner, can lead to false assumptions.

Rigorous question development is not a new concept. One of the early pioneers of question methodology was psychologist Rensis Likert. In use since the early 1930s, the Likert Scale uses a scale of possible responses. It can be used in multiple situations, including children and ESL individuals. The good (happy face) response is typically on the right end of the spectrum in Western cultures (reading left to right). diversity and inclusion self-assessments

In the diversity and inclusion field, the Likert Scale options would be used to measure opinions, feelings or beliefs.

Strongly disagree
Disagree
Neither agree nor disagree (or neutral)
Agree
Strongly agree

When the middle option is removed, it is called a “forced choice” method, since the neutral option is removed. Using the forced choice option (four choices) does not seem to impact results either positively or negatively.

Another Likert Scale option is a frequency choice, which asks people to look at their behavior and respond by how often they act or react in a certain way. Use a frequency scale when measuring behaviors.

Almost Always
Frequently
Sometimes
Rarely
Almost Never

A rigorous set of questions could also include questions that use reverse scoring in which a negative behavior is stated and the “good response” would be “almost never” rather than “almost always.” For example:

“Personal characteristics, such as race, gender, age, etc. hinder an individual’s career progression in this organization.”

In the example above, five points would be awarded to “almost never” and one point awarded for “almost always.” Points are assigned to the Likert Scale for the purposes of data analysis.

“In many spheres of human endeavor, from science to business to education to economic policy, good decisions depend on good measurement.” ~ Ben Bernanke

Less is More

Just like Goldilocks and the Three Bears, there seems to be a “just right” number of questions to include. Three questions added to an employee engagement survey is too few. Seventy-eight questions are probably too many. With too many questions, the survey participants may not finish answering the questions, or may not even start it if it looks like it will take a long time. The more a survey is set up with a similar repeated task (i.e. Likert Scale questions), the more likely the survey will be completed.2

Example: The U.S. Census asks 10 questions
Example: Spectra Diversity and Inclusion Profile™ uses 8 questions to measure Interpersonal Skills related to diversity and inclusion.

There are certain “should” and “should not” guidelines for question construction.3 Questions should:

Mean the same thing to all respondents (clear language, avoid colloquialisms, consider ESL respondents)
Evoke the truth (the deeper meaning you are seeking to uncover)
Be written at an 8th grade reading level to include all employee levels and ESL employees
Be able to be answered in only one dimension (one topic per question only)
Accommodate all possible answers (each respondent should be able to comfortably select a response from the Likert Scale)
Follow comfortably from the previous question

Real Examples:

“Management shows that diversity is important through their actions.” Succinct. Clear. One dimension measured. We like this one a lot.
“I understand what diversity and inclusion means at our organization.” Clear and succinct. Notice it is specifically asking what it means “at our organization” and not in the world, or in popular culture.

Questions should not:

Have mutually exclusive options (contain two concepts which cancel each other out)
Imply a desired answer (no leading the witness your honor)
Use emotionally loaded or vaguely defined words
Use unfamiliar words or abbreviations
Ask for a rating of 1-5 because simple numbers may not be clear enough. Is 1 a “never” and 5 an “always”? Or the other way around? And does a 3 mean, “I don’t understand the question”?
Depend on responses to previous questions (you should be able to randomize the order of the questions)

Corporate social responsibility is measured in terms of businesses improving conditions for their employees, shareholders, communities, and environment. But moral responsibility goes further, reflecting the need for corporations to address fundamental ethical issues such as inclusion, dignity, and equality.” ~ Klaus Schwab, German Economist

Real examples: diversity and inclusion self-assessments

“I have friends from a variety of ethnicities and abilities.” This has two variables. If someone has friends of varying abilities, but only one ethnicity, how would the question be answered?
“I don’t believe that my having a friend of color means that I’m culturally competent.” The phrasing in this one is unclear. Is “don’t believe” a good thing or a bad thing?
“Our organization has developed processes to hold all levels of leadership accountable for their support of the Company’s diversity and inclusion efforts.” This is a properly worded question, although it would be difficult for a regular employee to answer. How would a mid-level employee know if all levels of leadership are being held accountable? And would they know the processes in place?
“Our organization’s diversity/multicultural board committee meets consistently, has established meeting objectives, and regularly provides progress reports to the board of directors and/or senior management.” Too many variables in this one. One variable per question.
“Our organization is happy that it has an accurate and up to date awareness of the diversity profile of our beneficiaries and stakeholders.” Can an organization be happy? How does one know this? How is it measured?
“The profile of our employees appropriately reflects the diversity of our beneficiaries and stakeholders.” Once again, unclear. What does profile mean (age, gender, race, abilities, income level)? What does “appropriately” mean?

TIP: Avoid “don’t know” types of questions unless it is a factual question. Questions with a “don’t know” option produce a lesser volume of accurate data.

Process – Process – Process

Unlike item presentation which should be asynchronous, and could be randomized, item development should proceed in a sequential order or specific manner. Each step in the flow of the question creation depends upon the successful completion of all the previous steps. Some actions could loop, as additional research and testing is done. The process looks like this:

diversity and inclusion self-assessments

For example, the Spectra Diversity and Inclusion Profile began with research by experts and a list of 120+ questions that were presented to a diverse pilot group of 750 people. Some of these questions were reverse scored. The sample was statistically analyzed and revised to yield the final question set. A portion of the pilot group was given the profile a second time, and results compared to review for progress.

This concludes Part One of this article. Part Two discusses: