Check out our podcast!

Share on facebook
Share on twitter
Share on linkedin
Share on email

The Future of Assessment Depends on Elevating Culturally-Diverse Leadership

Delivered by Susan Lyons, Executive Director of Women in Measurement, at Beyond Multiple Choice's April 2021 virtual seminar.

You can also listen to this presentation on our podcast or watch it on our YouTube channel.

My name is Susan Lyons and it’s my pleasure to be kicking off this seminar and speaking with you today about the imperative we face as a field to amplify diverse voices. 

Celebrating contributions made by people of color

People of color, and other minoritized individuals, have a long history of leadership and impact on the field of educational measurement. As a shining example, professor Edmund Gordon, who is celebrating his 100th birthday this year has been a visionary in our field for more than 60 years, pushing us to envision a future of assessment that more directly serves teaching and learning. 

An equity crisis in the field of assessment

However, despite powerful examples like professor Gordon, the current state of the field in educational measurement is overwhelmingly white. 

Recent research published this spring from Randall, Rios, and Jung sheds light on racial disparities. In the field between 1997 and 2016, a disproportionately small number of doctoral degrees in educational measurement were awarded to Black and Hispanic students.

Just 6.8% were awarded to Black students and only 4.4% for Hispanic, while the U.S. Census estimates that Black and Hispanic individuals compose nearly a third of the population. 

These numbers are strikingly low, even accounting for the disparities that we know exist in earning bachelor’s degrees. in 2016, 13.7% of students enrolled in degree programs at post-secondary institutions were Black, while only 5% of students earning a doctoral degree in educational measurement identified as Black. 

This pattern is similar with Hispanic students. 

Even more shameful is that there were so few graduates to identify as native American multi-race or Pacific Islander that Randall Rios and Jung were unable to disaggregate data for these groups in their research. 

The researchers also looked at outcomes for ethnic minorities who do graduate from PhD programs in our field, and sadly found a statistically significant interaction effect between race and employment status post-graduation. That is, significantly larger percentages of white students reported offers of employment than their Black or Asian counterparts. 

These statistics seem especially problematic in light of the reality that the populations of students taking educational assessments are increasingly diverse.

Assessment lags behind progress made in curriculum and instruction

2014 marked the first year in which children of color comprised the majority of students in U.S. public schools. The fields of curriculum and instruction are rapidly changing to reflect the latest science of learning that emphasizes the central importance of culture and context. 

Curriculum and instruction are evolving to embrace movement and culturally responsive and culturally sustaining pedagogy, ethnic studies, and anti-racist curriculum reforms. However, we know assessments just aren’t there yet. The field of educational measurement is at a point of reckoning. We are grappling with the roles of assessment in both perpetuating and interrupting systemic oppression in our society. 

These snapshots are from two public forums held this spring by the National Council on Measurement in Education and the National Academy of Education, at which leaders advocated for more critically examining the role of assessment as a force for advancing, or hindering, racial justice. 

As a field, we are rethinking how our standards of practice, our measurement models, and our interpretations might change when we critically interrogate them with a racial justice lens. One of the potential problems for moving the field forward may be that the people who hold the most influential positions in our field continue to be overwhelmingly white and male. While this is not unique to educational measurement, there are likely specific undesirable effects on our assessment practices and products when the decision makers are almost exclusively members of the dominant culture. 

Consequences of cultural homogeneity

Two of the possible negative effects of a lack of cultural diversity in decision making may be: 1. the adoption of professional practices and values that reinforce existing social structures, and 2. assessment designs that center white culture and privileged dominant groups. 

Scholars Moss, Pullin, Gee, and Haertel remind us that the “beliefs and practices informed by psychometrics have become so ingrained in the American educational system that it’s become difficult for us to see them as choices arising in a particular sociocultural circumstance, or to imagine that things could be different.”

I like to interpret this as saying: “had our field leveraged the benefits of cultural diversity in leadership from the beginning, it’s highly likely that differences in our accepted professional practice and values would have emerged. Things would be different.”

Equity-minded scholars have recently pushed up against our current professional practices and values and argued for re-conceptualization.

For example, in their recent research report, doctors Jennifer Randall, and Kristen Huff confront the white hegemony in our definitions of construct validity, and in the methods we use to support them. They argue that the way we have operationalized our measured constructs privileges whiteness, and that our desire to minimize construct irrelevant variance in our measurement models may actually be wrongheaded for an assessment system used with an a pluralistic society. 

Similarly in his 2020 CME presidential address, Dr. Steve Sireci calls into question our mechanisms for producing comparability primarily through standardization, arguing that the hyper-focus on standardization leads to exclusion in favor of those who could conform to the most dominant culture. Instead, he challenges us to think about how we might achieve greater comparability by creating flexibility in how students produce evidence of learning. 

As with our professional practices and values, it’s highly likely that involving diverse voices in test construction will introduce differences in the types of evidence of learning we privilege in our assessment designs. 

Forging a path to diversity and inclusion

It’s incredibly challenging to develop items that are culturally relevant for all students, especially when those leading test design and development are mostly members of the culturally dominant group. Designers of any kind have to make assumptions, and in the case of test design, those assumptions may be limiting the relevance of the assessment for students from nondominant cultures. 

Research has shown that culture has significant effects on how test-takers both approach standardized tests and on test-taking performance. This research creates an imperative in our field to fully interrogate all of the possible explanations for group differences in test outcomes, and to evolve our assessment design practices to keep pace with the field of learning and instruction that have shifted to more explicitly account for the role of culture. 

Luckily as a field, we seem to be increasingly recognizing the overdue need to amplify diverse voices.

Many organizations and leaders are working to address disparities in access to the field and in the promotion and retention of diverse talent. 

The National Council on Measurement and Education has funded a number of special projects supporting students of color in measurement programs and highlighting diverse perspectives, including the work of Darius Taylor with the Wrong Answer Project. 

Additionally, late last year I had the privilege of launching a nonprofit, Women in Measurement, with a group of dedicated colleagues who are passionate about advancing gender and racial equity in the field. 

The structural racism and misogyny that have led to disproportionate access and representation are certainly not unique to educational measurement, but it is going to take the energy of all of us to address the persistent inequities within our corner of the world. 

Not only must we elevate the voices and perspectives of people of color, but perhaps even more importantly, we must all commit to raising our collective critical consciousness within the field so that we can effectively grapple with the most pressing issues we face in educational measurement. 

I’m looking forward to the opportunity for the upcoming question and answer session, but I certainly do not have all the answers. So I’ve posed some of my own discussion questions here for us all to discuss. 

If you have more questions, my contact information is provided. Thank you.

Discussion Questions: 

  1. What opportunities do you see for elevating diverse leadership within the field of educational measurement? 
  2. What do you see as the most critical lever for advancing racial equity within educational assessment? 
  3. How will the future of assessment better attend to the sociocultural nature of learning? 

Twitter: @susanclyons