Sunday, February 3, 2013

My Relationship Status with Data is "It's Complicated"

My relationship with Data goes way back.  We have had some good times, and some bad times, but it is officially time to set aside our differences and begin to work things out.  Let me explain.

As a math and science person at heart, I particularly find inquiry, research, and data very interesting and exciting.  However, I only find it exciting and interesting when I personally believe that my inquiry and investigation is meaningful and authentic and will help me to diagnose problems, or help me to solve problems that have already been diagnosed.  Participating in data analysis and statistics to which I find to be a waste of time, does not excite me, naturally.

Our Problem

While I find statistics to be a very powerful tool, I truly feel that statistics should be used appropriately for things that matter.  The Merriam-Webster Dictionary defines statistics as “a branch of mathematics dealing with the collection, analysis, interpretation, and presentation of masses of numerical or quantitative data”.  The key words to highlight in this definition are numerical and quantitative.  I find that numbers are often not the best way to assess learning.  I personally value qualitative data over quantitative data because I believe that qualitative data helps to paint a more detailed picture of the information gathered.

In other words, I often feel that Data has a warm, subjective human side which is qualitative in nature, but can also portray a more cold, objective, quantitative side. This split personality is not helping our relationship.

As an educator in today’s society, I find it disheartening that so much of our time and effort is spent on analyzing state standardized test data.  I honestly do not believe that state standardized tests are an accurate measure of student learning and understanding.  I find that they are mediocre at best.

To help me elaborate, I will borrow a sample of text from the book Shaping School Culture.  In their efforts to shape school culture,  Deal & Peterson (2011) argue that:

Scores on standardized tests are becoming the only acceptable evidence that students are learning what they need to know.  The narrow gauge of achievement tests captures only a small slice of what students take away from their time at school.  As a result evidence of success is spotty, tied at time to ethnicity or socioeconomic standing that students bring with them rather than important learning that may be taking place.  The cry for tangible evidence and the relative lack thereof undercuts confidence or faith in schools.  Over time it erodes teachers’ belief in their ability to make a difference.  What they once hoped for in a career falls victim to a constructive version of what education means.

In my opinion, having teachers analyze state standardized test scores (to which we are ill-equipped to do effectively, unless we receive proper training and support) to guide our instruction is truly a broken system.  I find this to be a broken system because I find the data to be extremely unreliable due to Campbell’s law.

Campbell’s Law states that "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor" (Wikipedia 2013).  In other words, having both high stakes and high validity is not possible. When there are high stakes, data becomes unreliable.  We simply can't have our cake, and eat it too!

Therefore, even in our best efforts as educators to analyze state standardized test scores effectively and accurately (as our current education system values so much) we are simply making decisions based on unreliable data which has not proven to significantly help student achievement.  Analyzing state standard test results might prove to increase test scores, but I have yet to find evidence that supports increased test scores correlate to increased student achievement and success.

By focusing our attention only on “content knowledge” we are lacking to foster and evaluate 21st century workplace readiness skills such as creativity, critical thinking, communication, and collaboration.  We are failing to assess and evaluate student inquiry, research, application, and innovation.  We are too focused on the “content” that students are learning. What's worse is the fact that we are currently operating on a broken system to evaluate this “learned” content.

My Proposed Solution

I do feel that data analysis is a very important part of teaching and instruction in order to create differentiated learning opportunities for our students so that we might best meet their individual learning needs.  However, we need to stop focusing on analyzing standardized test scores and start analyzing other data collected inside and outside of the classroom such as our own authentic assessments, formative assessments, observations, conversations, and student reflections, etc.

Keep in mind, that I did not say that we should abolish analyzing state standardized test scores.  I am simply suggesting that we need to have more focus on other types of assessments and data collection to help us paint a broader and more detailed picture of our students' learning and understanding.

In closing, while we will not be able to completely disregard the data from state standardized tests, we can collectively start focusing our time and energy on the types of data that are best for our individual students, and not what is best for our individual institutions!

Relationship status update ... back "In A Relationship" with Data.

You might also like

Rethinking Assessment
Leveraging the Strengths of Our Students
There is More than One Way to Climb a Tree
Keep it up America! We are Doing Something Right!
What Data Should Drive Data-Driven Decision Making?


  1. I hear what you're saying about quantitative testing/ high stakes tests - but, in a way it's starting to get cliche how educators are so quantitative data phobic.

    I'm glad I spent some time in the private sector engineering field before I became a teacher. The companies that are successful are fanatical about data and numbers. They don't make million dollar decisions based upon "qualitative" data.

    Do I believe teachers should be evaluated based upon standardized tests? No. But, not because quantitative tests are bad, but more because the standardized test have to many uncontrolled variables to isolate it just on the teacher.

  2. Jeremy, thank you so much for your comment. I really appreciate it! To be clear, I truly do value data in all forms. However, in terms of learning (which is so complex) I find it extremely difficult to measure it.

    Quantitative data, at best, is just a representation of student learning and understanding. Just like the famous pipe painting, the painting of a pipe is just a representation of pipe. The painting itself, will never be a pipe. The data we collect and analyze is just a representation of student learning and understanding. Quantitative data (in my opinion) cannot truly and effectively measure student learning and understanding.

    For example, what happens when a student doesn't know a specific word in a question that is vital to aswering the question correctly? If a student answers this question incorrectly, is it because she does not know the content, or that she did not fully understand the question that was being asked? There is no way to analyze this as a multiple choice style response. We can do our best at making inferences from the data, but there is no way to know for sure. It just tells us that the student did not select the "correct" answer.

    The multiple choice format is unfortunately the easiest, and quickest way to "attempt" assessing student learning and understanding. However, we shouldn't focus on what is easiest or quickest for us as educators to assess, we should be evaluating what assessments will best help us to evaluate and assess student learning and understanding.

    I truly believe that quantitative data is excellent at measuring and assessing other categories of information, but "learning" is just too complex to be measured accurately in the way that it is being assessed with our current standardized tests.

    So, our task as educators is to figure out the best possible ways to "paint" a representation of our students' learning and understanding. And I am convinced that (in its current state) state standardized testing is a sloppy, finger-painted representation at best.

  3. Thanks for sharing this informative post.