craig dennis howard

Home » 2014 » February

Monthly Archives: February 2014

Using and abusing the GRE in graduate admissions

I preface this statement with an acknowledgement that my views here are intended to stimulate discussion at Texas A&M–Texarakana and do not apply to my feelings about the use of the GRE at other institutions.

I have been asked to revisit the Graduate Record Exam (GRE) requirement for admission into the ITED program at TAMUT. I have tried to address the issue with as much clarity as possible, and have concluded, after closer inspection, that the policy of using the GRE as part of admissions criteria is not only inappropriate for TAMUT’s ITED program, but is also potentially misleading in making admissions decisions. This statement lays out my rationale and a proposed replacement to the GRE requirement.

There have generally been four purposes of the GRE in the admissions process. Those reasons are:

GRE test support materials

The industry surrounding improving test scores for admissions committees that will likely never read them.

(1.)  As a minimum cut-off used in order to manage large numbers of applications for programs where the application pool is larger than department or universities’ resources allotted to review applications.

(2.)  As a predictor of success in graduate school in order to locate learners who may need additional support once admitted;

(3.)  As an additional point of observation to compare similar boarder-line admissions candidates;

(4.)  As evidence of sincerity in a candidate’s desire for admittance to graduate study. I will address each point separately.

1. A minimum cutoff to manage large application pools

The ITED program is not presently overwhelmed with applications. To make a statement that this is the purpose for the requirement would be unethical. In an effort to make our admissions as transparent as possible, we should exclude this purpose from our application statements. This purpose is also advised against by the test developers: “A cutoff score based solely on GRE scores should never be used as the sole criterion for denial of admissions.” (from the ETS website.) Using the test in this way is misuse.

2. As a predictor of success

The test does not assess the skills associated with the program, and in doing so, provides no relevant measure of potential success in the program. ITED is a design program, but the test measures other types of knowledge not sufficient in the process of design. Design programs train learners to uncover unforeseen solutions, while the test measures how well a learner can mimic the test creator’s logic. Furthermore, even for more strictly academic programs, the use of the test as a predictor has been widely discredited (This Chronicle of Higher Education article and dozens of others).  Previous studies show that the GRE only explains 6% of the variation of first year scores, and even has negative correlations with graduation rates (Onash 1994). Using the test to predict success in the ITED program hinders our vision of who might actually be the most promising candidates because even high scores will have no relevance to program success.

3. Identifying students who may need support after admittance

The GRE measures analytical writing, verbal (primarily English language manipulations) and high school mathematics. Apart from the skill of constructing coherent and complex arguments explaining rationale, most of the areas measured on the GRE fail to align with problem areas common to the ITED program—difficulties in understanding and applying instructional and learning theories, and the technical aspects of computing and digital design.  Since the GRE does not identify problem areas, another criterion would be more appropriate for this purpose. For example, learners who have difficulty with digital skills could be identified early on via a digital artifact. They could be required to take a TAMUT basic computing course or web design course as a condition of enrollment. Using the GRE weakens our ability to detect these deficiencies in preparation for graduate study in Instructional Technology.

4. Evidencing sincerity in desire for graduate study

Were it the case that all applicants had access to the same financial resources, this might be a plausible use of the GRE. However, having no other purpose, imposing this financial barrier on the population that TAMUT is situated to serve is suspect on a number of levels. The price of the test may be equivalent in dollar value to each learner, but in terms of percentage of resources the cost may vary dramatically from learner to learner. Thus the practice of requiring the GRE may favor more financially well off students, and obfuscate the institution’s vision of the most qualified students. This dynamic works against our mission to provide equitable access to learning, and on these grounds cannot be assumed to discriminate fairly among candidates.

An alternative requirement: A designed artifact

In comparison to other institutions, an equal number use and forgo the GRE as a requirement. Among those that do require the test, we do not know why they use it. I can only conclude that the negative aspects of using the GRE on admission requirements outweigh the positive aspects of requiring the test. An alternative measure might provide the additional level of granularity some programs are searching for in their requirement of completion of the GRE. Thus, an alternative may prove to be a more useful and appropriate means of assessment for admittance.

One possible alternative is requiring a designed artifact. Requiring a designed artifact holds the potential to address the skills we hope to enrich in the ITED program- applied creativity and innovation in creating learning. An artifact of design can clue us into a candidate’s ingenuity, their ability to take information and apply it to real world contexts, and a learner’s tenacity to complete a design task. The practice may avoid the caveats presented with using the GRE as a criterion for admissions. Requiring a designed artifact may weed out learners who have little or no concept that ITED is a program where the creation of materials for learning is the primary objective, thus narrowing the pool to those who have a concept and desire to study in the program for which they are applying. A designed artifact also directly aligns with the learning objectives in the program and in so provides a more appropriate measure of potential success. Unlike the GRE, a designed artifact may expose learners’ limits in relevant skill areas, thus pointing to better means of preparing learners for graduate study in Instructional Technology in the early stages. And finally, requiring a designed artifact is a non-prejudicial practice, discriminating on relevant experience and knowledge as opposed to financial resources.

References:

Onasch, C. (1994). Undergraduate Grade Point Average and Graduate Record Exam Scores as Predictors of Length of Enrollment in Completing a Master of Science Degree. ERIC Document No. 375 739.

Sacks, P. (June 8, 2001) How Admissions Tests Hinder Access to Graduate and Professional Schools. The Chronicle of Higher Education.

Addendum on 4/3/2012

I presented the argument above to the education faculty in March. Subsequently, through consultations with Dr Bill McHenry, an alternative course of action was decided– an interview with supporting rubric for scoring the interview. The tentative draft of the rubric for scoring is as follows:

Interview Rubric ITED 2014

On April 10, this rubric was tested with a real, live, breathing student. Our ratings came within .5 point of each other. We fumbled with various questions which brought the rubric into a reality within th context of the interview into a reality. Kevin and I met for an hour afterwards to hash out some of the questions we felt produced the most clear cut and score-able answers. Those refined questions are:

1a. Describe something you designed?

1b. Describe how you use technologies you use now—in your teaching or other creative things that you do.  

2a. Explain how you handled a technological change in your work or creative endeavor.

2b. Describe a poor technological choice or poor technological implementation strategy in a learning context. How would you have done it differently?

3a. Where you see yourself in the field of Instructional Technology? How does what you like to do match with the tasks people who hold positions in this field actually do?

3b. If accepted, how will this degree help you meet your professional goals?

Forbidden words in Instructional Technology

Expanded title might read, Why I forbid students to use certain words when they describe a design, and the list itself

This week a student asked me to explain the differences among discussion boards, discussion forums, and blogs. I paused because I knew my answer was actually a blog post, and not really what a student wants to hear. I wondered if my answer held value for her the same as I felt it holds for me. I am someone who appreciates an accurate description of a design, but not a lot of people do. Maybe that’s because design communication is one of those things that really only get appreciated when it breaks down. So I paused, and thought about it, then I gave her and the whole class the whole long-winded lecture (via the Blackboard asynchronous message feature) about why it’s important to describe designs well. Here’s what I said.

picture of forbidden words

Editor Hugo Lindgren’s list of forbidden words at the NY Times Magazine. His list is far longer than mine, and thus suggests to me that after all, I am not such a bad guy. Check out his fill list at the link provided at the end of this entry.

The hard and fast difference among discussion boards, discussion forums, and blogs is nothing. They are all slang words for asynchronous configurations, and at this point in time, have generally become so varied in specific manifestations, that the terms are slightly more valuable than total gibberish. So to me, they all signify the same thing– asynchronous communication that needs to be described more accurately to be understood. Blogs are supposedly reverse chronological journals of substantial length. But of course there’s Twitter, the quintessential micro-blog with a character lengthy of what, 150? Obviously not very blog-like. Then you have discussion forums that have traditionally been open-access, threaded asynchronous messages that are persistent—meant to be knowledge development venues that stay available pretty much as long as those who contributed to them might ever want to retrieve them or look something up. Our limited access Blackboard forums erase themselves two weeks after the end of the course, so they are really not forum-like either.  The varied asynchronous communication configurations on Blackboard are given a bunch of these random titles simply for the purpose of selling the LMS to faculty who have better things to worry about than what is really a blog, and what is really a forum. (Incidentally, there is nothing done on Blackboard that can’t be done on WordPress for free, and Blackboard costs 20,000$ a year, just to run the platform. These sales tactics work really well.) The inaccuracy of these words in relation to their less than uniform manifestations is exactly WHY I give my students a list of forbidden words. “Blog, forum, discussion board, etc…” are all terms that are way too inaccurate for an instructional designer to use on the job without long explanations of what they mean in a given case. These are sales-language terms and inaccurate tech-slang, not design terms. I tell my students that clients may use them in talking to you, but you should not use them when talking to your clients. As an instructional designer, you need to know more than these vague words to talk about online learning, technology enhanced learning, or technology integration. While it may sound like a very small point I am making here, we are actually talking about a lot of money when it comes to the job itself. Imagine a design needs to be amended or reconfigured because of a communication error between client and designer. You’re likely imagining serious lost time, like entire weekends spent recreating a design to fix the miscommunication. If you’re billing by the hour, you might save your client (20 hours X 50) 1000$. If you’re doing this once a month, you’re 12,000$ more valuable as an instructional designer than your peers, simply because you use the right words when you sit down and talk about a design, not to mention the fact that your clients are happier and giving you more work. I say call a spade a spade, don’t use forbidden words, use accurate design terms, or, in this case, call your high-tech spade with embedded communication for learning a “persistent, asynchronous skeuomorph, affording multiple channel, limited access, converged CMC.”

The forbidden words

  • Web 2.0 (as if this ever actually meant anything anyway)
  • MOOC
  • Blog (Some Blogs are reverse chronological, some are chronological, and some are not even weblogs thus making them not blogs at all. The term is so misused it is no longer meaningful when talking about design. It’s ok if you say “this is my blog” because you have identified a specific Blog. However, if you say “It is designed like a Blog” the listener could be thinking one of one hundred different things.)
  • Wiki (“openly editable webpage” is far more accurate for some platforms commonly called “wikis”, otherwise wikis have so many versions it is hard to imagine what you mean, exactly)
  • Homepage
  • Social Bookmarks / Bookmarking
  • All proprietary words are forbidden except as archetype comparisons (e.g. Youtube, Twitter, Facebook, Flickr, Instagram, etc…)
  • Podcasts & Vlog (these are slang terms for uploaded video, but actually are aligned with not design, but content genre. Avoid these terms unless you’re analyzing content)
  • Chat room
  • All proprietary words are forbidden (e.g. Youtube, Twitter, Facebook, Flickr, Instagram, etc…)
  • Cell phone (no one knows what a cell phone is anymore because the concept of “phone” is gone when we text on a phone but make calls on computers, aka via VoIP, “Skype.” Avoid saying “phone” because everyone seems to suddenly get confused.)
  • Good, bad, best, better, worst, great, nice, ok, interesting, and information. These are all vague and confusing.
  • Computer ( I add this because different types  abound and unless you’re talking about a new one, such as wearable computing, you’re probably using the term in a way that could be much more precise with another word.)

The recommended terms

I preface this list of preferred terms with, “Here are words that are typically employed by professionals and demonstrate an understanding of emerging technology for instructional design: (You should use all applicable terms here instead of the ones above.) If you do not know any of these words, look them up.”

  • Synchronous/Asynchronous
  • Open access, single access, limited access, password protected
  • Persistent
  • Vertical / horizontal array
  • Handheld device
  • Computer-mediated communication (CMC)
  • Converged media, converged media CMC (or CMCMC)
  • Social Network /  collaborative access
  • Character allowance
  • Editable
  • Website
  • Images, Image-texts, imagetext
  • One-to-many / one-to-one / many-to-one / user-to-interface-  communication types
  • Multi-user
  • Password protected /  access protected
  • Responsive / unresponsive or “fixed” design (means the display does not adapt to the device)
  • Design tensions (I cannot stress learning this term enough. All designers use this word, and if you don’t recognize design tensions, you’re not analyzing very well. They are there in every design.)
  • Customizable
  • Learner experience
  • Video sharing site
  • Annotatable
  • Presence indicator
  • cloud storage

BONUS TERM: skeuomorph

The picture is graciously stolen from a New York Times Magazine article about their forbidden words list, linked here.