Assessment Challenges

• The "Perfect Data" Fallacy – False belief that the perfect measure is out there if only we could obtain it, but evaluation measurement is imprecise.
  
  
• The "Single Indicator Fallacy" – False belief that if we have only one measure then we have the truth. No single assessment technique can provide all the information needed to evaluate a program or to assess student learning. You must use multiple measures and several sources of data. 
  
  
• "Data Lust" Fallacy—False belief that only numerical data are really informative. Sometimes institutions pile up as much data as they can form as many sources as they can without giving adequate attention to how the data is used. Generally, the greater the amount of data collected and analyzed, the higher the cost of assessment.
  
  
• Direct vs. indirect effects 
  
  
• Ceiling effects and Regression effects (toward the mean) 
  
  
• Population effects (older, part-time, and low SES are often missing) 
  
  
• Selecting the appropriate statistical analysis 
  
  
• The "I’ll Remember Making Those Changes" Fallacy – KEEP GOOD RECORDS

KEY DECSIONS IN CHOOSING ASSESSMENT MEASURES (NCTLA)

• Choosing between Quantitative & Qualitative Measures
  
  
• Identifying how much measures overlap
  
  
• Finding which measures explain the most
  
  
• Determining the reliability of measures
  
  
• Choosing between assessing outcomes & overall academic achievement

HOW TO ASSESS: USE MULTIPLE MEASURES

• Standardized Testing
  
  
• Locally-designed Approaches
  
  
• Individualized Tailored methods
  
  
• Tracking Student Behavior

CHOOSING BETWEEN QUALITATIVE AND QUANTITATIVE MEASURES (NCTLA)

TECHNICAL CONCERNS

• Format of initial data sets and transferable media 
  
  
• Flat files, spreadsheets, fixed or variable record lengths, diskette, compact disk, e-mail, snail-mail, Internet, bulletin board, etc… 
  
  
• Hardware/software requirements, limitations, and capacity. 
  
  
• RAM and hard disk requirements, memory address conflicts caused by external devices such as CD players, file conversion and editing capabilities 
  
  
• Missing data: Identification, correction, estimation, or allowance 
  
  
• Data reduction and conversion 
  
  
• Collapse and recoding of categorical measures when appropriate, derivation of new measures. Data conversion to appropriate level i.e., normal level data converted to dummy or effect coding for analysis purposes, reverse coding of negatively worded items. 
  
  
• Keep good records of changes to measures and any data transformations 
  
  
• Doing and redoing the analysis 
  
  
• Univariate analysis
• Bivariate analysis
• Multivariable analysis

| Table of Content | Assessment Challenges and Issues | Previous | Next | Home |