EPPP Test Construction

Refers to the extent to which test items contribute to achieving the stated goals of testing.
Factors that influence:
Content Appropriateness - Does the item assess the domain it is designed to evaluate?
Taxonomic Level - Does the item reflect the appropriate cognitive level?
Extraneous Abilities - Does it assess other abilities?

Passing examinees/total examinees. P ranges from 0-1. .5 for most tests. .75 for T/F tests.

The extent to which a test items discriminates b/t examinees with low and high scores. D=U-L. Ranges from -1 to 1. Discrimination = Upper group-Lower group. (Most test look for .35 or .5)

In item response theory, the curve is made for each item and assesses the probability that the respondent will get the item correct. Diff level, discrimination, and guessing are accounted in the ICC.

An ICC provides 1-3 pieces of information about a test item – its difficulty (the position of the curve (left versus right); its ability to discriminate between high and low scorers (the slope of the curve); and the probability of answering the item correctly just by guessing (the Y-intercept).

X=T+E

A correlation coefficient (ranging from 0-1) which assesses true scores vs error scores. Interpret directly; no need to sq. Increased when similar items are added, the range of scores is unrestricted, and guessing is reduced.

Known as the coefficient of stability. Good for tests that are relatively stable over time and not affected by repeated measurement.

Good for tests that are stable. Most thorough method for estimating reliability.

Not appropriate for speeded tests. Split-half & Coefficient alpha are 2 types.

Two halves. Must have enough items or power is low. S-B helps when low.

Estimates reliability on short split-half samples.

Method of assessing internal consistency reliability (i.e., special formula) that provides an index of average inter-item consistency.

A substitute for coefficient alpha when test items are scored dichotomously (right or wrong).

Diff raters assessment. Assessed using kappa statistic.

SD times the Sq rt of 1-reliability coef. Makes the confidence interval. Ex: 68% confidence interval would be one standard error on both sides of the actual score.

The extent that a test adequately samples the content that it is designed to measure.

The extent that a test measures the hypothetical trait.

Methods to assess construct validity (multimethod-multitrait or factor analysis).

"Mono" means same and "Hetero" means different. Same trait-diff methods coeff are large = convergent validity. Diff traits-same method coff are small = discriminant validity.

Identifies the min # of common factors accounting for a set of tests. Factors can be sq. Communality is the % of accountability by the factors.

FA rotation resluting in seperate or uncorrelated factors.

FA rotation resulting in similar or correlated factors.

When test scores are to be used to draw conclusions about an examinee's likely standing an another measure.

When criterion data are collected prior to or during the predictor.

When the criterion data are collected after the predictor.

Index of error when predicting criterion scores from a predictor score. Uses Criterions SD and predictors validity coef. SD tiems the sq rt of 1 - validity coefficent

The extent to which a predictor increases decision-making accuracy. (Positive Hit Rate-Base Rate)

ID by predictor; meet criterion

ID by predictor; do not meet criterion

No ID by predictor; do not meet criterion

Determines if a S is +/-

No ID by predictor; do meet criterion

Determines of a S is T/F

True Positives + False Negatives/Total #

True Positives/Total Positives

When a criterion rater knows the predictor score.

...shrinkage may occur.

Norms must be derived from individuals w/ similar characteristics to be valid. Norms become obsolete quickly.

...high reliability does not always mean high validity.

Nonlinear. Distribution is always flat regardless of the shape of the raw scores. Disadvantage is that it is an ordinal scale.

Interpreting against a predetermined standard using either a percentage score or criterion score from an regression eqution and expectancy table.