Experimental Designs and Psychometrics

Introduction : Definition, purpose, choosing an experimental design

- carry over effects, individual differences; design problems

Types of Research Designs: Between subject designs – characteristics, advantages and disadvantages, application & statistical analysis Randomized group design, randomized block design (single subject in each cell); Factorial Experiments : two factors (equal cell frequencies) three factors (equal cell frequencies, Homogenity of variance (Hartley’s test)

Within Subject Designs : characteristics; Advantages and disadvantages; application and statistical analysis – single factor, two factors, trend analysis, mixed design : characteristics, advantages and disadvantages, application and statistical analysis – 2 factors

The assessment of individuals: Measurement in the social sciences; Historical highlights of measurement; statistical background – scales of measurement, the normal distribution, probability and statistical significance, sampling distributions, correlation, linear regression, score meaning; Identification of construct – links between constructs, construct cleanliness, single vs multiple construct

Designing and writing items: Empirical, theoretical and rational approaches to item construction, assessing behaviour – critical incident technique, plot testing, summary and next step.

Designing and scoring response: open-ended responses; close ended questions; Dichotomous responses; multiple choice tests- distracters, guessing, speed and power tests, omitted and partial credits; continuous responses – summated rating scales, likert-scale, visual analogue scales, pictorial, adjective rating scales.

Item analysis : Meaning steps; classical test theory, modern test theory

Reliability of test scores and test items : Test –retest reliability, alternative forms reliability, measures of internal consistency - split – half, cronbach’s alpha, coefficient theta, Kuder Richardson.

Assessing validity using content and criterion method : Asking the test takers and subject matter experts, assessments using correlations and regression; multiple criteria, group differences and test bias.