Interobserver Agreement

Fiss, J. L. (1971). Measure of the scale rated correspondence between many advisors. Psychological Bulletin, 76, 378-382. Shrout, P.E., Spitzer, R. L., Fleiss, J. L. (1987). Comment: Quantification of compliance in the resumed psychiatric diagnosis. Archives of General Psychiatry, 44, 172-178.

J. Cohen: Cohen. A coefficient of agreement for nominal scales. Educational and psychological measure, 20, 37-46. The evaluation of an inter-observer agreement is essential for the quality of the data in time and movement studies. A procedure to improve the credibility of the data by comparing the independent observations of two or more people from the same event. The IOA is calculated by calculating the number of agreements between independent observers and divided by the total number of agreements plus disagreements. The coefficient is then multiplied by 100 to calculate the percentage (%) Consent. Berk, R. A.

(1979). Generalization of behavioural observations: a clarification of the Interobserver agreement and the reliability of the inter-observer. American Journal of Mental Deficiency, 83, 460-472. Suen, H. K., Lee, P.S. (1985). Impact of the use of a percentage agreement on behavioural observation: a reassessment. Journal of Psychopathology and Behavioral Assessment, 7, 221-234.

Langenbucher, J., Labouvie, E., Morgenstern, J. (1996). Methodological evolution: measurement of the diagnostic agreement. Journal of Consulting and Clinical Psychology, 64, 1285-1289. Hartmann, D. P. (1977, Spring). Reflections in the choice of the reliability estimates of inter-observers. Journal of Applied Behavior Analysis, 10, 103-116. The Inter-Observer Agreement (IOA) is an important aspect of data quality in clinical work time and movement trials. To date, these studies have used simple and ad hoc approaches to the evaluation of IOA, often with minimal reports on methodological details. The most important methodological questions are how task time-stamping intervals are aligned, which rarely have start and end times, and how the IOA is evaluated for several nominal variables.

We present a combination of methods that addresses both problems at the same time and provides a more appropriate measure for the evaluation of IOA for time and movement studies. The problem of orientation is addressed by converting task-level data into small time slots and then indexing data from different observers over time. A method for multivariate nominal data, the Iota score, is then applied to time data.