Investigation of Coefficient of Individual Agreement in Terms of Sample Size, Random and Monotone Missing Ratio, and Number of Repeated Measures

Abstract

Studies based on longitudinal data focus on the change and development of the situation being investigated and allow for examining cases regarding education, individual development, cultural change, and socioeconomic improvement in time. However, as these studies require taking repeated measures in different time periods, they may include various error sources that are exclusive of the investigated variable and that change over time. In Addition, the period of observation is critical in these studies, and observant (variable) attrition or the inability to obtain measurements for various reasons (death, moving away, resignation etc.) may be observed. This study presents the coefficient of individual agreement (CIA) developed for studies in which more than one rater are needed to provide repeated scoring and examines the applications of this agreement coefficient in relation to different sample size, missing value and number of repeated measures. In this context, trial scores were generated based on different sample sizes, missing value ratios, and the number of repeated measures provided by two raters, and the expected agreement between raters was simulated as very high. Related data were replicated 1000 times. Data analysis included the mean and standard deviation of agreement ratios calculated for each combination. According to the results, independently from sample size and number of retests, the CIA takes much lower values than the expected value when random missing value rates reaches 20%. Similar conclusions can be drawn for the existence of monotone missing values except where the sample size is 30 and the number of retests are two and three. It was observed that the CIA was higher than the expected value for the data consisting of two repeated evaluations of the raters with 5% or 10% missing values when the sample size was 30. In general, the agreement coefficient did not exceed the expected level in the situation of three and four replications of measurements. As a result, it can be concluded that an increase in the replication number of measurement decreased the obtained agreement values. The obtained data showed that missing value ratio caused differentiations in the CIA. As a result, increases in missing value ratio cause inconsistencies in CIA. Therefore, it is suggested that data obtained from studies be interpreted considering missing value ratio.