Article

Academic Engagement and Learning Outcomes of the Student Experience in the Research University: Construct Validation of the Instrument

Yunhee Bae¹ Sunyoung Han²

¹Sungkyunkwan University, Korea.

²Sungkyunkwan University, Korea.

Abstract

The Student Experience in the Research University is an online survey instrument that serves to assess institutional functions of research universities in the U.S. and understand students’ behaviour, satisfaction, and achievement. The present study explored measurement models describing two primary domains (i.e., Academic Engagement and Self-Reported Learning Abilities), and examined the adequacy of the measurement models. Exploratory factor analysis was initially conducted to identify the number of structural components of each domain, which was followed by confirmatory factor analysis specifying a factorial structure and verifying how well the structural model fits into the collected data. Six and three factors were extracted from theacademic engagement and the self-reported learning abilities domains.The structural model verified indicated that the relationships among the academic engagement and self-reported learning abilities domains and covariates such asgrade point average, parental income, and education were consistent with the findings from relevant literature.The findings of the current study contribute to the understanding of student engagement in research universities in the U.S. Further, the implications for educational research and practice were discussed.

Keywords: Parental income • education level • self-reported learning abilities • research university • grade point average

For decades, the question of how to improve educational quality and standards in colleges and universities has been a common concern among the majority of the population including educators, employers, and policy makers (Brint, Cantwell, &Hannerman, 2008; Hu & McCormick, 2012; Humphreys & Gaston, 2015). Higher education stakeholders and government entities have made considerable efforts to improve college students’ competence in the global environment (Humphreys & Gaston, 2015). Firstly, such stakeholders must better understand both how students use their resources such as time and energy, and what students learn during their undergraduate studies. Once they possess this data, they can substantially endeavour to create effective policies and programs to boost students’ learning activities toward meaningful outcomes and help for their success in their professional and social environments (Hu & McCormick, 2012; Kuh, 2003). To this end, college educators, researchers, and policy makers have been focusing on more affordable, reliable, and valid approaches to obtaining the information about college students’ learning experiences (Carini, Kuh, &Klleint, 2006; Humphreys & Gaston, 2015).

Of the critical constructs related to college students’ learning and school lives, academic engagement is key to explain how they devote themselves to their college education. Furthermore, academic engagement is generally perceived as one of the better indicators of learning outcomes (Carini et al., 2006; Kuh, 2003), so resolutions to improve student’s educational experiences in universities have been widely considered as a judgement of college quality (Brint et al., 2008; Kuh, 2003).

Academic engagement is a multifaceted term that refers to how much students use of their resources such as time and energy during their undergraduate degree and how eager they are to learn during their college years (Fredricks, Blumenfeld, & Paris, 2004; Kuh, 2003). As the concept of engagement has been defined using various terms (e.g., academic engagement, educational engagement, school engagement, student engagement), the construct of academic engagement has also been discussed in diverse forms by scholars in previous studies (Alrashidi, Phan, &Ngu, 2016). Studies on the constructs of engagement (e.g., Appleton,Christenson, Kim, &Reschly, 2006; Fredricks et al., 2004; Jimerson, Campos, & Greif, 2003; Reeve & Tseng, 2011) commonly illuminate the construct with three dimensions: behavioural, cognitive, and emotional engagement. Behavioural engagement includes positive conduct adhering to classroom norms such as preparation and time use in class (Christenson et al., 2008; Skinner, Kindermann, &Furrer, 2009). Cognitive engagement refers to students’ investment in learning and willingness to expend their efforts toward task mastery, challenge preferences, and self-regulation (Appleton et al., 2006; Fredricks et al., 2004; Jimerson, et al., 2003). Finally, emotional engagement involves reactions to teachers, classmates, academics, and school (Reeve & Tseng, 2011).

Not surprisingly, these dimensions do not operate separately. Because they mutually and dynamically influence each other, engagement literature has failed to differentiate the definitions (Alrashidi et al., 2016). For instance, behavioural and cognitive engagement might share the indicator of effort, which should consider the distinction between effort toward merely fulfilling behavioural expectation of class norms (i.e., behavioural engagement) and toward mastering contents (i.e., cognitive engagement). Consequently, many studies on the concept of engagement have addressed only one or two of the three dimensions of engagement, which hinders investigating the distinctions among the types of engagement and also blurs the relationships among those types, learning outcomes, and external factors (e.g.,grade point average, family backgrounds) (Fredricks et al., 2004).

Meanwhile, higher education literature has suggested that learning outcomes should be a substantial agent in evaluating the quality and effectiveness of colleges and universities (Douglass, Thomson, & Zhao, 2012). Of various approaches to estimate students’ learning outcomes, surveys of students’ self-reports have been employed with assumptions of reliability and validity (Carini et al., 2006). Learning outcomes refer to students’ abilities demonstrated as consequences of academic engagement, assuming multidimensional components in higher education (Phillips, McNaught, & Kennedy, 2010; Zlatkin-Troitschanskaia, Pant, & Coates, 2016). Allan (1996) specified the construct of learning outcomes with three dimensions in higher education practice: (i) subject-based knowledge and skills; (ii) personal transferable abilities to gather and use information, to communicate effectively, and to work with others; and (iii) generic academic outcomes such as thinking critically, and analysing and synthesizing ideas and information.

Higher education literature has provided several responses to the demand for real parameters on college students’ productivity (Carini et al., 2006; Douglass et al., 2012). Of them, Student Experience in the Research University (SERU) is an online survey instrument created by the Centre for Studies in Higher Education (CSHE) at the University of California, Berkeley. Shortly after the establishment of the SERU, 12 major U.S. research universities formed a consortium to administer the survey and share results. However, few studies have explored the detailed dimensions of academic engagement and learning outcomes in the SERU. Therefore, encompassing multidimensional concepts of academic engagement and learning outcomes, this study pursued the construct validity of measurement models with specific factor structures for academic engagement and learning outcomes, which also allows the detection of unique contributions to students’ learning as shown by the SERU (Alrashidi et al., 2016; Fredricks et al., 2004).

Academic Engagement and Self-Reported Learning Abilities

Precise and understandable information about student learning engagement and outcomes in a college institution is necessary for estimating the quality of education provided by the institution.Academic Engagement(AE) generally refers to students’ educational practices that facilitate learning, and institutions use policies and practices to attract students to participate in educationally meaningful activities (Brint et al., 2008; Kuh, 2003).

The College StudentExperiences Questionnaire (CSEQ) and its successor, the National Survey of Student Engagement (NSSE) have been the most common instruments in studies on AE for college students in the U.S. (Brint et al., 2008). Self-assessment of AE indicates how much time and energy students dedicate to educationally purposeful activities (Carini et al., 2006). AE was specified in NSSE as having five dimensions (i.e., Active/Collaborative Learning, Student-Faculty Interaction, Level of Academic Challenge, Enriching Educational Experiences, and Supportive Campus Environment). LaNasa, Cabrera, and Trangsrud (2009) examined the construct validity of student engagement in the NSSE using a confirmatory factor analysis(CFA) and suggested an improved measurement model with more factors (e.g., eight constructs).

On the other hand, several studies have been conducted (Brint et al., 2008; Chatman, 2011) on the dimensional specifications of AE using the SERU survey. One study (Chatman, 2011) specified two-dimensional structures (i.e., Engagement with Studies and Academic Disengagement) of AE using the SERU data through principal component analysis of exploratory factor analysis (EFA). The dimension of Engagement with Studies contained three factors with 24 items in total: Academic Involvement and Initiative, Research or Creative Projects Experience, and Collaborative Work. The factor constructions were similar to NSSE benchmarks and are likely to favour students studying in humanities and social sciences departments, while unlikely to favour natural science and engineering students (Brint et al., 2008; Chatman, 2011). In contrast to AE, Academic Disengagement was comprised of four factors with 16 items in total: Extracurricular Engagement, Poor Academic Habits, Non-Academic Motivation, and Easy Major. Another study (Brint et al., 2008) proposed two structures of AE by conducting EFA. A structure with one factor of seven items that favoured students in the arts, humanities, and social sciences indicated individual assertion, classroom participation, and interest in ideas. The other with one factor of seven items that favoured students in the natural sciences and engineering designated a high level of interest in prestigious and high-paying jobs and in working toward quantitative competencies through individual and collaborative study. The studies that used SERU data (Brint et al., 2008; Chatman, 2011) employed EFA exclusively, which is still necessary for specific and precise constructs showing the adequacy of the measurement models and unique associations with other factors. Moreover, to guarantee the reliability and validity of the SERU survey’s results, it is critical to implement construct validation on the survey. However, very few studies have constructed and validated the measurement models of the specific constructs constituting SERU through CFA. CFA is more appropriate than any other approach for validating measurement models affording sophisticated analyses for construct validity (Brown, 2006). Hence, the current study detects the factorial structures and germane measurement models of AE in the SERU.

Additionally, the SERU survey provides information about students’ perceived learning gains from the point they enrol in college until the time at which they take the questionnaire. Chatman (2011) specified the components of Current Skills Self-Assessment and Gains in Self-Assessment of Skills by three factors of 16 items through EFA. The three factors are Critical Thinking and Communication,Cultural Appreciation and Social Awareness, and Computer, Research, and Presentation Skills. The specification includes students’ epistemic concepts such as aesthetic and social senses and self-awareness but excludes quantitative skills. However, the EFA for the principal dimensions of the SERU questionnaire was conducted disregarding modules, question patterns of grouped items, and response scales. Moreover, some of the items were based on a binary scale that likely raises problems in EFA (Gorsuch, 1983). Additionally, a hypothesis testing procedure should be entailed for the legitimate interpretation of the EFA results (Sawang et al., 2010). Therefore, the solution still leaves some questions on construct validities. Douglass et al. (2012) examined the validity of the self-reported learning outcomes in the SERU survey and suggested the significance of the self-assessment. To ensure measurement validation, the study investigated the differences in the four levels of current cumulative grade point average (GPA,under 2.8; 2.8-3.19; 3.2-3.59; and 3.6 and higher), and the self-reported academic skills of quantitative techniques, oral presentation, writing, reading, and critical thinking. There were only six single-itemed constructs in the study, which is too narrow to encompass the specific properties of learning outcomes.

Therefore, the current study investigated the measurement structures of the SRLA to assess students’ learning outcomes in the SERU in terms of their starting and current statuses during their college enrolment. We conducted separate factor analyses for two time points and referred to them as SRLA at starting (SRLA-S) and SRLA at present (SRLA-P). Furthermore, as is generally assumed, the more academically engaged students are, the more academic development (i.e., learning outcomes) they achieve. Recent studies have reported positive relationships between academic engagements and observed academic scores such as GPA, American college testing, andthe graduate record examinationsin higher education, but we still question to what extent such scores can adequately measure students’ intellectual abilities and reflect institutional quality (Carini et al., 2006). In this regard, suggesting the importance and adequacy of the construct measurement models in the current study, we investigated the relationships among the constructs both within and between AE and SRLA. We expect the findings to support the validity of our measurement models intending to offer richer indicators of student learning and institutional productivity.

Relationships with Criterion Variables: Multiple Indicators and Multiple Causes Approach

The validity and reliability of self-reported measurement is critical if such measurement should serve as an indicator of an institution’s educational effectiveness (Douglass et al., 2012; Gordon, Ludlum, &Hoey, 2008). A substantial amount of research has proposed the validity and reliability of student self-reports under appropriate conditions (Carini et al., 2006). However, such studies do not necessarily indicate a link between self-reported constructs and other criterion variables such as GPA and socio-economic status (SES).  Hu and McCormick (2012) investigated the relationship between students’ engagement and college outcomes including GPA and self-reported gains from the NSSE. The study found that more disengaged students were less likely to achieve high GPAs and gain self-reported learning outcomes, while more engaged students, though likely to gain self-reported learning outcomes, showed non-significant GPA achievements. Another group of NESSE researchers (Carini et al., 2006; Gordon et al., 2008; Zhao &Kuh, 2004) investigated the validity of self-reported benchmarks by examining the relationships between the NSSE benchmarks and the direct measurement of learning outcomes such as GPA and standardized tests. They revealed weak relationships of the self-reported constructs of NSSE and the directly measured learning outcomes and provoked some doubt about the effectiveness of the NSSE benchmarks for assessing institutional productivity. In this sense, the SERU survey may not necessarily be free from this concern.

In addition, educational literature has suggested a significant influence of SES (i.e., parent income and education) on student learning engagement and outcome. Dubow, Boxer, and Huesmann (2009) found that parent education acted as a mediator, was linked to children’s academic successes, and influenced adult higher education levels and occupation attainment even when controlling for family SES factors. The CSHE reported (2011) the relationships among AE, GPA, and socio-economic background but SRLA for undergraduate students at a university of California. The research reported a strong relationship between time spent studying and GPA and a reverse relationship between AE and more advantaged backgrounds. The sample was restricted to one university in California, lacking the generalizability of college experiences in the SERU. Moreover, no further descriptions were provided about the relationships among students learning outcomes, AE, GPA, and SES. Building on previous research, we examine multilateral relationships among AE, SRLA, and the criterion variables of GPA and SES including parent income and parent education for more extended populations of undergraduate students from six states in the U.S. by employing a Multiple Indicators and Multiple Causes (MIMIC) approach (Marsh, Hau, Artelt, Baumert, &Peschar, 2006). 

The present research

The present study aims to specify the factorial structures and validate the measurement models for each domain of AE and SRLA in the SERU 2012. The objective of the SERU survey is to assess institutional functions and understand students’ behaviour, satisfaction, and achievement (Douglass et al., 2012). The SERU 2012 Survey consists of five thematic fields: Academic Engagement, Academic Experiences and Globalization, Civic Engagement, Student Development, and Campus Climate for Diversity. Respondents are randomly assigned to one of these five fields. Of these modules, we extracted 23 items for AE constructs including time use, and 14 items for SRLA each at two time points (i.e., at starting and present). We initially identified the smallest number of factors through the independent EFAs for AE, SRLA-S, and SRLA-P. For more enhanced specification of measurement models, we investigated the dimensional structures underlying a set of variables, considering the results from EFA and the conceptual frameworks that obtained agreement among the researchers of this study (Brown, 2006; Marsh et al., 2006). We then conducted CFA to verify measurement model adequacy and construct validities and examine the reliability of internal consistency within each factor in the specified models. We initially used a smaller sample for EFA from a university in Texas to relieve burden of the huge sample size. Furthermore, we hypothesized that the correlations among the constructs of the measurement models should support construct validities.

Finally, we adopted the MIMIC approach to investigate the relationships between the constructs of AE, SRLA-S, and SRLA-P, as well as criterion variables such as students’ cumulative GPA, annual parent income, and parents’ education. We assumed that students with a high GPA and supportive SES should be more likely to have high values regarding self-perceived learning engagement and productivity. In addition, parents’ education was expected to be a positive factor for students’ academic achievement.

We examined the following research questions:(i)Do the factor structures of AE, SRLA-S, and SRLA-P provide good model fits, reliabilities, and validities enough for the adequacy of the measurement models?(ii) How do the self-reported constructs of AE, SRLA-S, and SRLA-P correlate with each other? To what degree do those relationships support the validity of the construct measurement model?(iii)How are self-reported constructs of AE, SRLA-S, and SRLA-P associated with criterion variables (i.e., GPA, parent income, and education)? To what degree do those relationships support the validity of the construct measurement model?

The findings from this study will offer specific frameworks of the SERU data for AE and SRLA, which are considered to be primary dimensions on college quality evaluation (Brint& Cantwell, 2012; Brint, Douglass, Flacks, Thomson, & Chatman, 2007; Carini et al., 2006). Furthermore, the findings on the associations among the primary constructs and criterion variables should improve the understanding of student learning in the SERU and institutional effectiveness. It should support construct validity and the adequacy of measurement models to use advanced analyses such as CFA and the MIMIC approach, which earlier analyses of the SERU data have not used. Therefore, we expect that this study will contribute practical and plausible perspectives on the concrete specifications of AE and SRLA as primary constructs and help researchers and institutional stakeholders understand students’ behaviour, perceptions, and outcomes in research universities.

Methods

Participants

The 2012 SERU survey included a set of core items and specific items for each of the following five unique modules: Academic Engagement, Academic Experiences and Globalization, Civic Engagement, Student Development, and Campus Climate for Diversity. All participants were presented with the core items, which consisted of AE, time use, academic and personal development, overall satisfaction, and assessment of their educational experiences. Respondents were randomly assigned to each module of Academic Experiences and Globalization, Civic Engagement, Student Development, and Campus Climate for Diversity. Additionally, respondents were asked to provide background information.

The 2012 SERU survey initially collected data from 42,260 undergraduate students in six research universities from six states in the U.S. (Minnesota (23.34%), New Jersey (25.75%), Pennsylvania (9.11%), South Carolina (14.35%), Texas (15.43%), and Virginia (12.02%)). The sample consisted of 24,739 females (59%) and 17,521 males (41%). The breakdown by division was 6,354 freshmen (15%), 9,121 sophomores (22 %), 10,471 juniors (25%), and 16,251 seniors (38%). Even though we used all of the initial data, the sample sizes for AE, SRLA at two time points and criterion variables vary because of the random assignment of respondents to each module and missing data. Table 1 reports the demographic information.

Table 1. Descriptive statistics of the participants in the 2012 SERU

Instruments

Academic engagement and self-reported learning abilities. We extracted all items for the measurement structures by considering the normality of sample distribution, factor loadings, and the consistency in question patterns where items were grouped. The module of AE included 22 items with a 6-point Likert scale that asked the extent to which students have engaged in learning activities. The scale on time use consisted of 18 items with an 8-point Likert scale that asked how students have spent their time. Of the 40 items in AE and time use, we selected 23 items for AE. The items on SRLA with a 6-point Likert scale came from the Student Development module and asked about students’ proficiency in their learning abilities at the starting point (SRLA-S) and at present (SRLA-P) during their college careers. Students simultaneously reported their retrospective and current learning abilities at the two time points. We selected the two sets of 14 items for SRLA-S and SRLA-P. Table 2 displayed the items and descriptive statistics for AE, SRLA-S, and SRLA-P.

Table 2. Descriptive statistics, factor loadings, and reliabilities in the measurement models

Note: M = mean; SD = Standard Deviation; CFA = Confirmatory Factor Analysis

Criterion variables. The cumulative GPA indicates students’ scores attained during their enrolment in college curricula and were standardized within each department of each university to eliminate the differences in disciplinary culture and grading policies between departments (Wolter, 2004). Parent income was self-reported on an 11-point scale ranging from less than $10,000 to more than $200,000. Students also self-reported the education levels of their mother and father separately with an 8-point scale ranging from non-formal education to doctoral degree. The average of both parents’ education was used as the variable of parent education. Table 3 shows the descriptive statistics of the criterion variables.

Table 3. Descriptive statistics of criterion variables

Note: Parents’ annual income: self-reported with 11-point Likert ranging from less than $10,000 to more than $200,000; Parental education: self-reported for mother and father separately with 8-point Likert scale ranging from non-formal education to doctoral degree; M = mean; SD = Standard Deviation.

Procedure

The population of the SERU survey was undergraduate students enrolled in 12 major U.S. research universities. The sample was a group of all undergraduate students participating in the 2012 SERU survey. The respondents to each module were randomly assigned and the sample sizes for AE, SRLA and criterion variables were finalized. To explore the undergraduate students’ experiences in the research university, as the current study was designed as a survey research using SERU 2012 survey data. To uphold the participants’ rights to confidentiality and privacy, students’ individual information such as name and student ID was encrypted or removed in the analysis procedures.

Data analysis

All constructs of the measurement model should be pre-specified through EFA, which provides information about the number of factors and the pattern of indicator-factor loadings (Brown, 2006). As all items met the assumption of normal distributions, we conducted all analyses based on the maximum likelihood estimations (MLE) and addressed the missing data through the expectation maximization (EM) for EFA using SPSS and the full information maximum likelihood (FIML) for CFA as the default of using AMOS. We initially conducted EFAs for the items on AE and SRLA separately using the maximum likelihood (MLH) with the Promax rotation. For the adequate number of factors, we adopted two criteria: (i) the eigenvalues should be greater than one, and (ii) the factors should be less before the elbow in the scree plot. We conducted EFA and CFA with different groups of students.That is, EFA was run with a smaller sample consisting of 6,524 students from a university in Texas, whereas CFA was run with the whole sample. After selecting those items with factor loadings over 0.30, we specified the factorial structures in considering factor loadings and the conceptual frameworks. We then conducted CFA to verify the adequacy of the measurement model for each domain and test for construct validity. The error correlations between the measurement errors of items that shared causal sources other than the common factors were specified (Brown, 2006). Through the CFA, we obtained evidence of the convergent and discriminant validities. Factor loadings in a measurement model indicate the causal effects of the latent factor (i.e., construct) on the observed variables (i.e., items), and higher factor loadings indicate the higher convergent validity (Brown, 2006). Correlations over .80 or .85 among the factors imply poor discriminant validity and the need for a more parsimonious solution (Brown, 2006). The reliabilities of the internal consistency (i.e., Alpha) were tested.

Additionally, we conducted two kinds of MIMIC as described by Marsh et al. (2006). As the MIMIC approach is based on full latent-factor structures addressing measurement error and providing evidence of the adequacy of measurement models, the approach is stronger than conventional analyses such as ANOVA and multiple regression analysis (Marsh et al., 2006). One MIMIC analysis was applied to the correlations between the constructs of the measurement models (i.e., AE, SRLA-S, SRLA-P) and the criterion variables (i.e., GPA, parent income, parent education). Another MIMIC approach was used for the path coefficients from the criterion variables to the constructs of each domain, indicating the unique effects of the criterion variables on the constructs after controlling for the influences of other criterion variables.

We applied three kinds of indices for goodness of model fit: chi-square and degree of freedom (χ2/ df), the root mean square error of approximate (RMSEA), and the comparative fit indices (CFI) that is revised from the normed fit index (NFI). RMSEA and CFI are robust to sample size, while the chi-squared value is so sensitive to sample size that studies with a large sample size can hardly obtain non-significance (Brown, 2006; Marsh et al., 2006). RMSEA favours models with fewer estimated parameters (Hooper, Coughlan, & Mullen, 2008). We adopted cut-off values of 0.90 for CFI and .08 for RMSEA for a satisfactory model fit, which seems to be the general agreement among authorities in this area (Brown, 2006; Hooper et al., 2008; Hu &Bentler, 1999;Marsh et al., 2006).

Results

Exploratory and Confirmatory Factor Analyses of the Two Measurement Models

We conducted separate EFAs for the three domains of AE, SRLA-S, and SRLA-P. The reliability of internal consistency (Cronbach’s alpha) ranged from .53 to .85 with a mean of .73. The reliability of Time Use with only two items was .53. It was low because the reliability value is dependent on the number of items (Voss, Stem, & Fotopoulos, 2000). The results from the CFA analyses showed convergences with high or acceptable factor loadings ranging from .32 to .94. The information regarding all factor loadings and reliability of internal consistency (i.e. Cronbach’s alpha) is shown in Table 2. All correlations among the factors within each domain were less than .80, indicating appropriate divergence.

Academic engagement. The EFA resulted in five factors of 23 items with eigenvalues greater than one and showing the scree plot before elbow, which explained 60% of the variance. The eight error correlations were fixed because they shared the same-worded patterns or common sources other than the specified factors (Brown, 2006). Based on the conceptual frameworks and the agreement among researchers, we labelled the five constructs of AE as follows: Attention and Interest in Class (CAI) with seven items, Class Preparation (CP) with five items and Time Use (TU) with two items, Consolidated and Peer Learning (CPL), and Extra-Interaction with Faculty (EIF) with four items. Of them, CP and TU are sub factors of Class Responsibility (CR). According to theoretical framework of AE (Alrashidi et al., 2016; Fredricks et al., 2004), we assumed CR as behavioural, CAI and CPL as cognitive, and EIF as emotional engagement. Except for TU (α = .53) with only two items, all constructs of AE showed good internal consistency (α> .73). The CFA for AE yielded factor loadings ranging from .36 to .86 (M= .63), showing acceptable fit indices (CFI = .90, and RMSEA = .06). As expected, the chi-square statistic was significant (χ2/df =32617.75/214) due to its sensitivity to the large sample size.

Self-reported learning abilities. SRLA included 14 items requesting respondents to estimate their self-reported learning ability at two time points in their college experiences: the starting time (SRLA-S) and present time (SRLA-P). Each EFA of SRLA-S and SRLA-P resulted in three-factor and four-factor solutions, explaining 56% and 61% of variance, respectively. Two items on internet and computer skills were segregated into a factor in SRLA-P. We unified the different number of factor solutions into a three-factor solution: Knowledge and Comprehension Skills (KCS) consisting of six items, Research Skills (RS) consisting of four items, and Communication Skills (CS) consisting of four items. Conforming to Allan’s (1996) dimensional scheme, we proposed KCS for subject-based knowledge and skills, RS for generic academic outcomes, and CS for personal transferable abilities. Reliabilities were strong and ranged from .75 to .82 at the two time points. The CFA yielded factor loadings ranging from .32 to .90, and the averages are .66 and .65 for SRLA-S and SRLA-P, respectively. The measurement models for the two time points were of overall good fit, as shown by the following indices: CFI = .92, and RMSEA = .07 for both time-point measurement models; and χ2/df=15869.70/73, and 14189.93/73 for SRLA-S and SRLA-P, respectively.

Correlations among Academic Engagement andSelf-Reported Learning Abilities at starting and present time

We examined the correlations among the constructs in the measurement models as reported in Table 4. The correlations within each domain of AE, SRLA (i.e., intra-correlations) were stronger than the correlations between the domains (i.e., inter-correlations), implying the convergence of constructs in each domain. Further, the inter-correlations were less than .80, supporting the discriminant validity (Brown, 2006).

Within AE, the correlation between CAI and EIF was the strongest of intra-correlations (r=.78) while the strongest inter-correlation was found between CAI of AE and CS of SRLA-P (r=.49). These findings indicate that students that invest their efforts toward task mastery in class (i.e., cognitive engagement) tend to have more meaningful interactions with faculty (i.e., emotional engagement) and to gain better communication skills. Noticeably, CAI and EIF had clearly stronger associations than other constructs of AE with SRLA-P (r= .49 to .33), indicating a greater contribution of cognitive and emotional engagement than behavioural engagement to SRLA-P. The correlations of SRLA between the two time points were strong (r=.63 to .82). KCS was the most correlated with CS both at starting (r=.74) and present (r=.77) within each domain. Particularly, SRLA-P (r= .19 to .49) was apparently more strongly related with the constructs of AE than SRLA-S (r= .02 to .31), indicating college students’ productivity.

Table 4. Inter-correlations among the constructs in the measurement models

Note: All correlation coefficients were statistically significant at two-tailed p< .001.

Multiple Indicators and Multiple Causes Approaches

We conducted MIMIC analyses to examine the relationships between factors of measurement models and criterion variables. As the disciplinary culture and grading policies between departments are different, GPA was standardized within each department of each university to eliminate such differences (Wolter, 2004). The CFA model produced the correlations between the measured constructs and criterion variables including GPA, parent annual income, and parent education. The SEM yielded the unique effects of each criterion variable after controlling for the effects of other criterion variables on the constructs of each measurement model (Marsh et al., 2006). Some of the relationships were statistically significant (p< .01 or p< .001 at two-tailed level), but weak, yielding small effect sizes (the multiple R = .00 to .24). The correlations among criterion variables were moderately positive (r = .13 to .46). GPA was correlated with both parent income (r = .13) and parent education (r = .16). Table 5 shows the relationships among AE, SRLA-S, SRLA-P, ECSM, and the criterion variables through the MIMIC analyses. The similarity of the values of correlation and regression coefficients between GPA and the factors of AE is due to two-digit rounding; when three-digit values are used, those coefficients differ more greatly. Moreover, the similarity may be a result of the standardization of GPA within each academic department and because it had a similar scale with the 6-point Likert scale of the construct measurement.

Table 5. Relations between the constructs in the measurement models and criterion variables

Note. ** p< .001; * p< .01; GPAa: GPA was cumulative and standardized within each disciplinary department of each university

We hypothesized that students with high GPAs are more academically motivated and engaged during their enrolment in academic courses (Fenollar, Román, & Cuestas, 2007). All correlations (r = .07 to .24) and path coefficients (β= .07 to .24) between GPA and AE factors were statistically significant (p< .001), supporting the assumption. However, the effect sizes were weak (r = 0.1) and medium (r = 0.3). As expected, CR was the most strongly related with GPA, and CPL was weakly related, suggesting that class preparation and study time with strong responsibilities for class (i.e., following class norms as behavioural engagement) are more crucial attitudes for higher GPAs than any other form of learning engagement.

Regarding the relationships between GPA and SRLA, most of them were weak aside from their statistical significances. Nevertheless, GPA was more substantially related with SRLA-P (r = .02 to .20, β= .02 to .18) than SRLA-S (r = -.02 to .09, β= -.03 to .07), implying the contributions of GPA to students’ university learning outcomes. In addition, GPA was the most closely connected to KCS of SRLA-P (r =.20, β= .18), indicating a substantial impact of GPA on subject-based knowledge and skills during their undergraduate study. It was generally expected that students’ family backgrounds would be related with their learning (Brint et al., 2007; Ishitani, 2003). However, this study could hardly suggest that SES has a meaningful influence on AE, even though most correlations of parent income and parent education with the constructs were statistically significant owing to the large sample size (r = .02 to .07). Specifically, parent income showed a weak influence on CPL (β= .03), a negative influence on CR (β= -.03), and a non-significant influence on the others of AE. The effects of Parent Education on all constructs (β= .03 to .05) of AE except for CL were significant. Interestingly, parent income was almost more related with SRLA-P (r = .03 to .15, β= .03 to .14) than SRLA-S (r = .04 to .12, β= .03 to .12), while parent education was more connected to SRLA-S (r = .05 to .13, β= .01 to .08) than SRLA-P (r = .02 to .11, β= -.02 to .02). This result indicated the different patterns between parent income and parent education in their relationships with students’ SRLA regarding pre- and post-enrolment in college. We concluded that parent income has a greater impact on student’ perceived learning gains during their college enrolment than parent education. Moreover, parent income and parent education were the most related with KCS of SRLA at starting (r = .12, β= .07 for parent income; r = .13, β= .08 for parent education) and at present (r = .15, β= .12 for parent income; r = .11, β= .02 for parent education).

Overall, GPA had stronger effects on all constructs of AE (β= .07 to .24) than those of SRLA except for KCS of SRLA-P (β= .18) in the path analyses where unique effects of GPA were shown after controlling for the effects of parental income and education on those measurement models.

Discussion

The SERU questionnaire comprises a large number of items used to measure students’ college experiences in various dimensions based on a module design in which participants are randomly assigned to each module. The dimensional structures are both practical and interpretable and are important for researchers and stakeholders of higher education. Moreover, students’ motivational and academic engagement and learning outcomes should be the crux of understanding their college experiences (Kuh, 2003; Schweinle& Helming, 2011).

We aimed to specify the factorial structures to measure the constructs on AE and SRLA and validated the measurement models in the SERU through EFA and CFA. The robust analyses of EFA and CFA produced the three adequate measurement models of AE, SRLA-S, and SRLA-P, and supported the construct validity, showing good or acceptable factor loadings and model fits as well.

The intra-correlations of the constructs within each domain of AE, SRLA-S, and SRLA-P, and inter-correlations between the domains showed the evidence of the convergent and discriminant validity for the measurement models. Most of all, the specific constructs of AE and SRLA enabled us to better understand their unique contributions to each other. Specifically, the strongest associations between CAI and EIF within AE, and their clearly stronger relationships than others of AE with SRLA-P, implied the close interactions between cognitive (i.e., CAI) and emotional engagement (i.e., EIF) and their substantial contributions to students’ self-perceived learning outcomes during college enrolment. In other words, students who pursue task mastery and who have more interactions with faculty tend to perceive more learning gains through their college life. Furthermore, the stronger correlations of SRLA-P than SRLA-S with AE apparently indicated students’ college learning gains because SRLA-P, rather than SRLA-S, is considered to be students’ learning outcome during their college enrolment.

The MIMIC analyses offered some insights into the relationships of the constructs of AE and SRLA with GPA, parental income and education. Unlike students’ self-perceived learning gains, GPA was most related with CR of AE, indicating that students who behaviourally commit in class by attending courses and completing assignments are most likely to attain a high GPA. This finding partially corresponds with the CSHE (2011) reporting a strong relationship between GPA and study time for university students in California. However, it was slightly different from the finding by Hu and McCormick (2012), which showed a non-significant association between GPA and the self-reported benchmarks of NSSE. However, the findings of this study provide more specific information about the links between crucial constructs of college learning and GPA by adopting more specific and accountable construct measurement models of students’ self-recognized academic engagement and learning gains encompassing theory-based conceptual types.

Meanwhile, the results of this study reveal the difference between self-reported and observed measurements. We suspect that the concerns raised by Pike (1996) and Gordon et al. (2008) about the ambiguity in the relationships between self-reported measurements and GPA can be explained by the method effect (Brown, 2006; Wolters &Pintrich, 1998). These notwithstanding, the stronger relationships of GPA with SRLA-P than with SRLA-S signified the productivity of GPA for university students’ cognitive growth, particularly in subject-based knowledge and skills (Allan, 1996).

Despite most of the statistical significances, the weak or even non-significant relationships of parental income and education with the constructs in the current research, however, leaves some questions about the impacts of SES on college learning. We speculated that the finding signified less influence of family factors on college learning than on K-12 education. Nevertheless, we found that parent income was more related with students’ learning abilities during college whereas parent education was more related with their learning abilities before college. We ruminated that students with greater access to monetary support were more likely to perceive their learning gains through college experiences.

Meanwhile, this study displayed the limitation of the method effect resulting from the different measurement approaches. For more validity of self-reported measurements, we suggest more comprehensive regression analyses including a greater number of constructs on college learning, handling the different patterns in fixed or latent groups. Moreover, the SERU survey is a lengthy instrument with many scales, and this study addressed only some of them. Further investigations should thus be followed as necessary. Last, the various relationships among the constructs of the SERU and the extended types of measurement variables from real and direct data sources should be examined in follow-up studies.

The SERU survey is a comprehensive instrument intended to advance the understanding of the depth and breadth of student engagement in large public research universities. We intend to determine whether there are stable and interpretable constructs related to student engagement and learning to be found in the SERU. In addition, we aimed to examine the link between student engagement and student learning. The findings from this research contribute to better understanding of research university students’ academic engagement, self-perceived learning outcomes, and the relationships among these factors. Moreover, the present study specified the impacts of criterion variables including GPA and parent education and income on students’ academic engagement and outcomes at different time points, employing MIMIC approaches of CFA and SEM. In terms of methodological aspects, this study verified the universal measuring instrument for reliability and validity and indicated the limitations of self-reported scales. Finally, we were able to see learning in college as a complex, multi-faceted enterprise with important components that must be thoughtfully put into place in order to achieve desired outcomes for students, which should contribute to insights on institutional effectiveness. Future users of the survey that we employed in the current study should be aware that constructs such as academic engagement and outcomes were measured with self-reported items. In the same vein, we recommend that future studies examine the findings of this research using a measurement instrument that consists of observable items.

References

Allan, J. (1996). Learning outcomes in higher education. Studies in Higher Education, 21, 93 - 108.https://doi.org/10.1080/03075079612331381487

Alrashidi, O., Phan, H., &Ngu, B. (2016). Academic engagement: An overview of its definitions, dimensions, and major conceptualisations. International Education Studies, 9, 41-52.doi: 10.5539/ies. v9n12p41

Appleton, J. J., Christenson, S. L., Kim D., &Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the student engagement instrument. Journal of School Psychology, 44, 427 - 445. doi: 10.1016/j.jsp.2006.04.002

Brint,S., & Cantwell, A. M. (2012). Portrait of the disengaged. Retrieved from http://cshe.berkeley.edu/ research/seru/papers.htm

Brint, S., Cantwell, A. M., &Hanneman, R. A. (2008). The two cultures of undergraduate academic engagement. Research in Higher Education, 49, 383-402.doi: 10.1007/s11162-008-9090-y

Brint,S., Douglass, J. A., Flacks, R., Thomson, G., & Chatman, S. (2007). A new generation: Ethnicity, socioeconomic status, immigration and the undergraduate experience at the university of California. General report on the results of the 2006 university of California undergraduate experience survey. Retrieved from http://www.cshe.berkeley.edu/sites/default/files/shared/publications/docs/SERU.New GenerationReport2007.pdf

Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.

Carini, R. M., Kuh, G. D., & Klein, S. P. (2006). Student engagement and student learning: Testing the linkages. Research in Higher Education, 47, 1-32. https://doi.org/10.1007/s11162-005-8150-9

Centre for Studies in Higher Education. (2011). Student experience in the research university. Retrieved from http://cshe.berkeley.edu/research/seru/index.htm

Chatman, S. (2011). Best practices when using student survey results in academic program review. SERU project technical report. Retrieved from http://www.cshe.berkeley.edu/sites/default/files/shared/research/seru/papers/1.SERU.AcademicProfileGuide10.25.2011.pdf

Christenson, S. L., Reschly, A. L., Appleton, J. J., Berman-Young, S., Spangers, D., & Varro, P. (2008). Best practices in fostering student engagement. In A. Thomas& J. Grimes (Eds.), Best practices in school psychology V (pp. 1099-1120). Washington, DC: National Association of School Psychologists.

Douglass, J. A., Thomson, G., & Zhao, C. (2012). The learning outcomes race: The value of self-reported gains in large research universities. Higher Education, 64, 317-335. https://doi.org/10.1007/s10734-011-9496-x

Dubow, E. F., Boxer, P., &Huesmann, L. R. (2009). Long-term effects of parents’ education on children’s educational and occupational success: Mediation by family interactions, child aggression, and teenage aspirations. Merrill-Palmer Quarterly, 55, 224-249. http://dx.doi.org/10.1353/mpq.0.0030

Fenollar, P., Román, S., & Cuestas, P. J. (2007). University students' academic performance: An integrative conceptual framework and empirical analysis. British Journal of Educational Psychology, 77, 873-891.https://doi.org/10.1348/000709907X189118

Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74, 59 - 109. doi:10.3102/00346543074001059

Gordon, J. L., Ludlum, J., &Hoey, J. (2008). Validating NSSE against student outcomes: Are they related? Research in Higher Education, 49, 19-39.

Gorsuch, R. L. (1983). Factor analysis. Hillsdale, NJ: L. Erlbaum Associates.

Hooper, D., Coughlan, J., & Mullen, M.R. (2008). Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods, 6, 53-60.doi: 10.21427/D7CF7R

Hu, L. T.,&Bentler, P.M. (1999). Cut-off criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives.Structural Equation Modelling, 6, 1-55.https://doi.org/10.1080/10705 519909540118

Hu, S., & McCormick, A. C. (2012). An engagement-based student typology and its relationship to college outcomes. Research in Higher Education, 53, 738-754. doi: 10.1007/s11162-012-9254-7

Humphreys, D.,& Gaston, P. (2015). Quality assurance and accreditation in challenging times: Examining priorities and proposed reforms. [online] Eric.ed.gov. Available at: https://eric.ed.gov/?q=source% 3A%22Liberal+Education%22&id=EJ1095372

Ishitani, T. T. (2003). A longitudinal approach to assessing attrition behaviour among first-generation students: Time-varying effects of pre-college characteristics. Research in Higher Education, 44, 433-449. https://doi.org/10. 1023/A:1024284932709

Jimerson, S. R., Campos, E., & Greif, J. L. (2003). Toward an understanding of definitions and measures of school engagement and related terms. California School Psychologist, 8, 7-27.https://doi.org/10.1007/BF033 40893

Kuh, G. D. (2003). What we're learning about student engagement from NSSE:Benchmarks for effective educational practices. Change: The Magazine of Higher Learning, 35, 24-32.doi: 10.1080/0009138 0309604090

LaNasa, S. M., Cabrera, A. F., &Trangsrud, H. (2009). The construct validity of student engagement: A confirmatory factor analysis approach. Research in Higher Education, 50, 315-332. doi: 10.1007/s11162-009-9123-1

Marsh, H. W., Hau, K., Artelt, C., Baumert, J., &Peschar, J. L. (2006). OECD's brief self-report measure of educational psychology's most useful affective constructs: Cross-cultural, psychometric comparisons across 25 countries. International Journal of Testing, 6, 311-360. https://doi.org/10.1207/s15327574ijt 0604_1

Pike, G. R. (1996). Limitations of using students’ self-reports of academic development as proxies fortraditional achievement measures. Research in Higher Education, 37, 89 - 114.https://doi.org/10.1007/BF01680043

Phillips, R., McNaught, C.,& Kennedy, G. (2010). Towards a generalised conceptual framework for learning: Thelearning environment, learning processes and learning outcomes (LEPO) framework. In J. Herrington & C. Montgomerie (Eds.), Proceedings of EdMedia + Innovate Learning 2010 (pp. 2495-2504). Waynesville, NC: Association for the Advancement of Computing in Education (AACE).

Reeve, J., & Tseng, C. M. (2011). Agency as a fourth aspect of students’ engagement during learning activities. Contemporary Educational Psychology, 36, 257-267.doi: 10.1016/j.cedpsych.2011.05.002

Sawang, S., Oei, T. P. S., Goh, Y. W.,Mansoer, W.,Markhum, E., &Ranawake, D. (2010). Confirmatory factor analysis of the way of coping checklist-revised (WCCL-R) in the Asian context. Applied Psychology: An International Review, 59, 202-219.https://doi.org/10.1111/j.1464-0597.2009.00378.x

Schweinle, A., & Helming, L. M. (2011). Success and motivation among college students. Social Psychology of Education, 14, 529-546. https://doi.org/10.1007/s11218-011-9157-z

Skinner, E. A., Kindermann, T. A., &Furrer, C. J. (2009). A motivational perspective on engagement and disaffection: Conceptualization and assessment of children’s behavioural and emotional participation in academic activities in the classroom. Educational and Psychological Measurement, 69, 493-525.http://dx.doi.org/10.1177/0013164408323233

Voss, E., Stem, E., & Fotopoulos, S. (2000). A comment on the relationship between coefficient alpha and scale characteristics. Marketing Letters, 11, 177-191.https://doi.org/10.1023/A:1008146924781

Wolters, C. A., &Pintrich, P. R. (1998). Contextual differences in student motivation and self-regulated learning in mathematics, English, and social studies classrooms. Instructional Science, 26, 27-47. https://doi.org/10.1023/A:1003035929216

Zhao, C., &Kuh, G. D. (2004). Adding value: Learning communities and student engagement. Research in Higher Education, 45, 115 - 138.https://doi.org/10.1023/B:RIHE.0000015692.88534.de

Zlatkin-Troitschanskaia, O., Pant, H. A., & Coates, H. (2016). Assessing student learning outcomes in higher education: challenges and international perspectives. Assessment & Evaluation in Higher Education, 41, 655-661.http://dx.doi.org/10.1080/02602938.2016.1169501