The Conceptualisation and Measurement of DSM-5 Internet Gaming Disorder: The Development of the IGD-20 Test

Figures

Abstract

Background

Over the last decade, there has been growing concern about ‘gaming addiction’ and its widely documented detrimental impacts on a minority of individuals that play excessively. The latest (fifth) edition of the American Psychiatric Association’s Diagnostic and Statistical Manual of Mental Disorders (DSM-5) included nine criteria for the potential diagnosis of Internet Gaming Disorder (IGD) and noted that it was a condition that warranted further empirical study. Aim: The main aim of this study was to develop a valid and reliable standardised psychometrically robust tool in addition to providing empirically supported cut-off points.

Methods

A sample of 1003 gamers (85.2% males; mean age 26 years) from 57 different countries were recruited via online gaming forums. Validity was assessed by confirmatory factor analysis (CFA), criterion-related validity, and concurrent validity. Latent profile analysis was also carried to distinguish disordered gamers from non-disordered gamers. Sensitivity and specificity analyses were performed to determine an empirical cut-off for the test.

Results

The CFA confirmed the viability of IGD-20 Test with a six-factor structure (salience, mood modification, tolerance, withdrawal, conflict and relapse) for the assessment of IGD according to the nine criteria from DSM-5. The IGD-20 Test proved to be valid and reliable. According to the latent profile analysis, 5.3% of the total participants were classed as disordered gamers. Additionally, an optimal empirical cut-off of 71 points (out of 100) seemed to be adequate according to the sensitivity and specificity analyses carried.

Citation: Pontes HM, Király O, Demetrovics Z, Griffiths MD (2014) The Conceptualisation and Measurement of DSM-5 Internet Gaming Disorder: The Development of the IGD-20 Test. PLoS ONE 9(10): e110137. https://doi.org/10.1371/journal.pone.0110137

Editor: Yijun Liu, University of Florida, United States of America

Received: April 3, 2014; Accepted: September 16, 2014; Published: October 14, 2014

Copyright: © 2014 Pontes et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: The authors confirm that all data underlying the findings are fully available without restriction. All relevant data are within the paper and its Supporting Information files.

Funding: The authors have no support or funding to report.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Over the last decade, there has been growing worldwide concern from researchers about ‘gaming addiction’. Official bodies such as the American Psychiatric Association [1] and numerous scholars [2]–[5] have suggested the need for unification and consensus for the assessment of gaming addiction if this phenomenon is to be considered as an independent clinical entity in the future. Despite the proliferation of research on gaming behaviour over the last few years [6], [7], the field has been hindered by the use of inconsistent and non-standardised criteria to assess and identify problematic and/or addictive video game use [6]. Moreover, this problem may be also reflected by the heterogeneity of nomenclatures used by researchers to address the same phenomenon including such terms as video game addiction [8], computer game playing dependence [9], internet addiction disorder [10], video game dependency [11], problematic online gaming [12], and pathological video-game use [13]. In addition to these issues, most psychometric tools developed for assessing behavioural addictions (including gaming addiction) have either used an ad hoc cut-off point or lacked a strong empirical base for establishing such cut-off points.

These problems may be partially explained by the lack of agreement amongst researchers on how to approach the assessment of the phenomenon. For instance, some studies [13], [14] adapted the definition of pathological gambling from the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV; [15]) to assess this phenomenon. Others have been based on the DSM-IV criteria of substance use dependence [16], or have combined these two approaches and used criteria from both pathological gambling and substance use dependence [17]. Additionally, some researchers have used criteria from various different behavioural addictions such as internet addiction [18] or exercise addiction [19].

In acknowledgement of the many studies now published in the area of problematic gaming, Section 3 of the fifth revision of the DSM [1] included ‘internet gaming disorder’ (IGD) for the first time. Here, IGD was viewed as a behavioural addiction that needs further study before being recognised as an independent clinical disorder. This represents a milestone achievement by attempting to (i) provide a consensual view of the phenomenon from a scientific point of view, and (ii) unify different approaches into a single one [3].

According to the APA [1], the clinical diagnosis of IGD comprises a behavioural pattern encompassing persistent and recurrent use of the Internet to engage in games, leading to significant impairment or distress in a period of 12 months as indicated by five (or more) out of the nine criteria that must be present. More specifically, the nine proposed criteria for IGD include: (1) preoccupation with internet games; (2) withdrawal symptoms when internet gaming is taken away; (3) tolerance, resulting in the need to spend increasing amounts of time engaged in internet games; (4) unsuccessful attempts to control participation in internet games; (5) loss of interests in previous hobbies and entertainment as a result of, and with the exception of, internet games; (6) continued excessive use of internet games despite knowledge of psychosocial problems; (7) deceiving family members, therapists, or others regarding the amount of internet gaming; (8) use of internet games to escape or relieve negative moods; and (9) jeopardising or losing a significant relationship, job, or education or career opportunity because of participation in internet games. Furthermore, it has been asserted that IGD may lead to school/college failure, job loss, or marriage failure as the compulsive gaming behaviour tends to displace usual and expected social, work and/or educational, relationship, and family activities [1]. It has also been noted [3] that the nine IGD criteria directly map onto the six criteria of Griffiths’ components model of addiction (i.e., salience, mood modification, tolerance, withdrawal symptoms, conflict and relapse) [20].

The aim of the present study was twofold. Our main goal was to examine whether the nine IGD criteria from the DSM-5 [1] can empirically correspond with the six dimensions of the components model of addiction by developing a new standardised psychometric tool. Our second goal was to provide evidence of its reliability and validity alongside an empirical cut-off point for future studies wishing to assess IGD in line with the DSM-5. If the results of the study support these two aims, then the newly developed tool represents a valuable instrument for future researchers to empirically investigate IGD.

Methods

This study was approved by the College Research Ethics Committee of Nottingham Trent University (UK). In order to participate in the study informed consent was sought amongst participants and the minimum age of participation in the study was 16 years old.

Sample, Procedure, and Participants

Participants were invited to take part in the study by clicking the survey link provided in 52 online gaming forums. In order to advertise the survey a thread was created and daily checked for a month on each of the 52 online forums specifying the nature of the study. The survey was created and hosted online. The online data collection methodology was chosen because of its inherent benefits, such as ease of access to larger sample pools, cost-efficiency, and its usefulness and practical advantages for researching behavioural addictions in general [21], [22], especially in the case of online gamers. This methodology might also increase participant’s self-disclosure [23] and disinhibition [24], which helps to decrease social desirability. A total of 1397 questionnaires were collected. However, 394 of these (28.2%) were not fully completed and were therefore excluded from the subsequent analyses.

Measures

Socio-demographics.

Information regarding gender, age, country of residence, age when they first began gaming, relationship status, ownership of mobile device with internet access and/or gaming console and other gaming devices were collected.

Weekly Gameplay.

This variable examined the player’s weekly time spent gaming on computers, consoles, and/or other gaming platforms (e.g., handheld devices). This was operationalised into distinct playing categories (i.e., less than 7 hours a week; between 8 and 14 hours a week; between 15 and 20 hours a week; between 21 and 30 hours a week; between 31 and 40 hours a week, and more than 40 hours per week). This variable was later recoded to distinguish between players that played more or less than 30 hours a week in order to fully reflect APA’s definition of IGD concerning the time spent playing.

Internet Gaming Disorder Test.

The IGD-20 Test includes 20 items reflecting the nine criteria of IGD as in the DSM-5 [1] and incorporated the theoretical framework of the components model of addiction (i.e., salience, mood modification, tolerance, withdrawal symptoms, conflict and relapse) [20]. Consequently, three items were devised for each of the following IGD criteria 1, 2, 3, 4 and 8 and another five items for criteria 5, 6, 7 and 9 altogether because these latter four criteria appear to reflect the conflict dimension (see Table 1). The IGD-20 Test examines both online and/or offline gaming activities occurring over a 12-month period, since the DSM-5 criteria for IGD are based on persistent and recurrent gaming. This most often involves specific internet games, but can also include non-internet computerised games [1]. Participants rated all items of this test on a 5-point Likert scale: 1 (“Strongly disagree”), 2 (“Disagree”), 3 (“Neither agree or disagree”), 4 (“Agree”), and 5 (“Strongly agree”).

Diagnostic Criteria of IGD in DSM-5.

The diagnostic features of the IGD in DSM-5 comprise nine criteria reflecting its key aspects. According to the APA [1], to be diagnosed with IGD a person has to endorse at least five (or more) of the nine criteria over a 12-month period. Since these nine criteria were developed to be used by clinicians as a form of checklist in a binary system (i.e., yes or no), we slightly modified the response option so that it could be presented to participants along a continuum using a 5-point scale (i.e., 1 “Never”, 2 “Rarely”, 3 “Sometimes”, 4 “Often”, 5 “Very Often”). This was done because the research team felt the restrictive two-option (yes/no) choice might be problematic from a statistical standpoint [25]. Additionally, previous research suggested that multiple-choice items traditionally yield more reliable test scores than scores derived from dichotomous items [26]. In the present study, the Diagnostic Criteria of IGD’s internal consistency as measured by the Cronbach’s alpha was.87.

Statistical Analysis

In order to test the proposed model for IGD, confirmatory factor analysis (CFA) was performed with maximum likelihood estimation with robust standard errors (MLR) in MPLUS 6.1 [27]. The goodness of fit was evaluated using a p value of Chi-square smaller than.05 for the test of close fit. Additional fit indices included the comparative fit indices (CFI), Tucker-Lewis Fit index (TLI), root mean square error of approximation (RMSEA) and its 90% confidence interval (90% CI), and standardised root mean square residual (SRMR). A model presents an acceptable fit by a CFI greater than.90 and a RMSEA value smaller than.08. A good fit is expressed by a CFI value higher than.95 and a RMSEA value close to.06 [28], [29].

In order to identify the groups of gamers with higher risk of IGD, latent profile analysis (LPA) was performed in MPLUS 6.1 [27]. The LPA is a mixture modeling technique used to identify groups of people that are similar in their responses to certain variables – in this case average sum scores given for the six IGD-20 Test dimensions (continuous manifest variables) [30]. In the process of determining the number of latent classes, the Bayesian information criteria parsimony index was used, alongside the minimisation of cross-classification probabilities, entropy and the interpretability of clusters. In the final determination of the number of classes, the likelihood-ratio difference test (Lo-Mendell-Rubin Adjusted LRT Test) was also used. This compares the estimated model with a model having one less class than the estimated model [27]. A low p value (<.05) suggests that the model with one less class is rejected in favour of the estimated model.

To determine the cut-off points of the IGD-20 Test, a sensitivity analysis based on membership in the “disordered gamers” group from the latent profile analysis as the ‘gold standard’ was carried out. Thus, the accuracy of the IGD-20 Test by calculating the proportion of participants classified as ‘disordered gamers’ versus other gamers could be assessed. The sensitivity (i.e., the proportion of true positives belonging to the disordered group based on LPA) and specificity (i.e., the proportion of true negatives among the non-disordered gamers) were defined as suggested by Altman and Bland [31] and Glaros and Kline [32]. In order to explore the probability that the IGD-20 Test would give the correct ‘diagnosis’, the positive predictive values (PPVs), the negative predictive values (NPVs), and the accuracy values for each possible IGD-20 Test cut-off points were calculated. PPV was defined as the proportion of participants with positive test results who are correctly diagnosed [32], [33]. The NPV was defined as the proportion of participants with negative test results who are correctly diagnosed [32], [33].

Additionally, to assess the validity of the IGD-20 Test, the LPA classes were compared alongside other variables (i.e., gender, age, weekly gameplay, IGD-9 scores, and IGD-20 Test scores) relevant to the phenomenon of IGD. In order to do these comparisons, Wald’s Chi-square test of mean equality for latent class predictors in mixture modeling was also performed because it takes into account the probabilistic nature of the LPA groups (for description of analysis, see www.statmodel.com/download/meantest2.pdf).

Results

Descriptive Statistics

The total sample comprised 1003 participants, with the majority (85.2%) being male (n = 855). Ages varied between 16 and 58 years, and the mean age was 26 years (SD = 8.2 years). All sample characteristics are presented in Table 2.

Confirmatory Factor Analysis

The analysis of the first-order model with the six factors (i.e., salience, mood modification, tolerance, withdrawal symptoms, conflict, and relapse) provided an acceptable model fit for the IGD-20 Test, χ2 (151, n = 1003) = 504.6, p<0.0001; CFI = 0.935; TLI = 0.918 RMSEA = 0.048 (90%CI: 0.044-0.053), pclose = 0.716; SRMR = 0.041 (see Table 3). With the exception of item 19, all factor loadings were higher than.50 with their respective factors. The correlations among the factors ranged from.42 to.94, with the highest correlation observed being between salience and tolerance and the lowest between mood modification and conflict (see Table 4).

Criterion-related Validity, Concurrent Validity, and Reliability

Criterion-related validity was assessed by the association between weekly gameplay and the IGD-20 Test scores (rs(1003) = .77, p<.001). Although time spent on games itself should not be the sole indicator of IGD, disordered players typically devote between 8 to 10 hours or more per day to gaming activity and at least 30 hours per week [1]. Therefore, the strong correlation between these two variables was considered an evidence of criterion-related validity. Concurrent validity was assessed by the association of the IGD-20 Test with the nine IGD criteria from the DSM-5 (rs (1003) = .82, p<.001). Additionally, the six IGD-20 Test dimensions were strongly correlated with their corresponding IGD criteria (see Table 5). The IGD-20 Test’s internal consistency as measured by the Cronbach’s alpha was.88.

Latent Profile Analysis

After performing the LPA on the six dimensions of the IGD Test, a five-class solution was found according to the adopted decision criteria. As shown in Table 6, the AIC, the BIC, and sample-size adjusted BIC continued to decrease as more latent classes were added. However, a levelling-off after the five-latent-class solution was observed. In inspection of entropy, the five-class solution provided an adequate level. Based on the L-M-R test, the five-class solution was accepted.

The features of the five classes are presented in Figure 1 and Table 7. The first and the second classes represent casual gamers (19.1%) and regular gamers (48.6%), that is, gamers that generally scored below the mean average. The third class represents low risk engagement gamers (10.4%), while the fourth class represents at risk high engagement gamers (16.7%). The main difference between these two classes is that the at risk high engagement gamers scored much higher on conflict and relapse, while the low risk high engagement gamers scored slightly higher in salience, mood modification, tolerance, and withdrawal symptoms. The final (fifth) class represents the disordered gamers (5.3%) that scored much higher on all six dimensions than the other four groups of gamers. Those players in the disordered gamers class were more likely to (i) be male, (ii) play for more than 30 hours per week, and (iii) have an overall higher score on the nine IGD criteria and IGD-20 Test (see Table 8).

The empirical cut-off for determining the disordered gamers: Sensitivity and specificity analyses

As shown in Table 9, the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy of the IGD-20 Test at possible cut-off points were calculated considering the membership in the fifth class (i.e., disordered gamers) as the ‘gold standard’. Based on this analysis, a cut-off score of 71 is suggested as an ideal empirical cut-off to distinguish disordered gamers from non-disordered gamers.

In this case, the specificity is 100%, while the sensitivity is 96%. That is, practically none of the non-disordered cases are considered as disordered, while only 4% of the truly disordered gamers are not identified by the measure. Additionally, PPV is 94% and NPV is 100%. In other words, only 6% of the individuals with a positive test result are mistakenly identified, while all individuals with negative test results are identified correctly. The accuracy was 100%. Increasing the cut-off points would result in more false negative cases, while decreasing would increase the number of gamers mistakenly diagnosed.

Discussion

Based on the need for a unified psychometrically sound measurement tool for the assessment of Internet Gaming Disorder (IGD), the present study aimed to develop and construct the IGD-20 Test based on a solid theoretical framework (i.e., components model of addiction) integrating in its model the nine IGD criteria presented in the DSM-5 as proposed by the American Psychiatric Association [1]. When administered to a large sample of heterogeneous gamers, the IGD-20 Test appeared to be an appropriate instrument for assessing IGD.

Overall, the psychometric analyses of the IGD-20 Test yielded good results in terms of validity and reliability. Additionally, the present model appears to have an acceptable model fit according to the results obtained from the CFA. More specifically, criterion-related and concurrent validity were warranted by the observed significant correlations between the (i) IGD-20 Test and weekly gameplay, and (ii) IGD-20 Test and the nine IGD criteria from DSM-5. Additionally, significant correlations between the IGD-20 Test’s six factors and its corresponding IGD criteria also supported the test’s concurrent validity. According to the latent profile analysis, 5.3% of the players belonged to the disordered gamers group, indicating a relatively conservative prevalence of disordered gamers among the sample, and is in line with other previously published and nationally representative studies (e.g., [11], [13]).

Previous research has attempted to distinguish between ‘addicted’ and ‘highly engaged’ players. Highly engaged players are non-disordered gamers displaying high levels of cognitive salience, tolerance and euphoria, while addicted players are those that display high levels of conflict, withdrawal, relapse, and behavioural salience in the first place [34], [35]. Interestingly, the low risk high engagement gamers group as shown in the LPA analysis, matched the profile described by Charlton and Danforth as highly engaged players. Hence, this group scored high on salience, mood modification, and tolerance, while scoring lower on the core components of addiction (conflict, withdrawal, and relapse). On the other hand the at risk high engagement group scored high on two core addiction components (conflict and relapse) in addition to scoring high on salience and mood modification. Although this group does not perfectly match the ‘addiction’ group defined by Charlton and Danforth [35], when compared to the low risk high engagement LPA group they might be at greater risk due to a higher displacement of conflict and relapse components. Therefore, in addition to using the suggested cut-off score (i.e., 71) to identify disordered gamers, we propose the use of a ‘pattern analysis’ for the remaining gamers to distinguish between low risk and at risk high engagement players. Players scoring high on the conflict, withdrawal, and relapse dimensions might be at greater risk than those scoring lower on these dimensions based on Charlton and Danforth’s findings. The disordered gamers group was more likely to be male, and play for more than 30 hours per week. This finding is supported by other studies that found higher rates of addiction among males [11], [36]–[38], and those that found addicted gamers spend significantly more time playing than non-addicted players [13], [37], [39].

Finally, the sensitivity and specificity analysis revealed an empirically optimal cut-off of 71 points for diagnosing IGD with the IGD-20 Test. Nevertheless, future studies should further assess this in a clinical sample in order to corroborate the present findings. Recent research has already addressed this issue using the original nine criteria for IGD as a semi-structured interview [40]. However, this should also be done using a standardised and unified measurement tool in order to warrant progress and unification of the field.

The present study is not without limitations. The study used a convenience sample of gamers that was self-selecting (and therefore was not necessarily representative of all gamers). Consequently, the findings need to be cautiously interpreted in terms of generalizability. Notwithstanding, future studies should aim to confirm or disconfirm these results in representative samples (at either a national level and/or among the gaming community). Another important and difficult issue to overcome is the use of self-report questionnaires and their associated possible biases (e.g., social desirability biases, short-term recall biases, etc.). Future research should also attempt to confirm these findings using behavioural data and assess IGD in clinical samples in order to achieve recognition of this disorder as an independent clinical entity that merits inclusion in future editions of the DSM. Future studies could also include such measures as the Marlowe-Crowne Social Desirability Scale [41] to help overcome such biases (although this would lengthen such surveys and may lead to less participants completing them). This may also be related to the issue of non-completion of the survey. In the present study, just over 28% of the participants started but did not finish the survey. There is no way of knowing why the non-completion rate was so high, but this may be related to the survey being too long and/or gamers wanting to know what the survey was about with no intention of completing it (i.e., doing it out of curiosity). Whether the non-completers were any different from those gamers that completed the survey is not known, but this should be taken into account when considering the study’s findings. Finally, participants in the present study were recruited from English-speaking online forums and communities, therefore, they were not filtered based on their first language. This may represent a possible limitation in that such people may not have fully understood the questions being asked. Therefore, future studies should take into account the first language of the participants.

Taken as a whole, the findings of the present study support the concept of IGD. It also supports the viability of its further study as reflected by the nine IGD criteria and the components model of addiction. Furthermore, the current findings also suggest that the IGD-20 Test satisfies the need for a standardised and psychometrically sound measurement tool for assessing this behavioural addiction in accordance to the IGD criteria outlined in DSM-5 [1]. Additionally, the IGD-20 Test was designed to be applicable and cover all gamers irrespective of the genre played, demarcating from previous trend of researching and assessing specific games and gamers such as those that play Massively Multiplayer Online Role Playing Games [6], [34]. Consequently, our hope is that this instrument facilitates the need to reach a consensus in the field in terms of assessment and conceptual definition of this increasingly studied phenomenon.

Supporting Information

Table S1.

The Internet Gaming Disorder Test, Dimensionality and Instructions.

https://doi.org/10.1371/journal.pone.0110137.s001

(DOCX)

Table S2.

Internet Gaming Disorder 9 Criteria, Instructions and Reliability.

https://doi.org/10.1371/journal.pone.0110137.s002

(DOCX)

Acknowledgments

The authors are grateful to all gamers who participated in this study and also the forum moderators who actively collaborated with the authors by facilitating the communication with the gamers.

Author Contributions

Conceived and designed the experiments: MDG HMP. Performed the experiments: HMP. Analyzed the data: HMP OK ZD. Wrote the paper: HMP OK ZD MDG.

References

1. 1.
   American Psychiatric Association (2013) Diagnostic and Statistical Manual of Mental Disorders (5th ed.). Arlington, VA: Author.
2. 2.
   Petry NM, Rehbein F, Gentile DA, Lemmens JS, Rumpf H, et al.. (2014) An international consensus for assessing internet gaming disorder using the new DSM-5 approach. Addiction.
3. 3. Griffiths MD, King D, Demetrovics Z (2014) DSM-5 internet gaming disorder needs a unified approach to assessment. Neuropsychiatry 4: 1–4.
4. 4.
   Petry NM, O’Brien CP (2013) Internet gaming disorder and the DSM-5. Addiction.
5. 5. King D, Haagsma MC, Delfabbro P, Gradisar M, Griffiths MD (2013) Toward a consensus definition of pathological video-gaming: A systematic review of psychometric assessment tools. Clin Psychol Rev 33: 331–342.
6. 6. Griffiths MD, Kuss DJ, Daniel K (2012) Video game addiction: Past, present and future. Curr Psychiatry Rev 8: 308–318.
7. 7. Kuss DJ, Griffiths MD (2012) Internet gaming addiction: A systematic review of empirical research. Int J Ment Health Addict 10: 278–296.
8. 8. Griffiths MD (1993) Are computer games bad for children? The Psychol 9: 401–407.
9. 9. Griffiths MD, Hunt N (1998) Dependence on computer games by adolescents. Psychol Rep 82: 475–480.
10. 10. Young KS (1998) Internet addiction: The emergence of a new clinical disorder. CyberPsychol Behav 1: 237–244.
11. 11. Rehbein F, Psych G, Kleimann M, Mediasci G, Mößle T (2010) Prevalence and risk factors of video game dependency in adolescence: results of a German nationwide survey. Cyberpsychol Behav Soc Netw 13: 269–277.
12. 12. Demetrovics Z, Urbán R, Nagygyörgy K, Farkas J, Griffiths MD, et al. (2012) The Development of the Problematic Online Gaming Questionnaire (POGQ). PLoS ONE 7: e36417.
13. 13. Gentile D (2009) Pathological Video-Game Use Among Youth Ages 8 to 18 A National Study. Psychol Sci 20: 594–602.
14. 14. Lemmens JS, Valkenburg PM, Peter J (2009) Development and validation of a game addiction scale for adolescents. Media Psychol 12: 77–95.
15. 15.
    American Psychiatric Association (1994) Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.). Washington, DC: Author.
16. 16.
    American Psychiatric Association (2000) Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.). Washington, DC: Author.
17. 17. Tejeiro Salguero RA, Morán RMB (2002) Measuring problem video game playing in adolescents. Addiction 97: 1601–1606.
18. 18. Thomas NJ, Martin FH (2010) Video-arcade game, computer game and Internet activities of Australian students: Participation habits and prevalence of addiction. Aust J Psychol 62: 59–66.
19. 19. Hussain Z, Griffiths MD (2009) Excessive use of massively multi-player online role-playing games: A pilot study. Int J Ment Health Addict 7: 563–571.
20. 20. Griffiths MD (2005) A‘components’ model of addiction within a biopsychosocial framework. J Subst Use 10: 191–197.
21. 21. Wood R, Griffiths MD (2007) Online data collection from gamblers: Methodological issues. Int J Ment Health Addict 5: 151–163.
22. 22. Griffiths MD (2012) The use of online methodologies in studying paraphilias—A review. J Behav Addict 1: 143–150.
23. 23. Joinson AN (2001) Knowing me, knowing you: Reciprocal self-disclosure in Internet-based surveys. CyberPsychol Behav 4: 587–591.
24. 24. Suler J (2004) The online disinhibition effect. CyberPsychol Behav 7: 321–326.
25. 25. Comrey AL (1988) Factor-analytic methods of scale development in personality and clinical psychology. J Consult Clin Psychol 56: 754.
26. 26. Haladyna TM (1992) The effectiveness of several multiple-choice formats. Appl Meas Educ 5: 73–88.
27. 27.
    Muthén LK, Muthén BO (2011) Mplus User’s Guide. Sixth Edition. Los Angeles, CA: Muthén & Muthén.
28. 28.
    Byrne BM (2013) Structural Equation Modeling with AMOS: Basic Concepts, Applications, and Programming, Second Edition. New York, NY: Routledge.
29. 29. Hu LT, Bentler PM (1999) Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct Equ Modeling 6: 1–55.
30. 30.
    Collins LM, Lanza ST (2010) Latent class and latent transition analysis: With applications in the social, behavioral, and health sciences. Hoboken, NJ: John Wiley & Sons.
31. 31. Altman DG, Bland JM (1994) Statistics Notes: Diagnostic tests 1: sensitivity and specificity. Br Med J 308: 1552.
32. 32. Glaros AG, Kline RB (1988) Understanding the accuracy of tests with cutting scores: the sensitivity, specificity, and predictive value model. J Clin Psychol 44: 1013–1023.
33. 33. Altman DG, Bland JM (1994) Diagnostic tests 2: predictive values. Br Med J 309: 102.
34. 34. Charlton JP (2002) A factor-analytic investigation of computer ‘addiction’ and engagement. Br J Psychol 93: 329–344.
35. 35. Charlton JP, Danforth IDW (2007) Distinguishing addiction and high engagement in the context of online game playing. Comput Human Behav 23: 1531–1548.
36. 36. Cole H, Griffiths MD (2007) Social interactions in massively multiplayer online role-playing gamers. CyberPsychol Behav 10: 575–583.
37. 37.
    Pápay O, Urbán R, Griffiths MD, Nagygyörgy K, Farkas J, et al.. (2013) Psychometric Properties of the Problematic Online Gaming Questionnaire Short-Form and Prevalence of Problematic Online Gaming in a National Sample of Adolescents. Cyberpsychol Behav Soc Netw.
38. 38. Griffiths MD, Davies M, Chappell D (2004) Demographic factors and playing variables in online computer gaming. CyberPsychol Behav 7: 479–487.
39. 39. Grüsser SM, Thalemann R, Griffiths MD (2007) Excessive computer game playing: evidence for addiction and aggression? CyberPsychol Behav 10: 290–292.
40. 40.
    Ko C-H, Yen J-Y, Chen S-H, Wang P-W, Chen C-S, et al.. (2014) Evaluation of the diagnostic criteria of Internet gaming disorder in the DSM-5 among young adults in Taiwan. J Psychiatr Res.
41. 41. Crowne DP, Marlowe D (1960) A new scale of social desirability independent of psychopathology. J Consult Psychol 24: 349–354.