How reliable is internet-based self-reported identity, socio-demographic and obesity measures in European adults?

Abstract

In e-health intervention studies, there are concerns about the reliability of internet-based, self-reported (SR) data and about the potential for identity fraud. This study introduced and tested a novel procedure for assessing the validity of internet-based, SR identity and validated anthropometric and demographic data via measurements performed face-to-face in a validation study (VS). Participants (n = 140) from seven European countries, participating in the Food4Me intervention study which aimed to test the efficacy of personalised nutrition approaches delivered via the internet, were invited to take part in the VS. Participants visited a research centre in each country within 2 weeks of providing SR data via the internet. Participants received detailed instructions on how to perform each measurement. Individual’s identity was checked visually and by repeated collection and analysis of buccal cell DNA for 33 genetic variants. Validation of identity using genomic information showed perfect concordance between SR and VS. Similar results were found for demographic data (age and sex verification). We observed strong intra-class correlation coefficients between SR and VS for anthropometric data (height 0.990, weight 0.994 and BMI 0.983). However, internet-based SR weight was under-reported (Δ −0.70 kg [−3.6 to 2.1], p < 0.0001) and, therefore, BMI was lower for SR data (Δ −0.29 kg m⁻² [−1.5 to 1.0], p < 0.0001). BMI classification was correct in 93 % of cases. We demonstrate the utility of genotype information for detection of possible identity fraud in e-health studies and confirm the reliability of internet-based, SR anthropometric and demographic data collected in the Food4Me study.

Trial registration: NCT01530139 (http://clinicaltrials.gov/show/NCT01530139).

Introduction

Non-communicable diseases (NCD) account for over half of global deaths (WHO 2010), with 4 million deaths annually attributed to cardiovascular diseases (CVD) alone (Nichols et al. 2013). Because modifiable risk factors, notably diet, smoking and physical activity (PA), account for more than 80 % of deaths from CVD and cerebrovascular diseases (WHO 2010), effective lifestyle-based interventions are important for minimising NCD burden. However, current strategies to improve diet and PA result in relatively modest behavioural changes (Hobbs et al. 2013; Lara et al. 2014) and may have limited ability to reduce NCD-related mortality. Traditionally, face-to-face interventions have been used to promote behavioural changes. By 2015, 85 % of the EU population are predicted to be internet users (Eurostat 2012) and internet-based interventions are increasing. The degree of behavioural change achievable via internet-based interventions is similar to (Steele et al. 2009; Weigold et al. 2013), or potentially greater than (Wantland et al. 2004), those conducted face-to-face.

The advantages of administrating nutritional interventions via the internet include scalability, efficient and cost-effective collection of data, and lower respondent and researcher burden (Celis-Morales et al. 2015). On the other hand, intervention studies conducted remotely via the internet may incur problems of fidelity in the self-reported (SR) data and in the collection of biological samples, the provenance of which may be uncertain or unreliable. Furthermore, SR anthropometric data may be prone to respondent biases and measurement errors. Validation studies (VS) in which trained researchers repeat measurements in a sub-sample of the population are integral to ensure the quality of data collected in internet-based interventions and provide some reassurance (Thorndike et al. 2011). However, verification of participant identity appears to have been neglected in previous validation studies (Bes-Rastrollo et al. 2011; Bonn et al. 2013; Lassale et al. 2013; Pursey et al. 2014). Using the internet to recruit participants into intervention studies delivered remotely provides opportunities for participant misrepresentation (identity fraud, i.e. pretending to be who they are not) which may undermine the objectives and findings of the study.

The Food4Me study, an internet-based randomised controlled trial (RCT) conducted across seven European countries, was designed to test the efficacy of personalised nutrition (PN) approaches on health-related outcomes (Celis-Morales et al. 2014). Using data from the Food4Me study, the present paper introduces a novel approach for validating participant identity and describes outcomes from a VS to assess the validity of internet-based, SR anthropometric, demographic and identity data, compared with standardised measurements performed face-to-face.

Methods

The present VS was performed in a subsample of the Food4Me PoP study, a four-arm, internet-based RCT conducted across seven European countries on the efficacy of PN approaches on health-related outcomes(Celis-Morales et al. 2014).

Design of the proof of principle study

The Food4Me PoP study protocol has been described in detail (Celis-Morales et al. 2014). In brief, participants across seven European countries were recruited via the internet to emulate an internet-based PN service. Recruitment was aided by local and national advertising via the internet, radio advertisements, posters, e-flyers, the use of social media and word of mouth. Identical standardised protocols for recruitment were used in the seven European countries, aiming for 1540 participants (i.e. 220 participants per country). The PoP study recruitment sites were: University College Dublin (Ireland); Maastricht University (The Netherlands); University of Navarra (Spain); Harokopio University (Greece); University of Reading (UK); National Food and Nutrition Institute (Poland); Technische Universität München (Germany).

Eligibility criteria

Participants aged ≥18 years were included in the study. To keep the cohort representative of the adult population, a minimal set of exclusion criteria were applied: (a) pregnancy or lactation; (b) no or limited access to the internet; (c) following a prescribed diet for any reason, including weight loss, in the last 3 months; (d) insulin-dependent diabetes, coeliac disease, Crohn’s disease or any metabolic disease or condition that alters nutritional requirements, e.g. food intolerances or allergies.

PoP study measures

Participants consented to report their measurements via the internet and to return self-collected biological samples (Dried Blood Spot Cards and Buccal swabs) by post, using prepaid stamped addressed envelopes. To ensure that procedures were similar in all recruiting centres, standardised operating procedures were prepared for all measurements, and researchers underwent centralised training. In addition, to enable participants to collect and report the required information and to collect, process and dispatch the biological samples correctly, participants were given printed detailed instructions, and video demonstrations of key procedures were available online. All instructions were provided in the local language.

Collection of demographic and anthropometric data

An online screening questionnaire collected detailed SR information about demographic, food choices, health and anthropometric data. Body weight, height and upper thigh, waist and hip circumferences were self-measured and reported by participants via the internet. Participants were instructed to measure body weight after an overnight fast, without shoes and wearing light clothing using a home or commercial scale, and to measure height, barefoot, using a standardised measuring tape provided by Food4Me (Celis-Morales et al. 2014).

Genotypic analyses

Buccal cell samples were collected from participants at baseline using Isohelix SK-1 DNA buccal swabs and Isohelix dried-capsules and posted to each recruiting centre for shipment to LCG Genomics (Hertfordshire, UK). LCG Genomics extracted DNA and genotyped 33 loci using KASP™ genotyping assays to provide bi-allelic scoring of single nucleotide polymorphisms (SNP) and insertions and deletions at specific loci (He et al. 2014).

Validation study design

To validate the SR demographic (identity, age and sex) and anthropometric (height, weight and estimated BMI) data, an intervention arm-balanced sub-sample of 140 participants (approximately 20 participants per country) from the PoP intervention study were randomly selected and invited to take part in the VS. Whereas participants for the intervention study were recruited nationally, for logistic reasons, participants living near research centres participated in the VS. Upon completion of the PoP online survey and measurements, participants attended a measurement session at their national research centre. To minimise variations in body mass due to time lags between the completion of SR measures online and the appointment at the research centre, participants were instructed to visit the centre within 2 weeks of their last completed online measurements.

At the research centre, researchers measured height and weight after an overnight fast, assessed sex visually, confirmed participant’s age and collected buccal cell samples which were sent to LGC Genomics to replicate genotyping of the 33 loci previously genotyped in baseline samples of the PoP study. Concordance between both sets of genotypic data was used to confirm participant identity.

Ethical approval and participant consent

The Research Ethics Committees at each centre administering the intervention granted ethical approval for the VS. Before participation, all participants signed two online consent forms, which were automatically directed to study investigators to be counter-signed and archived. All Ethical Committees accepted an online informed consent procedure, with the exception of The Netherlands and Germany whose ethics committees requested additional hard copy consent forms, which were posted to the respective recruitment centres. The Maastricht University Ethics Committee specified that an extra 10 % of the participants should be invited to participate to confirm their demographic SR data (age and sex). This check was performed by teleconference.

Data analysis

SR and VS data are presented as mean ± SD for continuous variables and as percentages for categorical variables. Kolmogorov–Smirnov tests for normal distribution were used for continuous variables. Differences between SR and measured height, weight and calculated BMI were assessed using paired t tests. Simple and multiple regression analyses were used to investigate determinants of differences between SR and measured values. General linear models were used to investigate differences between SR and measured values by age group, sex and country.

Intra-class correlation coefficients (ICC) were used to quantify associations and Bland–Altman analyses to investigate the degree of agreement between SR and measured height, weight and BMI (Bland and Altman 2010). Cohen’s κ statistics and the corresponding 95 % confidence interval (CI) for classification were used to assess the concordance of sex, age group and BMI status (underweight, normal weight, overweight and obesity) derived from SR and measured values. The degree of agreement between measured and SR overweight and obesity was assessed as follows: κ < 0 was none/poor; 0 ≤ κ ≤ 0.20 was slight; 0.21 ≤ κ ≤ 0.40 was fair; 0.41 ≤ κ ≤ 0.60 was moderate; 0.61 ≤ κ ≤ 0.80 was substantial; and 0.81 ≤ κ ≤ 1.0 was almost perfect (Landis and Koch 1977). The sensitivity and specificity of correctly classified BMI based on the SR data were assessed by ROC analysis. Data analyses were performed using STATA/SE version 13 (StataCorp. College Station, TX, USA) and MedCalc version 12 (Ostend, Belgium).

Results

Participant characteristics

Table 1 summarises characteristics of the 1607 Food4Me participants, and the sub-sample in the VS (n = 140). Of 194 participants invited to take part in the VS, 43 were unable to visit the research centre because of location, time constraints or personal reasons and 11 invitees did not respond. The baseline characteristics of these participants who did not take part in the VS were similar to those of participants who accepted to take part in the VS (age 41.3 ± 13.9; weight 72.8 ± 15.6; BMI 25.3 ± 4.7). Demographic and anthropometric characteristics of VS participants were similar to those of the Food4Me PoP Study participants (Table 1).

Table 1 Demographic and anthropometric characteristics of the Food4Me Proof of Principle (PoP) study and validation study participants

Validity and reliability of self-reported data

SR weight was slightly lower than measured weight (Δ −0.70 kg, SD 1.5, range −6.0 to 5.9, p < 0.0001), but there was no significant difference between SR and measured height (Δ 0.19 cm SD 1.2, range −3 to 5, p = 0.066). Thus, BMI calculated from SR height and weight was slightly lower (Δ −0.29 kg m⁻², SD 0.6, range −2.2 to 1.7, p < 0.0001) than measured values. There were no significant differences between SR and measured values by age group (<45 and ≥45 years), but men overestimated whereas women underestimated height (Table 2). Overweight and obese participants showed higher levels of under-reporting of body mass compared with normal weight participants (p < 0.0005). Results stratified by country are presented in supplementary material (Table S1).

Table 2 Summary statistics and correlation coefficients for self-reported and measured height, weight and BMI

Strong correlations (ICC) were observed between SR and measured values for height [0.990 (95 % CI 0.987–0.993), p < 0.0001], weight [0.994 (0.991–0.995), p < 0.0001] and BMI [0.983 (0.977–0.988), p < 0.0001; Table 2].

Self-reported and measured values

Outcomes of Bland–Altman analyses of SR versus measured values for height, weight and BMI with the corresponding lower and higher level of agreement (LOA) showed a small systematic under-reporting bias for SR weight [Δ −0.70 kg (LOA −3.6 to 2.1), p < 0.0001] and BMI [Δ −0.29 kg m⁻² (LOA −1.5 to 1.0), p < 0.0001] compared with the measured values (Fig. 1; Table 3). We noted trends for greater under-reporting with increasing body weight and BMI. Bland–Altman results stratified by country are presented in supplementary material (Table S2).

Fig. 1

Bland–Altman plots illustrating the agreement between self-reported (SR) and measured a height, b weight, c BMI and the corresponding means estimated by the two methods across all countries. Solid lines are mean differences, and dotted lines are the lower and upper 95 % limits of agreements; red lines illustrate the regression line for differences in measurements against the mean of both SR and VS measurements

Table 3 Bland–Altman analyses for self-reported and measured height, weight and BMI

Concordance of demographic and BMI classification

There was a strong concordance for BMI classification (underweight, normal, overweight and obese), estimated from SR and measured height and weight, weighted κ 0.94 (95 % CI 0.89–0.99). Five overweight participants (3.5 %) were incorrectly classified as being normal weight by the SR method. Of those who were obese, just one participant (0.7 %) was incorrectly classified as overweight using SR values, leading to a sensitivity of 94.1 % and a specificity of 87.8 % (Table 4).

Table 4 Validity and concordance of weight classification estimated from self-reported and measured values

Validation of identity

To validate the identity of the participants, the 33 SNPs genotyped previously for the intervention study were re-genotyped and the two datasets were compared. At the VS visit, we collected new buccal cell samples (n = 140) from which we obtained reliable genotypes for 135 (33 SNP × 135 individuals = 4455 genotypes). For the remaining five samples, the poor DNA quality precluded informative analysis. There was perfect genotype concordance between original and repeat samples for all but four participants, who had a total of four instances at two distinct SNPs (rs2282679, rs4680) where genotypes did not agree. This mismatch incidence is very low, 4/4455 = 0.09 %, and falls within accepted values for this technology (Smith et al. 2012). To explore possible reasons for the apparent genotype mismatches, DNA sequences in the neighbourhoods of these two SNPs were examined for possible copy number variants (CNVs). This analysis revealed that the two SNPs mapped to known CNVs. Participant sex and age showed perfect concordance between SR data and researcher assessed data.

Discussion

Main findings

A novel aspect of this study was the application of genotype analysis using DNA from buccal cell samples to validate the identity of participants recruited via the internet. By replicating the analysis of 33 genetic variants, we showed 99.9 % concordance between patterns of genotypic variants in DNA collected in the VS and those observed in DNA obtained from previous, self-collected buccal cell samples. This demonstrates the utility of this novel approach for identity checking—a potentially sensitive aspect of internet-based interventions delivered remotely which has not been investigated in earlier studies. In addition, our findings provide further evidence that SR data via internet for height, weight and BMI showed a high degree of reliability compared with face-to-face measurements made by experienced researchers using standard protocols. Concordance for BMI classification between SR and measured data was strong, and we observed perfect agreement for SR sex and age with that assessed in the VS.

Validation of participant identity

Administrating lifestyle-based interventions via the internet offers advantages of scale, efficiency and cost-effective data collection (Wright 2005; Celis-Morales et al. 2014). Nevertheless, internet-based intervention studies conducted remotely may result in problems of reliability in the recruitment of participants and in the collection of biological samples. To the best of our knowledge, the issue of validation of participant identity appears to have been overlooked in previous validation studies. Inevitably, the use of internet to recruit participants to intervention studies provides undesirable opportunities for participant misrepresentation, which may undermine the study objectives. In the current VS, we replicated the analysis of 33 genetic variants as a proxy of validation of identity. We found strong agreement for over 99.9 % of participant genotypes, with just four examples showing disagreement. As our results showed a perfect concordance for age and sex verification, these minor mismatches represent technical errors during genotyping or may reflect the presence of copy number variants (CNVs), which complicate genotyping. LGC Genomics reports that the average genotyping error in positive control DNA samples using Kompetitive Allele Specific PCR, or KASP™, is between 0.7 and 1.6 % and the assay design success rate is between 98 and 100 % (Semagn et al. 2014). We conclude that it is likely that we had perfect agreement in participant identity between samples collected remotely during the Food4Me study and those collected in the VS. Furthermore, we suggest that this novel genotype-based approach to validation of participant identity may be used in many internet-based observational and intervention studies.

Comparison with other studies

The magnitude of differences between SR and measured height (0.19 cm, SD 1.2), weight (−0.70 kg, SD1.5) and BMI (−0.29 kg m⁻², SD 0.6) observed here is similar to findings from previous internet-based studies in adult populations. NutriNet-Sante (Lassale et al. 2013) a French internet-based prospective cohort study including a VS in a sub-sample of 815 adults found that height was over-reported by 0.56 cm (SD 2.4) and that weight and BMI were under-reported by 0.49 kg (SD 1.4) and 0.34 kg m⁻² (SD 1.5), respectively. A study conducted in 177 adults (aged 18–35 years) in Australia (Pursey et al. 2014) observed a larger over-reporting bias for height (1.36 cm, SD 1.9), and a similar under-reporting bias for weight (−0.55 kg, SD 2.0) and BMI (−0.56 kg m⁻², SD 0.08) compared with the present study. In contrast, an internet-based study conducted in 149 adults in Sweden (Bonn et al. 2013) reported larger differences between SR and measured weight (1.2 kg, SD 2.6) compared with our results. A systematic review (Gorber et al. 2007) of validation of SR anthropometric data found that height was over-reported by 0.6–7.5 cm, whereas weight and BMI were under-reported by −0.1 to 6.5 kg and 0 to −2.2 kg m⁻², respectively. It should be noted that under-reporting of body weight is quite common particularly among overweight and obese subjects (Johansson et al. 1998; Spencer et al. 2002; Merrill and Richardson 2009; Lassale et al. 2013).

In agreement with some (Niedhammer et al. 2000; Spencer et al. 2002; Merrill and Richardson 2009) but not all previous studies (Bonn et al. 2013; Lassale et al. 2013), men in the Food4Me study were more likely to over-report height. Although women appeared more likely to under-report weight than men, this difference was not significant in our study. Previous studies have observed that women were significantly more likely to under-report their weight compared with men (Spencer et al. 2002; Merrill and Richardson 2009; Lassale et al. 2013). Whilst height was more likely to be over-reported with increasing age in previous studies (Kuczmarski et al. 2001; Bes-Rastrollo et al. 2011; Lassale et al. 2013), we did not find any effect of age on differences between SR and measured height.

In addition to sex and age, BMI was a strong predictor of differences between SR and measured methods. As a consequence of misreporting of the primary measurements of height and weight, differences in under-reporting of calculated BMI were 4.8 times higher in both overweight and obese individuals compared with normal weight participants (Δ −0.12, −0.54 and −0.53 kg m⁻² for normal, overweight and obese participants, respectively). Our results confirm previous findings of under-reporting of BMI by 0.16, 0.36 and 0.63 kg m⁻² for normal weight, overweight and obese participants, respectively (Lassale et al. 2013). However, we found smaller differences in weight misreporting between BMI categories than those observed by another internet-based study (Pursey et al. 2014) in which under-reporting among overweight and obese participants was −1.36 kg compared with −0.31 kg in those of normal BMI. A possible explanation for the greater degree of misreporting of body weight by overweight and obese individuals lies in the social desirability concept, which argues that perceptions are influenced by desires to conform to perceived societal norms and that, with respect to body weight, such pressures apply more strongly in obese participants (Larson 2000). However, the estimated proportion of subjects for whom SR height, weight and calculated BMI were within 5 % of the measured values were 100 % (n = 140) for height, 96 % (n = 135) for weight and 92 % (n = 129) for estimated BMI, respectively. This suggests that most Food4Me participants provided reliable measures of their anthropometrics.

Concordance of BMI classification

One of the main concerns arising from data collection, either SR via the internet or SR with paper-based forms, is the validity and accuracy of the data provided and its utility as a basis for provision of health-related advice. Several studies have reported greater underestimation of weight (and BMI) with remote SR collection methods than with face-to-face interviews (Hood et al. 2012). However, we observed a good agreement between the BMI classifications derived from SR and measured height and weight (κ = 0.939), with just six participants being wrongly classified when SR data were used. There were no differences in the proportions of those classified as underweight, and only small differences in the proportions of normal weight (3.6 %), overweight (−2.9 %) and obese participants (−0.7 %). These results are comparable with previous findings reporting a κ of 0.97 for BMI classification and prevalence differences between SR and measured values of 0.6 and 0.7 % for overweight and obese participants, respectively (Lassale et al. 2013). Similarly, Pursey et al. (2014) reported that the prevalence of overweight was 2.6 % lower when using SR compared with measured values, but there was no difference for obesity prevalence.

Although social desirability may drive differences between SR and measured values (Joinson 1999), we found very good agreement between the internet-based SR and validation measures for the key anthropometric variables height and weight, suggesting that, in an internet-based setting, participants may be less prone to social desirability bias. This apparently enhanced truthfulness may result from the greater feeling of anonymity when using the web rather than other media such as the telephone (Joinson 1999). However, the reliability of more difficult self-measurements such as waist and hip circumferences need to be explored in future studies.

Strengths and limitations

To our knowledge, this is the first internet-based study that has validated participant identity using genotypic analysis. Our findings of the utility, and practicability, of this approach to validation of participant identity provide proof of concept for remotely conducted, e.g. internet-based, studies in which participant misrepresentation is a potentially major, and often ignored, concern. A particular strength of this study was the collection of data via a novel internet-based server in European countries from a relatively large sample of the adult population with a wide range of ages and BMIs. Our ability to obtain reliable SR anthropometric data was enhanced by the use of standardised protocols by study participants. Protocols were provided in text format with pictures, but also as a series of online videos. In addition, during the VS, trained researchers collected the anthropometric data using the same standardised protocols. An additional strength of our study was the short period of time (i.e. up to 2 weeks) between the collection of internet-based SR data and direct measurement by the researchers. Furthermore, to ensure independence of measurements in the subsequent VS, subjects were invited to participate in the VS only after they had completed their internet-based measures.

A potential limitation of our study is that participants in the Food4Me study were recruited from those showing interest in an intervention study on PN (Livingstone et al. 2015). As a result, we may have recruited those with a particular interest in lifestyle-based interventions, but we have no reason to believe that this interest influenced the truthfulness of SR data. In addition, the BMI distribution among Food4Me participants was comparable with the prevalence of normal weight, overweight and obesity in the adult European population (OECD 2012; Celis-Morales et al. 2014; Livingstone et al. 2015).

In conclusion, we introduced and tested a simple genotype-based approach for validation of the identity of study participants recruited to internet-based studies. This approach is simple and robust, and given the low costs of genotyping we envisage that it may have wide utility for identity validation in the many types of studies (including internet-based studies) where participant recruitment and sample data collection are conducted remotely. Although overall agreement between SR and measured values was excellent, under-reporting of weight was more common among overweight and obese individuals, and such SR data should be interpreted with caution when adiposity is an important outcome. Overall, our findings clearly demonstrate the reliability of internet-based, SR anthropometric and demographic data collected in the Food4Me study.