Monitoring Food Affordability: Reliability and Validity of an Online Nutritious Food Basket

Food prices were collected online during a 1-week period in February 2021 in 12 grocery stores from four major grocery retailer chains, including premium and value banners. Store name and information are confidential as per our agreement with stores when we requested permission to access stores for this project. Stores sampled were required to be near PHUs across four regions in Ontario: Southwestern, Central, Eastern, and Northwestern for logistical reasons. PHUs were selected based on their geographic dispersion and the availability of major online retailers. Store addresses closest to PHU offices were entered on store websites and two raters independently collected food price data from all stores. Through searching and sorting, raters identified the lowest price per kilogram or defined purchase unit for each item in the ONFB. Sale items were priced provided they were not part of multi-price schemes (i.e. discounts for purchasing multiple items). Each rater completed 15 online store audits that allowed for eight intra-rater and ten inter-rater comparisons across 61 food items and seven food categories (see supplemental files¹ for ONFB items and categories).

Intra-rater reliability was measured as the difference in prices recorded by a single rater during two different times (morning and evening) on the same day from the same store. Inter-rater reliability was measured as the difference in prices recorded by the two raters for a single food item on the same day from the same store. The percentage perfect agreement two-way random intra-class correlations (ICCs) of prices between and within raters was calculated for each item in the ONFB across all stores. Since the raters aimed to record the item with the lowest price per kilogram, particular brands/products were not specified in the costing tool beyond the description and purchase units indicated.

After minor edits to the online tool and process following reliability assessment (e.g. clarified product types, decision pathway when products are unavailable), a second round of food price data collection was conducted in February 2022. The ONFB was priced in-store and online for 28 grocery stores from four major grocery retailer chains, including premium and value banners, that were purposively selected for diversity in geographic location and population size. Nineteen stores in the South (9 small/medium, 10 large) and 9 stores in the North (7 small/medium, 2 large) were included. Population centres were classified following Statistics Canada population size cut-offs (small: 1,000–29,000; medium: 30,000–99,999; large: 100,000+ persons) and the North/South regional classification followed alPHa categorization [9,10].

Two raters completed in-store and online food costing for the same store on the same day, recording product name/brand, purchase unit, and price. It was not possible to have the same individual complete costing by both methods for one store due to logistical challenges and capacity. Both in-store and online raters recorded the lowest price per kilogram or defined purchase unit for each item in the ONFB. Raters were encouraged to communicate with their partner to ensure the same products and lowest price possible were recorded by both raters, reducing the likelihood of inter-rater error obscuring the validity of the online method. When discrepancies existed, raters always recorded the cheapest items available by their method. To estimate agreement between online and in-store prices, percent agreement was calculated for all items and matched items (i.e. when raters recorded the exact same product/brand and purchase unit). Percent perfect agreement, paired t-tests, and Pearson’s correlation for all (n = 1708 pairs) and matched items (n = 1134) were calculated.

Since the purpose of the ONFB is to estimate the cost of a healthy food basket for a household, the difference in online and in-store food basket costs was examined as part of the validity assessment. Using a spreadsheet from Statistics Canada that generates the cost of the NNFB for each age and sex group, we calculated the food cost online and in-store for a family of 4 (female age 4–8; male age 14–18; male age 31–50; female age 31–50). The total cost per family per day was multiplied by 1.05 to account for miscellaneous items and then by 4.33 to calculate monthly costs. Usually, a Household Size Adjustment Factor is applied that accounts for economies of scale (lower cost per person to feed a larger family) where the weekly cost of the food basket is adjusted by 0.9–1.20, but the Household Size Adjustment Factor is 0.0 for a family of four so no adjustment was made.

IBM SPSS Statistics for Windows v28.0.1.0 (2021, IBM Corp., Armonk, New York) was used for all analyses with statistical significance level set at α = 0.05.

Inter-rater and intra-rater reliability was high for all stores, surpassing 80% and 90% agreement, respectively. Intra-rater reliability was high for all food categories (88–100%), but inter-rater reliability varied (Table 1). The foods within “Often close to dairy case”, “Bread or Bakery”, and “Meat Department” categories had lower percent agreement (73%, 63%, and 57% respectively). Overall, there was an average 95.4% intra-rater and 81.6% inter-rater agreement across all stores and food categories (ICC = 0.972, F = 69.9, p < 0.001).

To the authors’ knowledge, this is the first study to assess the reliability and validity of an online approach to food cost monitoring. Though disparate pilot studies employing online methods have been published [8], to date, no online approaches have been compared against the gold standard of in-person food costing. The results of this study provide meaningful insight into the modernization of food cost monitoring efforts in Ontario, Canada, and worldwide.

Some limitations must be noted. First, due to reliance on consumer-facing web platforms, this approach is limited to large multi-chain retailers with an online presence. As such, smaller local retailers, particularly independently owned ethnic grocery stores, continue to require in-store costing provided they meet the Ontario Dietitians in Public Health – Monitoring Food Affordability in Ontario subgroup criteria. This may pose concerns in more remote communities, which may be primarily served by independently owned local retailers. In effect, though there is homogeneity of large multi-chain stores, there may be cost difference regionally when in-store and online costing are integrated in a hybrid process. Nonetheless, due to larger national/provincial chains being the focus of the analysis, generalizability of findings is improved. Second, the ONFB excludes more expensive foods, including some higher priced convenience foods and pre-prepared items. As such, the calculated cost is likely lower than what consumers actually spend [17]. Further, the basket may not be representative of the typical diet. Finally, our data collection occurred during winter months, where most food costing activities in Canada usually take place in the spring or fall. Although food prices are affected by seasons, we do not have any reason to believe that the reliability and validity of online pricing would be affected by seasonality. We expect that any differences between in-store and online prices that would make the tool more or less valid would be more likely to be random than systematic errors due to seasons. Thus, we believe our findings on reliability and validity to be reasonably generalizable to the full year. The reliability and validity of online food pricing methods our results support may be time-bound due to its collection during the COVD-19 pandemic. Reliability and validity of tools are only as static as the contexts in which they are used. The design or function of online food purchasing platforms is likely to evolve due to the multiple micro and macro factors impacting food prices and consumer behaviour in Canada [18], and thus online food pricing methods should be reviewed in the future and adapted as necessary.