RWAs a barrier, Govt may let high-income households compile own spending data

Why in the News

The Ministry of Statistics and Programme Implementation (MoSPI) is considering a separate diary based method of recording expenditure for high income households living in gated societies. The proposal answers a refusal rate that has climbed fastest at the top of the income distribution. It also splits a single national survey across two different collection methods.

What is the Household Consumption Expenditure Survey?

1. What it measures: The Household Consumption Expenditure Survey (HCES) records how much a household spends on goods and services over a reference period. It covers rural and urban households across the country.
2. Who runs it: The National Statistics Office under MoSPI conducts it as a sample survey using tablets to record responses.
3. What the output is used for: The spending shares it produces fix the weights of the Consumer Price Index (CPI) basket, which forms the basis of headline retail inflation. The Reserve Bank of India (RBI) looks at that inflation measure while deciding on interest rates, against a CPI target of 4% within a band of 2% to 6%.
4. How often it runs: It was earlier conducted every five years. Two back to back rounds ran in 2022-23 and 2023-24 after an overhaul of methods, and the ministry now intends a round every three years or so.

What is diary based data collection?

1. The method: The household itself notes down the information as and when the relevant activity occurs, instead of answering a field official at the door. For the HCES this means jotting down monthly spending on different goods and services, ranging from food items to haircuts.
2. The form it may take: The record need not be a physical diary. The ministry may allow such households to enter consumption expenditure details on an online portal.

What is recall error in survey data?

1. The defect: Recall error is the gap between what a household actually spent and what a respondent remembers spending when asked later. It rises with the length of the reference period and the number of items being recalled.
2. Why the diary reduces it: A household writing an entry at the moment of purchase is not relying on memory at all. The error the interview method introduces is therefore absent from the diary record.

How far has participation in official surveys fallen?

1. Urban non response: The overall urban non response rate during the 2022-23 HCES rose to 9.8%, from 2.8% in the 75th round of the National Sample Survey conducted from July 2017 to June 2018.
2. Rural non response: The rural rate rose to 4.1% over the same period, from 1.5%.
3. The most affluent respondents: For the most affluent urban and rural respondents, the non response rate stood at 11% and 3.9% respectively.
4. The earlier baseline: In the 2011-12 survey the corresponding figures for those groups were 3.3% and 1.3%.
5. The scale of the last round: The most recent HCES, conducted from August 2023 to July 2024, surveyed 2.6 lakh households across the country, barring a few inaccessible villages in the Andaman and Nicobar Islands. It sought responses for a total of 405 goods and services.
6. The next round: The next edition is expected to begin in mid-2027 and continue for about a year, with the diary method proposed only for richer households in gated societies on a pilot basis.

Why do affluent households refuse to be surveyed?

1. Physical exclusion by the association: Resident Welfare Associations (RWAs) have cited security as the reason for not permitting survey staff inside gated societies. Field officers already inform the district collector, local bodies and the police station to obtain permission and support before entry.
2. Objection to the questions themselves: RWAs have objected to the sensitive and private nature of some questions asked in government surveys.
3. Fear of onward sharing: RWAs have voiced the apprehension that the details may be shared with other government departments. MoSPI has stated that data privacy is paramount and that the data is anonymised.
4. Inability to remember: Households have cited the difficulty of recalling expenditure details accurately during a door to door interview.
5. Discomfort within the family: Residents have cited unease at answering certain questions in front of family members, such as expenditure on alcohol and cigarette consumption.
6. No perceived reason to participate: MoSPI has recorded a lack of awareness of why these surveys matter for policy, which often leads to outright refusal. Eg. Residents of an affluent society in Gurugram refused to take part in the Time Use Survey.

Why does refusal concentrated at the top distort national estimates?

1. The sample shrinks: A rise in non response rates curtails the achieved sample size of a survey.
2. The sample changes shape: Non responses drawn from one segment leave the final composition of the sample different from what was intended, which produces incorrect estimates from the exercise.
3. Substitution moves the problem, it does not solve it: Where access failed, the ministry substituted the original residential society with a similar one, so the households actually surveyed are not the households the design selected.
4. The refusal is not confined to one survey: Similar incidents have been reported from high rises in Bengaluru, Kolkata, Udaipur, Mumbai and Bhopal for the HCES, the Periodic Labour Force Survey, the Annual Survey of Unincorporated Sector Enterprises and the Urban Frame Survey.
5. Policy is built on these numbers: Government policy is increasingly data and evidence driven, so a biased estimate leads to inappropriate conclusions and decisions that do not produce the desired result.

What does international practice show about diary based expenditure surveys?

1. United Kingdom: The Office for National Statistics runs the Living Costs and Food Survey, in which each adult in a selected household keeps a two week spending diary. The results feed the weights of the United Kingdom consumer price indices.
2. United States: The Bureau of Labor Statistics runs the Consumer Expenditure Surveys in two parts, a quarterly interview component and a separate diary component in which households record purchases for two consecutive one week periods.
3. Japan: The Statistics Bureau runs the Family Income and Expenditure Survey using a household account book kept by the household over a fixed period rather than a single recall interview.
4. Australia: The Australian Bureau of Statistics collects a two week personal expenditure diary from household members in its Household Expenditure Survey, alongside a face to face interview.
5. The limit of the evidence here: The proposal is defended on the ground that the diary method is used in other countries, without naming a country or a comparability finding from any of them.

Can one survey run on two collection methods without breaking its own comparability?

1. Two data sets, one estimate: The practical problem is how data compiled through two different methods will be stitched together into a single national estimate.
2. The error is asymmetric by design: Data collected door to door from poorer households would carry higher recall error than diary based data supplied by richer households. The difference in the numbers would then reflect the method as much as the spending.
3. The asymmetry runs the wrong way: India's survey samples are dominated by the low income group, so the method with the larger error would apply to most of the sample.
4. Literacy sets the boundary: Lower literacy rates in the low income group mean only higher income households can be expected to follow the diary method correctly.
5. The department's own position: MoSPI has stated that the integration of diary compiled data with the main survey is still being worked out and that the proposal is at a planning stage.

Challenges to the diary based collection proposal

1. No legal compulsion behind participation: Voluntary compliance is what has broken down, and a change of instrument does not create an obligation to respond. Eg. Residents of gated societies have simply stated that they do not want to participate in a survey, with no consequence following.
2. Self reporting understates socially sensitive spending: Items respondents are reluctant to declare in front of family are also the items most likely to go unrecorded in a self kept diary. Eg. Expenditure on alcohol and cigarette consumption was named by RWAs as a category respondents avoid.
3. A portal shifts the burden to the respondent: An online entry system asks an unpaid household to do the work a trained investigator was paid to do, which raises the risk of partial and abandoned records. Eg. The ministry already uses tablets for field recording, so the enumerator side of the process is not the bottleneck.
4. A pilot on one income class cannot be validated: Without running both methods on the same households, there is no way to separate a method effect from a real difference in spending. Eg. The 2017-18 consumption expenditure survey was junked in November 2019 after its results were questioned on data quality grounds, showing how a contested method destroys the entire round.
5. Privacy assurance rests on administrative practice: Anonymisation has been promised as a departmental assurance rather than as an enforceable statutory guarantee against onward sharing. Eg. RWAs specifically raised the fear that details would travel to other government departments.
6. Class segregated methods invite challenge to the inflation number itself: A CPI weight derived from two collection systems can be contested on the ground that the two halves are not measuring the same thing. Eg. The food group weight in the CPI was cut sharply on the basis of the 2023-24 HCES, a revision that depends entirely on the survey being internally consistent.

Conclusion

The proposal is at the planning stage, with a diary based pilot intended for high income households in gated societies before the 2027-28 consumption expenditure survey begins. The problem it addresses is real, since non response among the most affluent urban respondents has reached 11% against 3.3% in 2011-12. The unresolved question is the one the ministry itself has flagged, namely how a diary record and a door to door interview can be combined into one estimate when they carry different recall error. Until that is settled, the fix repairs coverage at the cost of comparability.

About India's Consumption and Price Statistics System

1. What the Consumer Price Index measures: It captures the price change experienced by the average urban and rural household across food, housing, transport, healthcare, education, clothing and services. It is the closest approximation to the cost of living for a typical household.
2. How the basket is organised: The CPI is built on 12 divisions of the Classification of Individual Consumption According to Purpose, 2018 (COICOP-2018), covering food and non-alcoholic beverages, pan, tobacco and narcotics, clothing and footwear, housing, water, electricity, gas and other fuels, furnishings and routine household maintenance, health, transport, information and communication, recreation, sport and culture, education, restaurants and accommodation services, and personal care, social protection and miscellaneous items.
3. The weight of food: Food and non-alcoholic beverages carry a weight of about 36.75% in the CPI, revised down from 45.86%.
4. The food price index: The Consumer Food Price Index (CFPI) is derived from Division 1 of COICOP-2018 and is published separately for rural, urban and combined series. Its sub components include cereals, milk, meat and fish, oils and fats, vegetables, fruits, pulses, spices and sugar.
5. Headline against core: Headline inflation includes every item in the basket and swings with monsoons, global crude and supply disruptions. Core inflation strips out food and fuel to give a cleaner read of demand driven, sticky inflation.
6. The wholesale index: The Wholesale Price Index (WPI), on a 2011-12 base, measures what the economy produces and trades at wholesale. Manufacturing alone accounts for about 64% of the WPI, and food articles at the farm gate together with food manufacturing account for only about 24%.
7. How the two indices enter national accounts: Goods producing sectors such as agriculture, mining and manufacturing are deflated using the WPI, since their transactions occur at the wholesale level. Services sectors are deflated using CPI components or dedicated services price indices.
8. Where consumption data feeds employment and enterprise statistics: The Periodic Labour Force Survey (PLFS), launched in 2017-18, tracks employment, workforce participation and unemployment. The Annual Survey of Unincorporated Sector Enterprises (ASUSE) captures output, employment, wages and value added in the informal business economy.

Laws and Rules Governing Official Statistics in India

1. Collection of Statistics Act, 2008: Provides the legal framework for the collection of statistics on economic, demographic, social, scientific and environmental matters by the Centre, States and local bodies.
2. It empowers a statistics officer to require information and penalises wilful refusal or supply of false information.
3. The Collection of Statistics (Amendment) Act, 2017 extended the framework to the erstwhile State of Jammu and Kashmir and clarified the Centre's powers over subjects in the Union and Concurrent Lists.
4. Collection of Statistics Rules, 2011: Lay down the procedure for notification of a statistical survey, appointment of statistics officers, service of notices and the handling of returns.
5. Census Act, 1948: Governs the conduct of the decennial Census and the appointment of census officers.
6. It makes information given to a census officer confidential and inadmissible as evidence, a confidentiality guarantee the Collection of Statistics framework does not replicate in the same terms.
7. Registration of Births and Deaths Act, 1969: Provides the civil registration system that supplies vital statistics independent of survey estimates.
8. Digital Personal Data Protection Act, 2023: Governs the processing of digital personal data and shapes how identifiable household records collected in surveys may be stored and shared.
9. Right to Information Act, 2005: Provides the route through which unit level survey data and methodology notes are sought from statistical agencies.

Government Initiatives

1. National Statistical Commission: Constituted in 2005 on the recommendation of the Rangarajan Commission, it advises on statistical priorities, standards and the release calendar of official statistics.
2. eSankhyiki portal: A MoSPI platform that brings macro indicators and survey outputs into a single searchable data lake for public and departmental use.
3. National Data and Analytics Platform: A NITI Aayog initiative to standardise and publish government datasets in machine readable form for researchers and administrators.
4. Data Governance Quality Index: Scores ministries and departments on the quality of their administrative data systems, aimed at raising the reliability of data generated outside sample surveys.
5. Revamped Periodic Labour Force Survey: From January 2025 the survey shifted to the calendar year, expanded its sample and moved to monthly reporting of key labour market indicators.
6. Sustainable Development Goals National Indicator Framework: Maintained by MoSPI, it fixes the national indicators against which progress on the Sustainable Development Goals is reported.

Key Facts about India's Statistical System

1. National Statistics Day: Observed on 29 June, the birth anniversary of Prasanta Chandra Mahalanobis, recognised as the architect of India's sample survey system.
2. World Statistics Day: Observed on 20 October, designated by the United Nations Statistical Commission.
3. Origins of the survey system: The National Sample Survey was set up in 1950 on Mahalanobis's initiative, making India one of the earliest large scale household survey systems in the developing world.
4. Institutional merger: The Central Statistics Office and the National Sample Survey Office were merged into the National Statistical Office in May 2019.
5. International standards: India was among the first countries to subscribe to the International Monetary Fund's Special Data Dissemination Standard, in 1996.

Back2Basics: National Sample Survey

1. What it is: A nationwide, large scale sample survey system that collects household and enterprise data through successive rounds, each round running for a fixed period.
2. Who runs it: The National Statistical Office under MoSPI, through a field operations wing with offices across the country.
3. How rounds work: Each round carries a principal subject, such as consumption expenditure, employment and unemployment, health, education or land and livestock holdings, with subjects rotating across rounds.
4. Design: It uses a stratified multi stage sample design covering rural and urban areas, with villages and urban blocks as first stage units and households as ultimate units.
5. Why the round number matters: Round numbers identify the survey period, so the 75th round refers to the survey conducted from July 2017 to June 2018.

Challenges in India's Official Statistical System

1. The sampling frame ages between Censuses: Village lists and urban blocks used to draw samples are anchored to the last Census, so the frame drifts from reality as migration and new construction accumulate. Eg. The decennial Census due in 2021 was deferred, leaving the 2011 Census as the frame for over a decade of surveys.
2. Base years lag the structure of the economy: An index built on an old base assigns weights drawn from a consumption or production pattern that no longer exists. Eg. The Wholesale Price Index still uses 2011-12 as its base year.
3. Comparability breaks at every methodological revision: A redesigned questionnaire produces a series that cannot be compared with its own predecessor, which destroys the ability to measure change. Eg. The 2011-12 and 2022-23 consumption rounds used different questionnaire designs, so poverty change between them cannot be read off directly.
4. Contested releases erode trust in the system: A withheld or discarded round leaves policy without a number and invites the charge that inconvenient results are suppressed. Eg. Two members of the National Statistical Commission resigned in January 2019 over the withholding of employment survey results.
5. No updated official poverty line: Welfare targeting continues on a threshold fixed against a consumption pattern from an earlier decade. Eg. No official poverty line has been revised since the estimates based on 2011-12 data.
6. Administrative data sits outside the statistical system: Rich transaction records held by other departments are not routinely used to validate or supplement survey estimates. Eg. Goods and Services Tax returns, e-Shram registrations and direct benefit transfer records are maintained in separate systems from the household survey series.
7. Privacy law raises the cost of collection: Stricter obligations on identifiable personal data increase the compliance burden on an agency that collects household level detail at scale. Eg. The Digital Personal Data Protection Act, 2023 applies to digital personal data held by government bodies with limited carve outs.

Way Forward

1. Run both methods on the same households first: Conduct a calibration study in which a subset of households is covered by interview and diary together, so the method effect can be measured and adjusted before the two data sets are combined.
2. Give the survey a statutory response obligation with a privacy guarantee: Invoke the notification powers under the Collection of Statistics Act, 2008 for the HCES, paired with a published confidentiality and anonymisation protocol that binds onward sharing.
3. Shorten reference periods rather than change the respondent's job: Use shorter recall windows and item specific reference periods to cut recall error for the interview sample instead of relying on the diary alone.
4. Publish non response by income group with every release: Report achieved sample and non response rates decile wise alongside each estimate, so users can see where the sample is thin.
5. Negotiate access through housing federations rather than society by society: Build standing memoranda with apex RWA federations and municipal bodies so that field access does not depend on a fresh permission at every gate.
6. Refresh the sampling frame on the 2027 Census: Rebuild urban blocks and rural village lists on the new Census the moment enumeration closes, so the diary pilot is drawn from a current frame.
7. Use administrative data as a cross check: Validate high income consumption estimates against Goods and Services Tax turnover, card and digital payment aggregates and vehicle and property registration data, without linking them to individual households.

Matching Previous Year Question

“[2020] Consider the following statements: 1. The weightage of food in Consumer Price Index (CPI) is higher than that Wholesale Price Index (WPI). 2. The WPI does not capture changes in the prices of services, which CPI does. 3. Reserve Bank of India has now adopted WPI as its key measure of inflation and to decide on changing the key policy rates. Which of the statements given above is/are correct? (a) 1 and 2 only (b) 2 only (c) 3 only (d) 1, 2 and 3 | Answer: (a)”