A Population Stitched from Its Statistics

The problem of choice

Anyone who wants to answer a question about a population faces the same problem of choice. You can go to the survey that asked the question closest to yours and accept whatever cut of the population it happened to publish. You can commission your own survey, and wait a year, and spend a budget, and still end up with a sample too small to say anything about the subgroup you cared about. Or you can take two published tables that each answer half your question and try to join them, and discover that they were drawn from different samples, in different years, with different definitions of a household, and that the join you want does not exist.

The same problem faces very different people. An engineer who needs a realistic staging database, so that the code paths depending on the relationship between age, income, tenure and occupation actually get exercised. A modeller who needs a population for an agent-based simulation of a city’s housing market, and needs it to age, earn and turn over at rates that were actually recorded. An analyst who wants to know how many households hold a particular combination of characteristics, and can find each characteristic’s share on its own but not their intersection. A researcher who wants to follow a cohort through time, and has only a sequence of unrelated cross-sections.

In every case the difficulty is the same. What is published about a population is a collection of slices. Each slice is careful, documented and correct. No two slices were cut from the same block.

Slices of a population

Consider what the public record actually gives you about any large city. A census microdata sample describes demographics, work, income, housing and commuting for the same household, which is enormously valuable, and says nothing about that household’s wealth. A national wealth survey describes assets and debts in detail, interviews a few thousand families, and cannot say anything about one metro area. A city housing survey knows about rent regulation, building condition and tenure length, and little about balance sheets. A regional bank publishes debt by type per capita by year. A labour statistics agency publishes spending shares by income group for the metro. A polling organisation publishes religious affiliation for the metro with a margin of error. A public health survey publishes smoking, insurance and self-reported health by borough and age. Each is a slice, and there are dozens of them, and the number grows every year.

The point that is easy to miss is that these slices are not independent pieces of information about different populations. They are all measurements of the same population, taken through different windows. Every one of them is a constraint on what the underlying population can look like. If enough of them are honoured at once, the set of populations consistent with all of them is small, and any member of that set is a close facsimile of the truth, close enough to answer questions none of the individual slices can. That is the whole idea. Treat every published statistic as a constraint, satisfy all of them simultaneously with a population of synthetic individuals and households, and you have stitched the slices back into a block.

Two things follow from taking this seriously. First, the more statistics you honour, the closer the facsimile, so the collection of constraints should never stop growing. Second, the facsimile is only as good as its weakest constraint, so every value that shapes the population must be traceable to where it came from, and anything that is a modelling judgement rather than a measurement must be marked as one.

What SparkyData is

SparkyData develops and publishes synthetic demographic datasets that combine information from many public statistical sources into a single, internally consistent population model. The records are generated synthetically and calibrated to published aggregate statistics, so that the relationships between demographics, household structure, employment, income, wealth, spending, housing and the other attributes remain realistic and mutually consistent. No record represents or identifies any real individual or household.

The first dataset covers the New York metropolitan area, chosen because it is large, unusually diverse, and among the most thoroughly studied populations in the world, which means the collection of constraints available to stitch together is deep. Each release contains thousands of synthetic people grouped into households, with the ambition to scale to millions, and a multi-decade annual history in which people are born, arrive, age, earn, leave and die at recorded rates. The list of attributes carried per record is long and will continue to expand as further published statistics are incorporated. New York is the starting point, not the scope. The method is general, and the intention is to apply it to other cities, then other countries.

Everything is deterministic given a seed. Every distribution in the data traces to a named published source recorded with its retrieval date and a cryptographic hash of the file. Every modelling constant is declared in one registry as a measurement, a value derived arithmetically from measurements, or an assumption with its reasoning.

How it is built

The method has five steps, and the order matters: the demographic core is inherited from real records, everything modelled sits on top of it, and the constraint solver runs last. This section gives each step in plain terms. The next five go deeper.

1. Sample whole households from real microdata. The joint distribution of the core demographic, economic and housing variables is inherited from real, disclosure-protected records rather than modelled.
2. Calibrate the sample to the published margins, at the household and person level simultaneously, by solving a constrained optimisation over record weights.
3. Layer on the attributes the microdata does not carry, each drawn conditional on the attributes already fixed and renormalised to its own published marginal.
4. Check every record and every aggregate against the constraint system and repair violations minimally, to a fixed point.
5. Extend the cross-section into an open-cohort history whose year-by-year anchors are recorded series, with the provenance of every year carried on every row.

Step one: real joint structure, inherited

The single most consequential decision is where the demographic core comes from. There are two schools. One synthesises individuals attribute by attribute, drawing each from its published marginal and imposing correlations through a model. The other samples whole records from a public-use microdata file and adjusts their weights. The first school is fully general and fully wrong in the joints unless every pairwise and higher-order relationship has been modelled, which is never the case. The second school is limited to the variables the microdata carries, and in exchange gets every joint among those variables for free, including the ones nobody thought to model, because the records are real.

SparkyData takes the second path for the core. Households are drawn from the census microdata records covering the study area, with probability proportional to their survey weight, and everyone in a drawn household comes along. A drawn household therefore arrives with its own internal structure: the retired teacher’s plausible pension, tenure and commute, and the spouse whose age and education fit hers, because the combination was observed rather than generated. Family relationships are then made explicit as symmetric edges between people, spouse and partner, parent and child, sibling, grandparent, in-law and housemate, with each edge recording whether the relationship was stated in the survey or inferred from household composition.

The draw itself uses systematic probability-proportional-to-size selection rather than independent draws. The weighted records are laid along a line in random order, the line is divided into as many equal intervals as households are needed, and one record is selected in each. The realised sample lands far closer to the weighted marginals than independent sampling would, at any size, which means the later calibration has less work to do and the design effect of the weights stays small.

The cost of this choice is that the microdata’s geography is coarse. Public-use microdata areas are the smallest unit at which records are released, and each covers a population the size of a small city. Nothing in the dataset supports claims below that scale, and nothing in the dataset should be used to make them.

Step two: calibration as constrained optimisation

A sample of records drawn from state-level microdata files has to be re-weighted to match the study area’s published margins. This is the classical survey-calibration problem, and it is solved here in its classical form. Given starting weights $w^0$ and a set of published margins, find the weights $w$ that are closest to $w^0$ under a chosen distance and satisfy every margin exactly:

$$\min_{w}\; \sum_i D(w_i, w_i^0) \quad \text{subject to} \quad \sum_i w_i\, x_{ik} = T_k \;\; \text{for every margin } k,$$

where $x_{ik}$ is how much of category $k$ record $i$ carries and $T_k$ is the published total. The Lagrangian conditions reduce to a small nonlinear system in one multiplier per margin, and that system is solved by Newton’s method with backtracking. Convergence is judged on the achieved shares, which is what the user actually asked for, not on the multipliers.

Two details matter. The first is the choice of distance. With the raking distance the solution is the familiar iterative proportional fitting fixed point, and weights can run away. With the logit distance the ratio $w_i / w_i^0$ is confined to a bounded interval as part of the solution, which is different from solving and then clipping. Clipping after the fact silently breaks the very margins the procedure claims to have fitted. Bounding inside the objective keeps the margins exact and the weights sane at the same time.

The second is how person-level statistics are honoured when the units being weighted are households. A margin on people, the share who are foreign-born, say, is not controlled by household weights unless the household’s contribution to that margin is made explicit. Each household is therefore given one carrier column per person category holding the count of its members in that category, and the calibration constrains the weighted sum of carriers. Household margins and person margins become rows of the same constraint matrix and are fitted in one pass. This matters because household composition is correlated with almost every person-level attribute: people in larger households differ systematically from people in smaller ones, so a sample that is right at the household level is biased at the person level unless both are fitted together.

The calibration reports the design effect of the resulting weights, $\text{deff} = n \sum_i w_i^2 / (\sum_i w_i)^2$, and the effective sample size $n/\text{deff}$. A design effect that climbs is the signal that the margins being asked for are far from the sample and the population is leaning on a few heavily up-weighted records. It is recorded with every build and reused, as the next sections explain, to size the tolerance bands the population is later held to.

Step three: the layers

Everything the microdata does not carry is drawn conditional on what it does, and each layer is then renormalised so its own published marginal holds. The conditioning is what makes the layer coherent with the person it is attached to; the renormalisation is what makes the layer agree with the survey it came from. Four layers illustrate the range of techniques.

Wealth. The national wealth survey is the only public microdata on household balance sheets, and it has no geography below the region. The bridge has three planks. First, condition rather than average: every household is matched to a survey cell defined on age band, education, tenure and income band, and the cell’s net-worth quantile curve is used as the conditional distribution. Cells are walked down a fallback ladder, dropping the least informative conditioning variable at each rung, until a cell has enough donors to estimate its quantiles, and the rung used is recorded on the household so that a thin-cell draw can be identified later rather than silently trusted. Second, draw a rank rather than a number. The household’s income rank within its cell, $u_y$, is carried into a wealth rank $u_w$ through a Gaussian copula,

$$u_w = \Phi\!\left(\rho\,\Phi^{-1}(u_y) + \sqrt{1-\rho^2}\; z\right), \qquad z \sim \mathcal{N}(0,1),$$

and the cell’s quantile curve is interpolated at $u_w$. The correlation $\rho$ is the residual rank correlation measured inside the survey cells, not the marginal income-wealth correlation, because the cell already conditions on income band. Applying the marginal correlation on top of an income-conditioned curve counts the dependence twice, and the population’s realised income-wealth correlation comes out well above the survey’s. Third, anchor the house locally. The primary residence is the largest asset on most balance sheets and precisely the asset on which a national survey is most wrong about an expensive metro, so the survey-implied home value is replaced with the household’s actual reported property value from the census microdata, and the mortgage is rescaled to preserve the donor’s loan-to-value ratio. The remaining assets and debts are scaled to preserve the donor’s composition, and the balance sheet is reconciled so that its components sum exactly. The upper tail is handled separately: a survey top-codes its wealthiest respondents, so above the highest tabulated quantile the curve is extended with a Pareto tail whose index is implied by the cell’s own quantile ratios.

Rent regulation. A housing survey publishes the share of rentals that are regulated. Assigning that share uniformly across renters would put regulated tenants in buildings that cannot legally contain them. Regulation is instead assigned only within the eligible building stock, defined by construction year and units in structure, both of which the microdata carries, and the published share is then hit within that eligible pool. The pool is deliberately drawn a little wider than the strict statutory rule, because units built later can enter regulation through tax-incentive programmes.

Spending. The consumer expenditure survey publishes metro-average spending shares by category. Applying the average to every household would have a low-income household and a very high-income household allocating their budgets identically, the opposite of the oldest empirical regularity in economics. Each category’s share is instead shifted along an Engel curve, $s_c(y) \propto \bar{s}_c\,(y/\bar{y})^{\varepsilon_c}$, with elasticities below zero for necessities and above zero for discretionary categories, and the shares renormalised. Housing is not drawn at all: the microdata already carries each household’s actual rent or owner costs, which beats any average, so the survey shares are re-based over the non-housing categories. Total outlay is bounded by income net of estimated tax and saving, and the categories are scaled to reconcile. The elasticities are modelling assumptions and are registered as such.

Categorical layers. Religion, hobbies and time budgets, health behaviours and civic attributes come from surveys that publish shares, sometimes by age, race or borough. Each is drawn as a categorical distribution tilted by the conditioning variables the survey itself reports, then renormalised to the published metro marginal. Where a survey reports a rate for a subgroup, that rate becomes a target in its own right. Where no survey reports a rate at all, the value is an assumption and is flagged on the record, not only in the documentation. Names are drawn conditional on race, ethnicity, sex and birth cohort from the published name frequency files, with surnames propagated through households. Employers are fictional firms generated to match the published establishment counts by industry, county and size class, and are labelled as fictional in the data.

The ordering principle across all of these is the same. Inherited variables are never changed by a layer. A layer conditions on inherited variables and on layers drawn before it, never on layers drawn after it. A cross-tabulation between two modelled layers is therefore a product of two models and is only as real as the conditioning variables they share, a point the validation section returns to.

Step four: the constraint system

Generating a believable population is a constraint satisfaction problem with three families of constraint pulling against one another.

Record constraints are invariants that must hold within one row or between a row and its parent. A person’s year of arrival cannot precede their birth. A renter has no mortgage. Someone holding a bachelor’s degree is not twelve. A household row must agree with its member rows on size and composition.

Aggregate constraints are the published statistics. A share of the population, a median over a subgroup, a quantile, a rank correlation, a whole two-way table: each is a statistic $S(X, w)$ of the population and its weights that must sit within a band $B$ of a target $T$.

Trajectory constraints apply across a person’s history. Age increments by exactly one. Education never decreases. Nobody earns after they have left. Income and wealth do not move by an order of magnitude in a year without an event that explains it.

Satisfying any one family is easy. Draw people independently and the aggregates drift. Force the aggregates by reweighting and individuals start to look odd. Repair individuals one by one and the aggregates move again. Satisfying all three at once is the actual problem.

Each constraint carries a severity, and the severities are load-bearing. A hard constraint describes an impossibility; a build with any hard violation is broken and does not ship. A soft constraint describes an implausibility: a very old person working long hours exists, a large share of the population doing so does not. Soft violations are counted and reported, not fatal. A target constraint is a published number with a tolerance band, and the band widens as the target’s provenance weakens.

The solver is repair-based local search to a fixed point, not backtracking search. The choice is deliberate. Backtracking is the right tool when a solution must be constructed from nothing and any partial assignment might be a dead end. Here the starting point is already close to feasible, because the core records are real, and the aggregate constraints are global statistics that no single assignment decides. Search over individual assignments would be combinatorially hopeless and would throw away the one thing the sampling step bought. Instead the solver iterates

$$\text{check} \;\rightarrow\; \text{repair the worst violations} \;\rightarrow\; \text{re-fit the aggregates} \;\rightarrow\; \text{check},$$

until a pass changes nothing or a budget is exhausted. Repairs are applied in priority order, the cheap record-level ones first, because they alter individual fields, and the aggregate ones last, because they reassign whole categories and would otherwise undo the record repairs. Every repair is minimal by construction: it clamps or blanks the offending field, or demotes a value along an ordered scale with awareness of the person’s age, and it never redraws the record. Redrawing would destroy the joint distribution inherited from the real donor, which is exactly the thing the whole method exists to preserve. Repairs must also cascade: a demotion that makes one constraint pass can create a violation of another, and the fixed-point loop is what catches it.

The solver reports on whatever it could not satisfy. A population with a few constraints it could not reconcile and a note saying so is far more useful than one that silently pretends everything agrees.

Step five: time

A cross-section is a photograph. A panel is the film it was cut from, and the film has to be reconstructed against anchors that were actually observed.

There are more anchors than one might expect. The census microdata asks retrospective questions, and each answer pins a date: year of arrival in the country, state of birth, highest credential, and exact age. Published annual series supply year effects that an age profile cannot: a real median income series pins the level of every year, a wealth series pins the trajectory of net worth, and a metro unemployment series pins each year’s labour market so that recessions actually happen in the panel.

Series with provenance. A multi-decade panel needs a target for every metric in every year, and statistics do not arrive that way. The wealth survey is triennial. The housing survey is roughly triennial. Microdata begins only part-way through the period. So most metric-years are not observed and must be predicted, and the rule is that a predicted value is never allowed to look like a measured one. Every observation carries a provenance, and every value produced by a series carries the provenance of how it was produced: observed for a published estimate in that year, interpolated between two published estimates, extrapolated outside the published range, modelled when derived from another series, and assumption when there is no source at all. Shares are interpolated on the logit scale so they stay bounded. Extrapolation beyond the observed window uses a damped trend: a slope fitted on a short window says almost nothing about a point decades away, so the projected change after $h$ years is $\text{slope} \cdot \sum_{k=1}^{h} \varphi^k$ with $\varphi < 1$, which converges to a finite limit instead of running away. Each provenance is assigned a confidence, and the confidence widens the tolerance band any target from that year is held to.

Walking people through time. Fixed for life: birth year, sex, race and ethnicity, birthplace, year of entry. Monotone: age and education, with education walked backwards using typical credential ages, so a middle-aged graduate did not hold the degree as a teenager. Modelled per year: income, employment state, household net worth and debts, each in constant dollars using the metro price index. Income is walked from the observed base-year value:

$$y_{i,t} = y_{i,T}\;\frac{g(a_{i,t})}{g(a_{i,T})}\;\frac{I_t}{I_T}\;\exp\!\Big(\sum_{s=t}^{T-1}\epsilon_{i,s}\Big),$$

where $g$ is the cross-sectional age-income profile fitted on the base year itself, $I_t$ is the published real-income index, and the $\epsilon$ are persistent idiosyncratic shocks accumulated as a random walk in logs. A level penalty applies in years the person held a lower credential than their final one, applied once and never compounded. The labour market is imposed exactly: each year’s unemployment rate is enforced on that year’s labour force, with job losses assigned by a persistent, earnings-tilted exposure ordering so that the same marginal workers lose their jobs in bad years. Household wealth is walked along an age-period surface built from the wealth survey’s age-band medians across waves. Using an all-families index for the year effect and an age gradient for the life cycle would double-count the population’s ageing, because the all-families median of an earlier year already reflects that year’s younger families. One surface, no double count.

An open cohort. People are not a fixed group followed backwards. Each year, people enter by birth, by immigration dated by their year of entry, and by domestic in-migration; they leave by death and out-migration. The annual volumes are the recorded components of change for the study counties where those are published, and damped backcasts before. The subtle problem is the people who were here in an earlier year and are not here now. Walk a base-year cross-section backwards and every historical year is a sample of survivors, people who were still alive and still resident at the end, and survivors are not a random subset: they skew away from the old, who died, and from young adults, who moved. Left uncorrected, this makes every historical year look younger and more settled than it was. The fix is to align each year to the recorded age structure. For every year the census publishes the population of each study county by age band; scaled to the synthetic population’s size, that says how many people should be present in each band. The survivors fill part of each band, and the gap is filled with leavers: each a copy of a real base-year person of the same age band, so that sex, race, education, income and nativity are jointly consistent with age; each opening a spell in the year the band needs them and exiting in a year drawn from the age-specific hazard, conditional on exiting before the base year. Leaver income and wealth are walked through the spell with the same profiles, year effects and shocks as everyone else. Every person-year carries the weakest provenance among the series that shaped it, so a predicted year can never be mistaken for a measured one.

What is assumed

Every number the generator uses is declared in one registry with its source or its reasoning, and the registry is audited so that nothing can be added without provenance. Assumptions are flagged on the records they touch, not only in the documentation, and they are the places a reader should look hardest.

The important ones are these. The wealth layer is a national survey conditioned on local demographics and anchored to local home values; the conditioning absorbs most of the difference between the metro and the country, but not all of it, and the conditioning variables do not include everything a reader might wish to cross wealth with. Flows before the observed window and debt before the earliest published series are damped-trend backcasts, and the earliest panel years lean heavily on them. Participation rates for activities no survey measures are assumptions. So are the age profiles of mortality and fertility until the city’s vital statistics are incorporated, the factor converting published net migration into gross departures, the Engel elasticities, the persistence of income shocks, and the copula’s residual correlation, which is measured but is measured nationally. Preferences included for realism, such as favourite cuisine or news source, are assumptions and are marked so.

Nonetheless, everything a test or a model is most likely to depend on, the joint of age, sex, race, education, occupation, income, tenure, commute and household structure, is inherited from real records and is not an assumption at all. That is why the demographic core comes first, and why no layer is allowed to change it.

How it is validated

A build that has not been validated has not been built. Validation is not a step at the end; it is the same constraint system, run again on the finished population, with the bar set higher than the solver was asked to clear.

Every statistic is a record. A published statistic enters the system as a structured target, never as a number in code. The target records the source it came from, the survey period, the statistic type (share, count, mean, median, quantile, correlation, or a full two-way table), the filter defining the subpopulation it describes, the filter defining its denominator, the value, the published margin of error where one exists, and whether the value is a measurement or an assumption. A share of renters in the city is a different target from a share of renters in the metro, and the filters make that difference explicit. Targets are kept in reviewable files that trace to the source manifest, and each one becomes an aggregate constraint automatically.

Bands, not points. A target is met when $|S - T| \le B$. The band combines two sources of uncertainty in quadrature:

$$B = \sqrt{B_{\text{pub}}^2 + B_{\text{samp}}^2}, \qquad B_{\text{pub}} = \frac{\text{MOE}}{c(\text{provenance})}, \qquad B_{\text{samp}} = z \sqrt{\frac{p(1-p)}{n_{\text{eff}}}},$$

where $c$ is the confidence attached to the target’s provenance, so an interpolated or extrapolated value is held to a looser standard than a measured one, and $n_{\text{eff}}$ is the effective size of the actual denominator after the design effect recorded at calibration. A share over renters in the city has the sampling error of the city-renter count, not of the whole population. For a quantile the sampling term is $z\sqrt{q(1-q)/n_{\text{eff}}}\,/\,f(x_q)$, with the density $f$ estimated from the neighbouring quantiles of the achieved distribution, which is what makes a ninetieth-percentile band several times wider than a median’s. The reason for the sampling term is fairness: a finite synthetic draw cannot reproduce a share to within a survey’s margin of error, and holding it to that standard would flag honest sampling noise as calibration failure and send the solver chasing it.

Joints, not only margins. Margins can all be met while the joint structure is wrong, and that failure is invisible to marginal checks. So the two-way tables that matter most, age by education, age by employment, tenure by income quintile, household size by tenure, race by region, are compared cell by cell against the weighted microdata tables using the standardised root mean square error,

$$\text{SRMSE} = \frac{\sqrt{\tfrac{1}{K}\sum_{k=1}^{K}(\hat{p}_k - p_k)^2}}{\tfrac{1}{K}\sum_{k} p_k},$$

with a band equal to pure sampling noise for a population of this size. This is the check that sees a population with the right margins and the wrong joints, and it is deliberately the tightest test in the suite.

Three levels, three verdicts. Verification runs at the level of the individual (no impossible facts on any record, including cross-table agreement between a household and its members), the population (every target within its band, every joint within noise), and the panel (every trajectory coherent, every year’s aggregates within the band for that year’s provenance). The acceptance criterion is strict on what can be strict and honest about the rest: zero hard violations, and every target whose published value is a genuine measurement within its band. A build that fails either does not ship. Soft violations and misses on interpolated or extrapolated targets are reported with their counts and examples, broken out by provenance, so that “most targets met” can always be read as “of which this many were actually measured”.

Out of sample. A population that reproduces the statistics it was calibrated to has demonstrated consistency, not accuracy. So the verification also compares the population against figures that were never calibration inputs: decennial census values for the panel’s historical years, and published statistics deliberately held out of the target set. Matching those is a genuine test rather than a restatement of the fit.

How the system is updated. When a new published statistic is incorporated, it is registered in the source manifest with its retrieval hash, entered as a target with its filters and margin of error, and the build is re-run. Three outcomes are possible. The population already satisfies the new target within band, which is evidence that the layers conditioning on shared variables are coherent. The population misses it and the solver can reach it by minimal repair, in which case the target joins the constraint set and the verification report records the before and after. Or the new target conflicts with an existing one, and no population can satisfy both. That third case is treated as information, not as a nuisance: it almost always means two sources define a population, a denominator or a period differently, and the resolution is to fix the filters so both targets describe what their sources actually measured, never to force one of them. The discipline is the same one that governs the build: nothing is hand-tuned in the output, only in the declared constraints.

What validation cannot see. A cross-tabulation between two modelled layers is validated only at each layer’s marginal, because no published joint exists to check it against. Any use of the dataset that rests on such a joint should say so. The same is true of transitions in the panel: year-to-year movements in income and wealth are model walks anchored to published levels, and where an independent longitudinal source exists for a transition rate it should be used to check the panel before the panel is used for anything else.

What it is for

This is a dataset for asking questions that the published slices cannot answer on their own. Not because the slices are wrong, but because each answers one question about one cut of the population in one year, and the interesting questions live in the intersections and along the time axis.

How many households hold a particular combination of tenure, income and wealth. What the balance sheet of a regulated tenant looks like, and how it differs from a market-rate neighbour of the same age. Who in a shared household is related to whom, and whether the adult child or the parent is the one carrying the rent. Which households could afford to buy tomorrow and which are one shock from missing a payment. How many people who lived in the city two decades ago are still here, how many aged in place, and what happened to the incomes of an arrival cohort over its first twenty years. Whether an income trajectory that looks like mobility in the aggregate is the same households moving up, or different households cycling through.

A published table answers each of these for one variable at a time. Joining separate aggregate tables cannot answer them at all, because the join does not exist. A synthetic population built as described here answers them directly, with an answer that is consistent with every published slice it was constrained by, and with an explicit label on every value saying whether it was inherited, modelled or assumed, and whether the year it describes was observed or predicted.

That same property makes it useful for the more familiar purposes. Loaded into a staging database, the joins behave, because a household’s people, jobs, balance sheet and spending were generated to agree with one another. Fed to an agent-based model, the population ages, earns and turns over at recorded rates. Shown in a demonstration, no one in it can be re-identified, because no one in it exists.

Where it goes next

The method is not specific to New York, and the constraint collection is not finished. Each release adds statistics as they are published and as further sources are incorporated, which tightens the facsimile. Each annual vintage adds a year to the observed window, which pushes the backcast boundary further into the past and turns interpolated years into observed ones. The attribute list per record grows as new layers are added, always under the same rules: condition on what is already fixed, renormalise to the published marginal, register every constant, and let the solver do the matching. And the same pipeline, pointed at another region’s microdata and that region’s collection of published slices, produces the same kind of population for another city or another country.

Every release ships with its verification report, its source manifest with retrieval dates and hashes, and a synthetic-data marker carried as a column in the data itself, so that the fact survives being loaded into a warehouse and stripped of its documentation.

Sources

1. U.S. Census Bureau — American Community Survey Public Use Microdata Sample — the demographic core; whole households are sampled from these records
2. Federal Reserve Board — Survey of Consumer Finances — the wealth layer's conditional distributions
3. NYC Housing and Vacancy Survey — rent regulation share and the eligible building stock
4. Bureau of Labor Statistics — Consumer Expenditure Surveys — metro spending shares by category
5. Federal Reserve Bank of New York — Household Debt and Credit — per-capita debt by type, by year