Research

Generated engineering data is usually either accepted uncritically or dismissed as unusable. The position taken here is that a generated corpus can be a scientific object: measurable, auditable, and improvable, provided its provenance, failure modes, and verification path ship with it. GenBOM is the first report in that program. BOMwiki-Bench is the instrument it builds, from the frozen v1.0 snapshot of this site's own graph: 192,471 nodes, 263,907 quantity-bearing edges, 4,879 products decomposed into 50,843 assemblies and 136,749 parts across 53 domains, all authored by a documented model pipeline.

GenBOM: Benchmarking and Auditing a Large Generated Product-Structure Graph. MathproBro and protosphinx, ERP•AI Research Team, technical report, July 2026. Download the paper (PDF, 10 pages)

The central result

Can a generative model author usable engineering benchmark data? Decomposed carefully, the answer splits in a way neither optimists nor skeptics predict. On a population matched by domain and size, a classifier restricted to structural features cannot tell generated product trees from human ones. Give the same classifier the text and separation becomes perfect.

Generated vs human separability, decomposed by feature set (logistic classifier, matched n = 28, within-stratum permutation test). Structure sits on chance; text alone separates perfectly. Detectability of generated BOMs is a writing-style artifact, not a structural signature, and any detector that does not ablate text from structure is measuring prose.

The third result closes the triangle: models trained on the generated graph do not yet carry to real product structure. Completion skill measured on the benchmark collapses to the popularity floor when scored against human-authored BOMs. Structural plausibility and distributional fidelity are different properties, and the gap between them is now a number.

The benchmark

dataset v1.0	value
nodes	192,471 (4,879 products / 50,843 assemblies / 136,749 parts)
edges	263,907 (HAS_PART, quantity-bearing; a DAG after excluding four labeled self-loops)
product domains	53 (10 to 498 products each)
nodes with text summaries	175,690
typed artifact registry	1,350 items, six disjoint classes
frozen	2026-06-22; the live site changes nightly and is not the benchmark

Four tasks: BOM completion (rank the removed true child against 500 fixed negatives; MRR and Hits@k), where-used retrieval (part to products), domain transfer (53-way), and graph repair (detect planted corruptions and the registry's natural artifacts; gradient-boosted structural features reach ROC-AUC 0.929 on the planted stratum). Splits, negatives, loaders, and evaluators are versioned and shipped.

Why evaluation is the hard part

Part reuse in a real catalog is heavy-tailed: most parts appear in one product, while a small population of generic components appears in thousands. That hub structure is realistic for engineered products and fatal for naive evaluation, because a ranker that just predicts the popular part inherits its score from the degree distribution rather than from any understanding of the product.

Complementary CDF of part reuse: four orders of magnitude from the median part (in-degree 1; 98% of parts have a single parent) to the most-reused component. Under uniform negatives a pure popularity ranker reaches MRR 0.99 on hub edges; the benchmark neutralizes this with degree-matched negatives, leakage-audited splits (product-held-out, three domain-held-out rotations, hub-stress), and a per-split feasibility floor that observed popularity must sit within 0.02 of. On all six splits it lands within 0.009.

Results

question	test	result
structural signature?	matched structure-only classifier, within-stratum permutation (100,000 relabelings)	AUC 0.510, 95% CI [0.278, 0.734], p = 0.92: chance
text signature?	same matched population, text-style features	AUC 1.000 (full-feature 0.995): complete separation
does skill transfer?	completion fusion trained on the graph, scored on human BOMs	MRR 0.622 (product-held-out) drops to 0.095
baseline headroom	best heuristic fusion vs popularity floor	0.59 to 0.63 against a floor near 0.09

The validated GraphSAGE reference (reproduces published ogbl-collab numbers before being trusted here) tells the protocol's cautionary tale in one row: MRR 0.96 on hub edges under uniform negatives, 0.004 to 0.015 on every leakage-resistant split. Structure-only GNNs collapse exactly where the held-out subtrees enter evaluation with empty training neighborhoods, which is why the open modeling challenge this benchmark defines is inductive, text-aware link prediction. The GNN baseline is validated, not victorious.

Transfer to real BOMs

Completion models trained on the generated graph, scored on the human control corpus with degree-matched negatives built in the human-candidate space. Both the fusion ranker and the text baseline collapse to the popularity floor; only text cosine retains any signal, and only on the deep trees (MRR 0.264). Skill on BOMwiki-Bench does not, at present, transfer to real BOMs.

The artifact registry

Generation artifacts are labeled and shipped rather than silently repaired, because they are the raw material of the audit and repair tasks. The six classes are disjoint:

artifact class	count
kind/granularity-inconsistent edges	822
quantity outliers (1,000 < q ≤ 10,000)	282
duplicate-name candidates (node-level, 69 clusters)	142
quantity outliers (q > 10,000)	93
zero-quantity edges	7
self-loops	4
total	1,350

Manual inspection reframed the largest class: kind-inconsistent edges are modeling-convention violations (a node typed part that itself has children), not physical impossibilities. The duplicate-name clusters are the product-graph analogue of the duplicate-node leakage documented in popular graph benchmarks; every cluster lies within one product subtree, so the shipped splits keep each cluster on one side of the train/test boundary automatically.

The human control corpus

The control corpus is 75 human-authored BOM trees from license-verified public repositories (71 flat electronics BOMs and 4 multi-level trees, including the NASA JPL Open Source Rover parts list), plus 5 deep CAD assemblies at depth 3 to 6 extracted from STEP structure, with a 48-tree quarantine retained as an audit trail. Each tree records its source and keeps its original license. The harvest produced its own finding: machine-readable multi-level product BOMs barely exist in public. Deep hierarchy lives in proprietary PLM systems, CAD files, and PDF manuals, so the deep human stratum is thin by nature, the shallow comparison is primary by design, and the 3-to-6-level trees of BOMwiki-Bench are structurally unusual among published artifacts.

From audit to community verification

Because the content is generated, plausibility is an empirical question rather than an assumption. The paper freezes the instrumentation before any labels exist: a three-way rubric (plausible / implausible / cannot-judge) with a six-way error taxonomy, a stratified 617-item analysis packet, a companion set covering all 1,350 registry artifacts, and an annotation harness with pre-registered integrity rules (agreement gated at Cohen's kappa of at least 0.6; labels never suggested or prefilled). The framework is designed to execute as community verification on this site, edge by edge, with redundancy and gold items substituting for annotator training. Accumulated labels ship as versioned additions; v1.0 stays frozen. The deterministic layer already runs in bomwiki-intelligence, which screens every proposed edit.

Limits, stated plainly

One artifact, one generator. Results characterize this graph, not generated engineering data at large.
The structure-only null is weak by construction: the matched shallow stratum has n = 28 and is powered only for large effects (AUC of roughly 0.75 or more). The deep stratum (n = 16) is too small for a valid permutation test at all.
The strength of text-based completion is plausibly generator-specific: model-written names are stylistically consistent from parent to child in a way real SKU-style BOM text is not.
The graph is a benchmark object, not industrial ground truth. Nothing in it should inform real sourcing or engineering decisions.
Plausibility is not yet human-validated; the framework ships first, the labels come from its execution.

Cite

MathproBro and protosphinx. GenBOM: Benchmarking and Auditing a Large Generated Product-Structure Graph. ERP•AI Research Team, technical report, July 2026. Dataset v1.0, frozen 2026-06-22. bomwiki.com/research/genbom.pdf

@techreport{genbom2026,
  title       = {GenBOM: Benchmarking and Auditing a Large Generated
                 Product-Structure Graph},
  author      = {MathproBro and protosphinx},
  institution = {ERP AI Research Team},
  year        = {2026},
  month       = jul,
  note        = {Dataset v1.0, frozen 2026-06-22},
  url         = {https://bomwiki.com/research/genbom.pdf}
}

Questions and discussion: @MathproBro and @protosphinx on X.