Technical Companion: Building an AI-Ready Dataset from SAP Variant Configuration

By Sachin Shinde

Scope

This companion documents the engineering behind the flattened configuration dataset described in the blog post: the SAP tables involved, the join spine, the point-in-time validity logic, a simplified version of the query, and the options for replicating the source tables into a lakehouse. It is written for data engineers. The blog post covers the business case.

The table map: three layers

SAP stores a configured order across three layers. All three must be walked to produce one readable row per order line, characteristic, and chosen value.

Layer	Table	What it holds
Sales order	`VBAK`	Order header: document type, sales org, distribution channel, division
	`VBAP`	Order items: material, description, higher-level BOM item; `CUOBJ` links the item to its configuration instance
	`VBKD`	Sales document business data at header and item level
Configuration instance	`IBIN`	The configuration instance itself, with instance validity dates
	`IBINVALUES`	Links an instance to the values assigned in it
	`IBSYMBOL`	The assigned value symbols: characteristic key (`ATINN`), value (`ATWRT`), numeric value (`ATFLV`)
Characteristic dictionary	`CABN`	Characteristic master: name, data type, status, valid-from
	`CABNT`	Characteristic descriptions (language-dependent)
	`CAWN`	Allowed characteristic values, with value counter `ATZHL`
	`CAWNT`	Value descriptions (language-dependent)

This table set is a working baseline, not a fixed contract. The data structure can be extended as required. Add header fields from VBAK/VBKD (customer, currency, incoterms), pull pricing conditions, join master data such as MARA/MAKT, or bring in class and dependency tables when the use case calls for it. The join spine below stays the same; new sources hang off it.

The ATINN spine

Nearly every dictionary join runs on ATINN, SAP's internal characteristic number:

IBSYMBOL.ATINN → CABN.ATINN         (characteristic master)
              → CABNT.ATINN         (characteristic text)
              → CAWN.ATINN + ATWRT  (value master → ATZHL)
              → CAWNT.ATINN + ATZHL (value text)

Follow the spine and the model reads cleanly: symbol, then characteristic, then value, then descriptions.

Point-in-time correctness

Characteristics and their value descriptions are date-dependent. They change through whatever change-management process an organization runs, and SAP stores each change as a new version. Dictionary records (CABN, CABNT, CAWN, CAWNT) carry a valid-from date (DATUV) but no valid-to. The pattern:

Derive a valid-to per version with a LEAD() window over DATUV (partitioned by ATINN, and by ATWRT/ATZHL for value-level tables), subtracting one day from the next version's valid-from. The last version gets 9999-12-31.
Keep only the dictionary rows where the order's reference date falls between valid-from and valid-to.

A note on the reference date: each organization may use a different date for validity, depending on its change-management and pricing practices. Common choices are the pricing date, the order creation date, or the requested delivery date. This document uses the pricing date throughout as the example.

This is what makes a two-year-old order read with the option list that existed when it was priced, not today's. Two additional time rules from the production job:

Instance validity. Configuration instances (IBIN) carry their own valid-from and valid-to. Only the currently valid instance version is kept.
Pricing-date fallback. When an order item has no pricing date of its own, it inherits the pricing date of its higher-level BOM item.

The query, simplified

One row per order line by characteristic by chosen value. Pricing date is assumed resolved onto the order line per the fallback rule above.

SELECT
    vbap.vbeln                             AS sales_order,
    vbap.posnr                             AS order_item,
    vbap.matnr                             AS material,
    cabn.atnam                             AS characteristic,
    cabnt.atbez                            AS characteristic_desc,
    COALESCE(cawnt.atwtb, ibsymbol.atwrt)  AS chosen_value,
    pricing_date                           -- order's point-in-time reference date
FROM vbap
JOIN      ibin       ON ibin.instance       = vbap.cuobj           -- item -> configuration instance
LEFT JOIN ibinvalues ON ibinvalues.in_recno = ibin.in_recno       -- instance -> assigned values
LEFT JOIN ibsymbol   ON ibsymbol.symbol_id  = ibinvalues.symbol_id -- value -> symbol (ATINN, ATWRT)
LEFT JOIN cabn       ON cabn.atinn          = ibsymbol.atinn       -- characteristic master
LEFT JOIN cabnt      ON cabnt.atinn         = ibsymbol.atinn
                    AND cabnt.spras         = 'E'                  -- characteristic text (English)
LEFT JOIN cawn       ON cawn.atinn          = ibsymbol.atinn
                    AND cawn.atwrt          = ibsymbol.atwrt       -- value master -> ATZHL
LEFT JOIN cawnt      ON cawnt.atinn         = ibsymbol.atinn
                    AND cawnt.atzhl         = cawn.atzhl
                    AND cawnt.spras         = 'E'                  -- value text (English)
WHERE vbap.cuobj <> '000000000000000000';                          -- configurable items only

The production version applies the point-in-time filter from the previous section at each dictionary join.

Production hardening notes

The production job (a PySpark pipeline producing a table we call vc_choice_flat here) adds several rules the simplified query omits:

CDC awareness. Every source read filters on the replication change flag (header__change_oper IN ('I','N','U')). Records flagged as deletes set an active-record flag to 0 so downstream MERGEs can retire them. A last-updated timestamp travels with every row.
Client filter. All reads are scoped to the productive SAP client (MANDT).
Numeric characteristics. For characteristics with data type NUM, the value comes from the internal floating-point field (ATFLV on IBSYMBOL), not from CAWN/CAWNT text lookups. The value-text joins are skipped for NUM types.
Deletion and language filters. Dictionary reads exclude records with the deletion indicator set (LKENZ <> 'X') and take language-dependent texts in English (SPRAS = 'E').
BOM-aware product rollup. Sub-items are attributed to their higher-level BOM item, so options roll up to the sellable product rather than the component line.
Performance. Small dictionary tables are broadcast-joined. The large fact-side tables drive the shuffle.

The output grain: one row per sales order line by characteristic by chosen value, with plain-English names and descriptions, faithful to the moment the order was priced.

Step zero: replicating the SAP tables into the lakehouse

The query assumes the SAP tables already live in the lakehouse as Delta tables with a change feed. The pipeline is built for change data capture, not a nightly full dump. Options, by landscape and licensing:

SAP Business Data Cloud (BDC) Connect for Databricks. GA in 2025. Bi-directional, live, zero-copy sharing through Delta Sharing, with no pipelines and no duplication. The modern default if you are moving to BDC.
SAP SLT (Landscape Transformation Replication Server). Trigger-based CDC replicating tables like VBAP, IBIN, and CABN in near real time, including inserts, updates, and deletes.
ODP-based connectors (for example, the Azure Data Factory SAP CDC connector). Full and delta extraction via SAP's Operational Data Provisioning framework. Caveat: SAP Note 3255746 restricts third-party ODP API use for some source editions, so confirm licensing before building.
Partner CDC tools. BryteFlow, Fivetran (HVR), Qlik Replicate, Theobald, and SNP all land SAP data into Delta with log- or trigger-based CDC.

Whichever route you take, the pattern is the same medallion architecture:

Bronze: land the raw SAP tables as Delta via CDC.
Silver / Gold: run the flattening join to produce the final table. The CDC flag drives incremental MERGE operations, so only changed data is reprocessed.

The replication method is a swappable detail. The modeling is what makes the data trustworthy.

References

Databricks: Announcing the GA of SAP Business Data Cloud Connect to Databricks. https://www.databricks.com/blog/announcing-general-availability-sap-business-data-cloud-connect-databricks
SAP News: SAP and Databricks, A Bold New Era of Data and AI. https://news.sap.com/2025/02/sap-databricks-open-bold-new-era-data-ai/