SAP Data Post-processing Framework

We developed a simple-but-powerful framework to simplify and automate everyday tasks when working with SAP data in Databricks.

Automated conversion of CDC event stream to SCD2 tables

If you use CDC (Changed Data Capture) based data ingestion tool like Azure Data Factory or SAP Data Intelligence, you will need to convert all those change events into a historized fact table.

By adding a few lines to the YAML config file of our framework, you can flatten the CDC events automatically. In the following example, the CDC events will be flattened using the primary key column MANDT.

The OPTYPE column values describe whether the event is an insert, update or delete.

cdc_ingest:
	my_table:
    source_path: "stage/raw/my_table.parquet"
		primary_keys: ["MANDT"]
    partition_by: ["AR_YEAR"]
		transformation: ["currency_shifting"] # fixing SAP floating point incompatibility

All the events get automatically converted into a fact table my_table

Automated data type conversion, schema enforcement

RAW data are usually stored as string values only. Our framework supports loading original SAP schemas and data types and applying them to the datalake raw data to produce Hive/Unity Catalog tables with the correct schema.

# generic data types mapping
data_type_conversions:
- abap_type: ACCP
  raw_type: STRING
  hive: VARCHAR
- abap_type: CHAR
  raw_type: STRING
  hive: VARCHAR
- abap_type: CURR
  raw_type: DECIMAL
  hive: DECIMAL

SAP tables & columns metadata to be transferred to Databricks

Simplified stream processing

By stream processing simplification

id: "abc123_billing_doc"
description: "Billing documents"
source_systems: ["vfq", "lena", "mariot", "xf99"]
primary_keys: ["ID_MA", "VB_ID", "XA"]
transformation_function: ["mycomp.billing_stream_transform"]

Data Reconciliation

← Return to homepage