DATAMIMIC — Governed Test Data for Regulated Enterprises
Permalink: DATAMIMIC — Governed Test Data for Regulated Enterprises
This repository contains the DATAMIMIC Community Edition (CE). MIT-licensed, Python-native, MCP-ready.
CE is fully usable standalone for deterministic synthetic data generation and PII-aware pseudonymization. The Enterprise Platform adds governed workflows, PII scanning, role-based access, audit logging, scheduling, multi-system execution, and the full operational layer that regulated enterprises require.
👉 Enterprise Platform: datamimic.io | 📘 Docs: docs.datamimic.io | 📅 Book a strategy call: datamimic.io/contact
🤖 AI agent? Start at
AGENTS.mdand use the project CLI: preserve new intent asmodel.dm.json, submit an early best attempt viadatamimic scaffold ... --format json, repair from the structured issues, declare an expectation per stated requirement, and stop onverified=true. Existing raw XML uses lint plus bounded dry-run.
What is DATAMIMIC?
DATAMIMIC CE is the open-source deterministic data engine at the core of the DATAMIMIC Enterprise Platform. It is usable standalone for synthetic data generation and PII-aware pseudonymization in any local, CI, or agent-driven workflow.
The Enterprise Platform adds the governed workflows, scanners, dashboards, and execution layer that regulated enterprises require for production-scale test-data operations.
Available in CE (this repo):
- Generate fully synthetic, deterministic datasets — model-driven, no source data required
- Pseudonymize staging/QA exports — deterministic (seeded) or privacy-maximized (non-seeded) field transformation; PII fields identified and modeled manually in the XML pipeline
- Execute single-system pipelines against PostgreSQL · MySQL · Oracle · MS SQL · SQLite · MongoDB · CSV · JSON · XML · XLSX · DbUnit · fixed-width (
.fcw) - Model behavior — weighted state machines, composite multi-field references, control flow (
,), and a scriptable memstore for staged aggregation - Emit provenance — append-only execution logs and per-output content hash for audit re-execution
- Guide agents — machine-readable capabilities, progressive reference queries, and one canonical CLI scaffold transaction; an optional MCP adapter exposes the same authoring service
The Enterprise Platform adds:
- PII scanner — probability-scored field detection with configurable thresholds via DataWorkbench
- Multi-system execution — Oracle, MongoDB, Kafka in coordinated workflows with referential integrity
- Industry message templates — EDIFACT / SWIFT MT / HL7 v2.x / HL7 FHIR generated as deterministic test/training artefacts
- Governance layer — role-based dashboards, audit trails, approval flows, reusable enterprise templates, scheduler
- Performance core — Rust fastpath, ML/auto-regressive engine for complex distributions, keyset and manifest building
- On-premise / air-gapped deployment — podman-compose or Helm, with consulting-led rollout
Deployed in regulated EU banking environments for deterministic test data across Oracle, MongoDB, and Kafka pipelines. Reference customers available under NDA — see also datamimic.io case studies.
AI agents: author, verify, and run data models
Permalink: AI agents: author, verify, and run data models
The CLI is the baseline agent contract. Install CE with pip install datamimic-ce;
inside this checkout, use .venv/bin/datamimic so a stale global installation cannot
change the available schema or commands.
| Need | CLI tool | Contract |
|---|---|---|
| Discover the live structural surface | datamimic capabilities | Compact machine-readable JSON index by default; --full for the complete manifest, --section for one section. |
| Learn the Intent Model progressively | datamimic reference authoring, then datamimic reference authoring --category --kind | Start with the query catalogue, then load only the typed fragment needed. |
| Author a new model | Preserve model.dm.json; run datamimic scaffold model.dm.json --format json | One compile/lint/bounded-run/acceptance transaction per changed attempt. Stop on verified=true; generated XML is runtime output. |
| Work with existing raw XML | datamimic lint model.xml --format json, then datamimic dry-run model.xml --format json | Fix diagnostics, inspect bounded samples for intent, then use datamimic run model.xml only when real execution is requested. |
| Find a DSL detail | datamimic reference overview, then a narrow reference topic/name | Query the live model and rule registries instead of guessing elements, generators, scope, distributions, or rules. |
capabilities, authoring-reference projections, and the commands shown with
--format json return machine-readable JSON. On a failed scaffold attempt, change
model.dm.json using its structured validation issues, typed repair, or rule
diagnostics before retrying. A typed max_count remediation instead changes only
the bounded scaffold parameter to at least its reported minimum. Never repeat an
identical failed call. A successful scaffold result is terminal for authoring, so
do not lint or dry-run its generated XML again. Exact source fragments are
discoverable through queries such as --category source --kind memstore.
Optional MCP adapter
Permalink: Optional MCP adapter
When the calling environment already exposes DATAMIMIC MCP tools, they map to the same
canonical contracts and implementations: reference → datamimic_reference,
scaffold → datamimic_scaffold, lint → datamimic_check, and dry-run →
datamimic_run. Install the adapter with pip install "datamimic-ce[mcp]";
registration details belong in the MCP quickstart,
not in the authoring workflow. The adapter intentionally exposes only the four
canonical reference, scaffold, check, and bounded-run operations; domain generation
remains a Python/CLI capability rather than a parallel MCP authoring path.
Prompts to paste into your agent
Permalink: Prompts to paste into your agent
Author and verify a new model
Create the dataset I describe with DATAMIMIC.
Read AGENTS.md first. In a repository checkout use `.venv/bin/datamimic`;
otherwise use the current `datamimic` CLI. Preserve my intent as
`model.dm.json`; do not hand-write XML.
Start from the minimal valid document shape in AGENTS.md ("Authoring a new
model"). Two rules prevent most rejections: the top level allows ONLY
version, seed, products, expectations; product-level "kind"
(generated/source/time_series) is a different vocabulary from field-level
"kind" (increment, values, weighted, int_range, decimal_range, pattern, constant, script). Range fields take minimum/maximum, never min/max.
Submit EARLY: run `datamimic scaffold model.dm.json --format json` with your
best attempt after at most one discovery call. Repair from the structured
issues (path/code/message/allowed_fields) and diagnostics (fix_hint) — they
teach the schema faster than more discovery. Never resubmit an unchanged
document. If a remediation requests a larger max_count, retry scaffold with
at least that value without changing the intent.
Declare an expectation for every requirement I state (counts as exact_count
with a "count" field, uniqueness, allowed values, ranges, foreign keys) —
verified=true certifies only what you declared. Stop on verified=true; do
not lint or dry-run the generated XML. If I request real execution, save the
returned XML as a generated artifact and run that descriptor. Return the
model.dm.json path and concise verification evidence.
Relational hierarchy with referential integrity (fully supported — no XML needed)
Seed a relational dataset with referential integrity: 4 customers, each with
exactly 2 orders.
Customers get an incrementing unique id and a region from
{north, south, east, west}. Each order carries the REAL parent customer id
as a foreign key and an amount between 10.0 and 500.0.
Follow AGENTS.md's "Authoring a new model" and its structural recipes:
orders nest inside the customer product's "children" array; the FK field is
{"kind": "script", "script": "parent.id"} with a foreign_key role — a
randomly generated FK passes schema validation but fails per-parent-count
acceptance. Declare expectations for the customer count, customer id
uniqueness, exactly 2 orders per customer (per_parent_count), the
orders->customers foreign key, and the amount range. Stop on verified=true
and show the acceptance evidence.
Raw XML remains supported for existing descriptors (lint → dry-run → run; see AGENTS.md). For new models it is a last resort: only when a scaffold issue explicitly classifies the requirement as unsupported_intent should an agent hand-author XML, preserving that evidence.
CE vs Enterprise Platform
Permalink: CE vs Enterprise Platform
CE and EE are not the same engine with a feature flag. They share the DSL and determinism contract, but EE is an independently optimised execution engine built for enterprise-scale throughput and operational control.
Engine comparison
| Capability | Community Edition (CE) | Enterprise Platform (EE) |
|---|---|---|
| Deterministic data generation | ✅ | ✅ |
| Deterministic seeding in the DSL | ✅ entities + standalone literal `` (4.0.0) | ✅ same, plus sandboxed script expressions and stdlib random calls |
| Pseudonymization — seeded (GDPR Art. 4(5); supports Art. 25 / Art. 32) | ✅ manual model | ✅ automated via DataWorkbench |
| Pseudonymization — non-seeded (privacy-maximized) | ✅ manual model | ✅ automated via DataWorkbench |
| Python API + XML pipelines | ✅ | ✅ |
| Domain models: Finance, Healthcare, Demographics | ✅ | ✅ |
| Time-series generation (``, ISO 8601, prefix-stable) | ✅ | ✅ |
| MCP server for AI agent integration | ✅ | ✅ |
| CLI + local execution | ✅ | ✅ |
| Scale | millions of records via Python multiprocessing (and optional Ray) | designed for billion-record workloads — Rust fastpath, optimised multi-process execution, and keyset/manifest building on top of the shared Ray distribution layer |
| PII scanner | ❌ | ✅ probability-scored field detection, configurable threshold, DataWorkbench integration |
| Runtime configuration profiles | ❌ | ✅ Performance · Balanced · Flexibility |
| Memory management | standard | optimised for high-volume batch and streaming |
| Logging granularity | flat execution log | configurable: minimal · standard · deep nested tracing |
| Nested structure evaluation | basic | deep nested generation with extended condition + ruleset evaluation |
| Importer / exporter logging | ❌ | per-stage logging for importers and exporters |
| Error handling | standard exceptions | structured error catalog with recovery strategies |
| Rust fastpath | ❌ | performance-critical paths in Rust |
| Keyset and manifest building | ❌ | reads live DB schemas to build coordinated multi-table generation plans |
| ML / auto-regressive engine | ❌ | combine statistical models with conditions, rulesets, validators for complex distributions |
Platform capabilities (EE only)
Permalink: Platform capabilities (EE only)
| Capability | EE |
|---|---|
| Multi-user collaboration | ✅ |
| Role-based access control (RBAC) | ✅ |
| Audit logs + provenance dashboards | ✅ |
| PII scanner — probability scoring, threshold-based field flagging | ✅ |
| DataWorkbench — visual field mapping and pseudonymization model builder | ✅ |
| Reusable enterprise template library | ✅ |
| Scheduled execution + task runner | ✅ |
| CI/CD pipeline integration (Tosca, Jenkins, GitLab) | ✅ |
| Multi-system execution: Oracle, MongoDB, Kafka | ✅ |
| Template engine: schema-aware editors for EDIFACT, SWIFT MT, HL7 v2.x, and HL7 FHIR — customer-uploadable specs, further industry formats built per engagement on the same framework | ✅ |
| Audit-evidence artefacts for GDPR Art. 30 records, PCI DSS 4.0 Req. 6.5.5 (test data) reviews, and — for US Covered Entities / Business Associates — HIPAA §164.312 evidence packs | ✅ |
| On-premise deployment + air-gapped environments | ✅ |
| LSP-powered IDE tooling for DSL authoring | ✅ |
👉 Explore the Enterprise Platform | Book a platform demo
EE runtime profiles
Permalink: EE runtime profiles
The EE core supports three runtime configuration profiles, selectable per execution context:
| Profile | Optimises for | Typical use case |
|---|---|---|
| Performance | Maximum throughput via Rust fastpath, optimised multi-process execution, and Ray-based distribution | Bulk generation at billion-record volumes to PostgreSQL, Oracle, Kafka |
| Balanced | Throughput + full audit logging | Standard enterprise pipeline runs with compliance requirements |
| Flexibility | Deep nested evaluation, extended condition and ruleset processing | Complex domain models with ML engine combinations, multi-level referential structures |
Logging depth is independently configurable per profile — from minimal (throughput-optimised) to full nested tracing across importers, exporters, and generation stages.
EE template engine
The EE template engine generates industry-standard financial messages from DATAMIMIC models. The workbench parses uploaded message samples, auto-detects the message type, and validates edits against the registered spec version in real time.
Capabilities
- Spec-aware form editing — segments and elements rendered as structured forms with mandatory/optional indicators, per-field value suggestions, and inline custom-extension support
- Strict validation against baked spec versions, with segment- and element-level error reporting
- Advisory mode when a spec is unregistered or in draft — editing stays enabled, validation continues as guidance
- Round-trip between the structured form view and the authoritative template text — no fidelity loss
- Download / adjust / upload your own spec — customers can extend or override the baked spec catalogue without waiting for a release
- Live structure tree + preview for every edit
- File auto-detection — upload an existing message, the editor identifies the type and loads the matching spec
Format coverage
| Format | Coverage |
|---|---|
| UN/EDIFACT | Schema-aware form editor; spec versions and subsets per engagement |
| SWIFT MT | Schema-aware form editor; categories and SR versions per engagement |
| HL7 v2.x | Schema-aware form editor; versions per engagement |
| HL7 FHIR | Schema-aware form editor for FHIR resources (Patient, Observation, Encounter, …); profiles per engagement |
| Further industry formats (ISO 20022 / MX, vertical dialects) | Built into the editor catalogue per customer engagement, on the same framework |
Customers can extend the spec catalogue between releases by downloading, adjusting, and uploading their own spec files directly.
Generated messages are deterministic and traceable to their source model, and syntactically valid against the registered spec. They are intended for test and training environments only — they are not network-validated and must not be transmitted on production SWIFTNet or EDI networks. See the SWIFT CSP note below.
Who is DATAMIMIC for?
Permalink: Who is DATAMIMIC for?
Enterprise Platform (EE)
Permalink: Enterprise Platform (EE)
| Role | What DATAMIMIC solves |
|---|---|
| QA / Test Manager | Eliminate manual test data requests. Self-service, governed, always ready. |
| Business Analyst | Define data requirements in business-readable models — no scripting needed. |
| Platform / DevOps Engineer | Integrate deterministic test data generation into CI/CD and scheduled pipelines. |
| Compliance / Audit | Full audit trail for every generation run. Regulator-ready logs, no production data exposure. |
| Enterprise Architect | One governed standard across Oracle, MongoDB, Kafka, flat files, and custom systems. |
Community Edition (CE)
Permalink: Community Edition (CE)
Developers and data engineers who need deterministic synthetic data generation or PII-aware pseudonymization in local environments, CI pipelines, or agent-driven workflows. PII field identification is manual — the EE DataWorkbench automates this step.
Why deterministic generation matters
Permalink: Why deterministic generation matters
Most test data tools produce random output. That breaks regression tests, audit trails, and cross-team reproducibility.
DATAMIMIC's determinism contract (CE):
- Same engine version + same model + same seed = byte-identical output, every run, every machine. Holds at three layers: the
generate_domainfacade, every domain service called directly, and every literal generator that accepts anrng=argument. Verified per-service on every CI run viatests_ce/architecture/test_service_replay_determinism.py. - DSL-level seeding:
makes the whole model deterministic — every seed-lessderives a reproducible child RNG from it, andoverrides it for that block (no seed anywhere → wall-clock random). Verified by [`tests_ce/integration_tests/test_determinism_seed_scenarios`](https://github.com/rapiddweller/datamimic/tree/development/tests_ce/integration_tests/test_determinism_seed_scenarios). As of 4.0.0 the same seed also reaches standalone literal generators (), typed/pattern keys,DateTimeGenerator, and cross-pageuniquepicks — machine-independently. - Source reads:
distribution="ordered"reads a data source in stable
