Skip to main content

JSON Schemas

Module json-schema

Incubating

Important Capabilities

CapabilityStatusNotes
DescriptionsExtracts descriptions at top level and field level
Detect Deleted EntitiesWith stateful ingestion enabled, will remove entities from DataHub if they are no longer present in the source
Extract OwnershipDoes not currently support extracting ownership
Extract TagsDoes not currently support extracting tags
Platform InstanceSupports platform instance via config
Schema MetadataExtracts schemas, following references

This source extracts metadata from a single JSON Schema or multiple JSON Schemas rooted at a particular path. It performs reference resolution based on the $ref keyword.

Metadata mapping:

  • Schemas are mapped to Datasets with sub-type Schema
  • The name of the Schema (Dataset) is inferred from the $id property and if that is missing, the file name.
  • Browse paths are minted based on the path

CLI based Ingestion

Install the Plugin

pip install 'acryl-datahub[json-schema]'

Starter Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

pipeline_name: json_schema_ingestion
source:
type: json-schema
config:
path: <path_to_json_file_or_directory or url> # e.g. https://json.schemastore.org/petstore-v1.0.json
platform: <choose a platform that you want schemas to live under> # e.g. schemaregistry
# platform_instance: <add a platform_instance if there are multiple schema repositories>
stateful_ingestion:
enabled: true # recommended to have this turned on

# sink configs if needed

Config Details

Note that a . is used to denote nested fields in the YAML recipe.

View All Configuration Options
Field [Required]TypeDescriptionDefaultNotes
path [✅]UnionType (See notes for variants)Set this to a single file-path or a directory-path (for recursive traversal) or a remote url. e.g. https://json.schemastore.org/petstore-v1.0.jsonNoneOne of string(file-path),string(directory-path),string(uri)
platform [✅]stringSet this to a platform that you want all schemas to live under. e.g. schemaregistry / schemarepo etc.None
platform_instance [✅]stringThe instance of the platform that all assets produced by this recipe belong toNone
use_id_as_base_uri [✅]booleanWhen enabled, uses the $id field in the json schema as the base uri for following references.None
env [✅]stringThe environment that all assets produced by this connector belong toPROD
uri_replace_pattern [✅]URIReplacePatternUse this if URI-s need to be modified during reference resolution. Simple string match - replace capabilities are supported.None
uri_replace_pattern.match [❓ (required if uri_replace_pattern is set)]stringPattern to match on uri-s as part of reference resolution. See replace fieldNone
uri_replace_pattern.replace [❓ (required if uri_replace_pattern is set)]stringPattern to replace with as part of reference resolution. See match fieldNone
stateful_ingestion [✅]StatefulStaleMetadataRemovalConfigBase specialized config for Stateful Ingestion with stale metadata removal capability.None
stateful_ingestion.enabled [❓ (required if stateful_ingestion is set)]booleanThe type of the ingestion state provider registered with datahub.None
stateful_ingestion.ignore_new_state [❓ (required if stateful_ingestion is set)]booleanIf set to True, ignores the current checkpoint state.None
stateful_ingestion.ignore_old_state [❓ (required if stateful_ingestion is set)]booleanIf set to True, ignores the previous checkpoint state.None
stateful_ingestion.remove_stale_metadata [❓ (required if stateful_ingestion is set)]booleanSoft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.True

Configuration Notes

  • You must provide a platform field. Most organizations have custom project names for their schema repositories, so you can pick whatever name makes sense. For example, you might want to call your schema platform schemaregistry. After picking a custom platform, you can use the put platform command to register your custom platform into DataHub.

Code Coordinates

  • Class Name: datahub.ingestion.source.schema.json_schema.JsonSchemaSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for JSON Schemas, feel free to ping us on our Slack