Skip to main content

Glue

Module glue​

Certified

Important Capabilities​

CapabilityStatusNotes
Detect Deleted Entities✅Enabled by default when stateful ingestion is turned on.
Domains✅Supported via the domain config field
Platform Instance✅Enabled by default

Note: if you also have files in S3 that you'd like to ingest, we recommend you use Glue's built-in data catalog. See here for a quick guide on how to set up a crawler on Glue and ingest the outputs with DataHub.

This plugin extracts the following:

  • Tables in the Glue catalog
  • Column types associated with each table
  • Table metadata, such as owner, description and parameters
  • Jobs and their component transformations, data sources, and data sinks

IAM permissions​

For ingesting datasets, the following IAM permissions are required:

{
    "Effect": "Allow",
    "Action": [
        "glue:GetDatabases",
        "glue:GetTables"
    ],
    "Resource": [
        "arn:aws:glue:$region-id:$account-id:catalog",
        "arn:aws:glue:$region-id:$account-id:database/*",
        "arn:aws:glue:$region-id:$account-id:table/*"
    ]
}

For ingesting jobs (extract_transforms: True), the following additional permissions are required:

{
    "Effect": "Allow",
    "Action": [
        "glue:GetDataflowGraph",
        "glue:GetJobs",
    ],
    "Resource": "*"
}

plus s3:GetObject for the job script locations.

CLI based Ingestion​

Install the Plugin​

pip install 'acryl-datahub[glue]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: glue
  config:
    # Coordinates
    aws_region: "my-aws-region"

sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

View All Configuration Options
Field [Required]TypeDescriptionDefaultNotes
aws_access_key_id [✅]stringAWS access key ID. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.None
aws_endpoint_url [✅]stringAutodetected. See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.htmlNone
aws_profile [✅]stringNamed AWS profile to use. Only used if access key / secret are unset. If not set the default will be usedNone
aws_proxy [✅]map(str,string)None
aws_region [✅]stringAWS region code.None
aws_secret_access_key [✅]stringAWS secret access key. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.None
aws_session_token [✅]stringAWS session token. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.None
catalog_id [✅]stringThe aws account id where the target glue catalog lives. If None, datahub will ingest glue in aws caller's account.None
emit_s3_lineage [✅]booleanWhether to emit S3-to-Glue lineage.None
extract_owners [✅]booleanWhen enabled, extracts ownership from Glue directly and overwrites existing owners. When disabled, ownership is left empty for datasets.True
extract_transforms [✅]booleanWhether to extract Glue transform jobs.True
glue_s3_lineage_direction [✅]stringIf upstream, S3 is upstream to Glue. If downstream S3 is downstream to Glue.upstream
ignore_unsupported_connectors [✅]booleanWhether to ignore unsupported connectors. If disabled, an error will be raised.True
platform [✅]stringThe platform to use for the dataset URNs. Must be one of ['glue', 'athena'].glue
platform_instance [✅]stringThe instance of the platform that all assets produced by this recipe belong toNone
use_s3_bucket_tags [✅]booleanIf an S3 Buckets Tags should be created for the Tables ingested by Glue. Please Note that this will not apply tags to any folders ingested, only the files.None
use_s3_object_tags [✅]booleanIf an S3 Objects Tags should be created for the Tables ingested by Glue.None
env [✅]stringThe environment that all assets produced by this connector belong toPROD
aws_role [✅]UnionType (See notes for variants)AWS roles to assume. If using the string format, the role ARN can be specified directly. If using the object format, the role can be specified in the RoleArn field and additional available arguments are documented at https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html?highlight=assume_role#STS.Client.assume_roleNoneOne of string,union(anyOf),string,AwsAssumeRoleConfig
aws_role.ExternalId [❓ (required if aws_role is set)]stringExternal ID to use when assuming the role.None
aws_role.RoleArn [❓ (required if aws_role is set)]stringARN of the role to assume.None
database_pattern [✅]AllowDenyPatternregex patterns for databases to filter in ingestion.{'allow': ['.*'], 'deny': [], 'ignoreCase': True}
database_pattern.allow [❓ (required if database_pattern is set)]array(string)None
database_pattern.deny [❓ (required if database_pattern is set)]array(string)None
database_pattern.ignoreCase [❓ (required if database_pattern is set)]booleanWhether to ignore case sensitivity during pattern matching.True
domain [✅]map(str,AllowDenyPattern)A class to store allow deny regexesNone
domain.key.allow [❓ (required if domain is set)]array(string)None
domain.key.deny [❓ (required if domain is set)]array(string)None
domain.key.ignoreCase [❓ (required if domain is set)]booleanWhether to ignore case sensitivity during pattern matching.True
table_pattern [✅]AllowDenyPatternregex patterns for tables to filter in ingestion.{'allow': ['.*'], 'deny': [], 'ignoreCase': True}
table_pattern.allow [❓ (required if table_pattern is set)]array(string)None
table_pattern.deny [❓ (required if table_pattern is set)]array(string)None
table_pattern.ignoreCase [❓ (required if table_pattern is set)]booleanWhether to ignore case sensitivity during pattern matching.True
profiling [✅]GlueProfilingConfigConfigs to ingest data profiles from glue tableNone
profiling.column_count [❓ (required if profiling is set)]stringThe parameter name for column count in glue table.None
profiling.max [❓ (required if profiling is set)]stringThe parameter name for the max value of a column.None
profiling.mean [❓ (required if profiling is set)]stringThe parameter name for the mean value of a column.None
profiling.median [❓ (required if profiling is set)]stringThe parameter name for the median value of a column.None
profiling.min [❓ (required if profiling is set)]stringThe parameter name for the min value of a column.None
profiling.null_count [❓ (required if profiling is set)]stringThe parameter name for the count of null values in a column.None
profiling.null_proportion [❓ (required if profiling is set)]stringThe parameter name for the proportion of null values in a column.None
profiling.row_count [❓ (required if profiling is set)]stringThe parameter name for row count in glue table.None
profiling.stdev [❓ (required if profiling is set)]stringThe parameter name for the standard deviation of a column.None
profiling.unique_count [❓ (required if profiling is set)]stringThe parameter name for the count of unique value in a column.None
profiling.unique_proportion [❓ (required if profiling is set)]stringThe parameter name for the proportion of unique values in a column.None
profiling.partition_patterns [❓ (required if profiling is set)]AllowDenyPatternRegex patterns for filtering partitions for profile. The pattern should be a string like: "{'key':'value'}".{'allow': ['.*'], 'deny': [], 'ignoreCase': True}
profiling.partition_patterns.allow [❓ (required if partition_patterns is set)]array(string)None
profiling.partition_patterns.deny [❓ (required if partition_patterns is set)]array(string)None
profiling.partition_patterns.ignoreCase [❓ (required if partition_patterns is set)]booleanWhether to ignore case sensitivity during pattern matching.True
stateful_ingestion [✅]StatefulStaleMetadataRemovalConfigBase specialized config for Stateful Ingestion with stale metadata removal capability.None
stateful_ingestion.enabled [❓ (required if stateful_ingestion is set)]booleanThe type of the ingestion state provider registered with datahub.None
stateful_ingestion.ignore_new_state [❓ (required if stateful_ingestion is set)]booleanIf set to True, ignores the current checkpoint state.None
stateful_ingestion.ignore_old_state [❓ (required if stateful_ingestion is set)]booleanIf set to True, ignores the previous checkpoint state.None
stateful_ingestion.remove_stale_metadata [❓ (required if stateful_ingestion is set)]booleanSoft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.True

Compatibility​

To capture lineage across Glue jobs and databases, a requirements must be met – otherwise the AWS API is unable to report any lineage. The job must be created in Glue Studio with the "Generate classic script" option turned on (this option can be accessed in the "Script" tab). Any custom scripts that do not have the proper annotations will not have reported lineage.

Code Coordinates​

  • Class Name: datahub.ingestion.source.aws.glue.GlueSource
  • Browse on GitHub

Questions​

If you've got any questions on configuring ingestion for Glue, feel free to ping us on our Slack