Grafana

Important Capabilities

Capability	Status	Notes
Column-level Lineage	✅	Enabled by default
Detect Deleted Entities	✅	Enabled by default
Extract Ownership	✅	Enabled by default
Extract Tags	✅	Enabled by default
Platform Instance	✅	Enabled by default
Table-Level Lineage	✅	Enabled by default

This source extracts the following:

• Folders
• Dashboards
• Charts
• Tags
• Owners
• Lineage information

CLI based Ingestion

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

Field	Description
service_account_token ✅ string(password)	Grafana API token
url ✅ string	URL of Grafana instance (e.g. https://grafana.company.com)
auto_tag_dimensions boolean	Automatically tag dimension fields in charts Default: False
auto_tag_measures boolean	Automatically tag measure fields in charts Default: False
platform_instance string	Platform instance for DataHub
platform_instance_map map(str,string)
env string	The environment that all assets produced by this connector belong to Default: PROD
connection_to_platform_map map(str,PlatformConnectionConfig)
connection_to_platform_map.key.platform ❓ string	Platform to connect to
connection_to_platform_map.key.database string	Database name
connection_to_platform_map.key.platform_instance string	Platform instance
connection_to_platform_map.key.env string	Environment
stateful_ingestion StatefulStaleMetadataRemovalConfig	Base specialized config for Stateful Ingestion with stale metadata removal capability.
stateful_ingestion.enabled boolean	Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False Default: False
stateful_ingestion.remove_stale_metadata boolean	Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled. Default: True

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "GrafanaSourceConfig",
  "description": "Base configuration class for stateful ingestion for source configs to inherit from.",
  "type": "object",
  "properties": {
    "env": {
      "title": "Env",
      "description": "The environment that all assets produced by this connector belong to",
      "default": "PROD",
      "type": "string"
    },
    "platform_instance_map": {
      "title": "Platform Instance Map",
      "description": "A holder for platform -> platform_instance mappings to generate correct dataset urns",
      "type": "object",
      "additionalProperties": {
        "type": "string"
      }
    },
    "stateful_ingestion": {
      "$ref": "#/definitions/StatefulStaleMetadataRemovalConfig"
    },
    "url": {
      "title": "Url",
      "description": "URL of Grafana instance (e.g. https://grafana.company.com)",
      "type": "string"
    },
    "service_account_token": {
      "title": "Service Account Token",
      "description": "Grafana API token",
      "type": "string",
      "writeOnly": true,
      "format": "password"
    },
    "platform_instance": {
      "title": "Platform Instance",
      "description": "Platform instance for DataHub",
      "type": "string"
    },
    "auto_tag_dimensions": {
      "title": "Auto Tag Dimensions",
      "description": "Automatically tag dimension fields in charts",
      "default": false,
      "type": "boolean"
    },
    "auto_tag_measures": {
      "title": "Auto Tag Measures",
      "description": "Automatically tag measure fields in charts",
      "default": false,
      "type": "boolean"
    },
    "connection_to_platform_map": {
      "title": "Connection To Platform Map",
      "description": "Map of Grafana connection names to their upstream platform details",
      "default": {},
      "type": "object",
      "additionalProperties": {
        "$ref": "#/definitions/PlatformConnectionConfig"
      }
    }
  },
  "required": [
    "url",
    "service_account_token"
  ],
  "additionalProperties": false,
  "definitions": {
    "DynamicTypedStateProviderConfig": {
      "title": "DynamicTypedStateProviderConfig",
      "type": "object",
      "properties": {
        "type": {
          "title": "Type",
          "description": "The type of the state provider to use. For DataHub use `datahub`",
          "type": "string"
        },
        "config": {
          "title": "Config",
          "description": "The configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19).",
          "default": {},
          "type": "object"
        }
      },
      "required": [
        "type"
      ],
      "additionalProperties": false
    },
    "StatefulStaleMetadataRemovalConfig": {
      "title": "StatefulStaleMetadataRemovalConfig",
      "description": "Base specialized config for Stateful Ingestion with stale metadata removal capability.",
      "type": "object",
      "properties": {
        "enabled": {
          "title": "Enabled",
          "description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
          "default": false,
          "type": "boolean"
        },
        "remove_stale_metadata": {
          "title": "Remove Stale Metadata",
          "description": "Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.",
          "default": true,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    },
    "PlatformConnectionConfig": {
      "title": "PlatformConnectionConfig",
      "type": "object",
      "properties": {
        "platform": {
          "title": "Platform",
          "description": "Platform to connect to",
          "type": "string"
        },
        "database": {
          "title": "Database",
          "description": "Database name",
          "type": "string"
        },
        "platform_instance": {
          "title": "Platform Instance",
          "description": "Platform instance",
          "type": "string"
        },
        "env": {
          "title": "Env",
          "description": "Environment",
          "type": "string"
        }
      },
      "required": [
        "platform"
      ],
      "additionalProperties": false
    }
  }
}

Code Coordinates

• Class Name: datahub.ingestion.source.grafana.grafana_source.GrafanaSource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Grafana, feel free to ping us on our Slack.