Plugins

Discoverable plugins #

Discoverable plugins that are supported out of the box are defined in the discovery.yml manifest, which can be found inside the Meltano repository at src/meltano/core/bundle/discovery.yml.

Making a custom plugin discoverable #

If you’ve added a custom plugin (or variant) to your project that could be discoverable and supported out of the box for new users, please contribute its description to this file to save the next user the hassle of setting up the custom plugin. The GitLab Web IDE makes it very easy to contribute changes without requiring you to leave your browser.

Discoverable plugin definitions in discovery.yml have the same format as custom plugin definition in your meltano.yml project file, so a copy-paste is usually sufficient. The format and further requirements are laid out in more detail below.

Besides the new definition in discovery.yml, a new discoverable plugin should be documented in the Extractors or Loaders section of the MeltanoHub, which live inside the MeltanoHub repository under /_extractors and /_loaders. However, it is not required to include documentation when you contribute a new plugin definition to discovery.yml, as members of the core team are happy to any missing docs themselves as part of the review process.

Plugin definitions #

At a minimum, a plugin definition must have a name and a namespace, and at least one variant definition with a pip_url (its pip install argument).

It is recommended to add a label, logo_url, and description to the plugin definition, and docs and repo URLs to the variant definition(s).

Most of the time, variant definitions should also have a settings array with setting definitions.

Additionally:

capabilities should be specified for extractor variants,
non-default variant executable names can be specified using executable, and
default values for plugin extras can be specified at the plugin definition level and further overridden at the variant definition level.

Variant definitions #

If a plugin will only ever have a single variant (as is typically the case for all types except for extractors and loaders), the variant definition can be embedded in the plugin definition (variant properties can be mixed in with plugin properties), and a variant name should not be specified using a variant key.

If a plugin currently only has a single variant, but more might be added later (as is typically the case for extractors and loaders), the variant definition can be embedded in the plugin definition, and a variant name should be specified using the variant key, matching the organization name on GitHub/GitLab, e.g. meltano, singer-io, or transferwise.

If multiple variants of a plugin are available, the plugin definition should have a variants array where each entry represents a variant definition with its own name, again matching the organization name on GitHub/GitLab. The first variant is considered the default, and the original variant supported by Meltano should be marked with original: true. Deprecated variants should be marked with deprecated: true.

Setting definitions #

Each extractor (tap) and loader (target) variant in the discovery.yml has a settings property. Nested under this property are a variable amount of individual settings. In the Meltano UI these settings are parsed to generate a configuration form. To improve the UX of this form, each setting has a number of optional properties:

- settings:
    - name: setting_name # Required (must match the connector setting name)
      aliases: [alternative_setting_name] # Optional (alternative names that can be used in `meltano.yml` and with `meltano config set`)
      label: Setting Name # Optional (human friendly text display of the setting name)
      value: "" # Optional (Use to set a default value)
      placeholder: Ex. format_like_this # Optional (Use to set the input's placeholder default)
      kind: string # Optional (Use for a first-class input control. Default is `string`, others are `integer`, `boolean`, `date_iso8601`, `password`, `options`, `file`, `array`, `object`, and `hidden`)
      description: Setting description # Optional (Use to provide inline context)
      tooltip: Here is some more info... # Optional (use to provide additional inline context)
      documentation: https://meltano.com/ # Optional (use to link to specific supplemental documentation)
      protected: true # Optional (use in combination with `value` to provide an uneditable default)
      env: SOME_API_KEY # Optional (use to inject `value` into plugin context with a custom environment variable name, in addition to the default env var name)
      env_aliases: [OTHER_ENV] # Optional (use to delegate alternative environment variables for overriding this setting's value)
      value_processor: nest_object # Optional (Modify value after loading it from source: env, meltano.yml, system database. Target type needs to match `kind`. Options: `nest_object`, `upcase_string`)
      value_post_processor: stringify # Optional (Modify loaded value before passing it to plugin. Target type does not need to match `kind`. Options: `stringify`)

Protected settings #

Until role-based access control is implemented in Meltano, we need to prevent user editing of certain settings from the UI.

Adopting a plugin #

When the maintainer of the default variant of a discoverable plugin becomes unresponsive to issues and contributions filed by the community, that plugin is considered up for adoption, which means that we are looking for a different variant of the plugin with a more engaged maintainer to become the new default.

This new variant can either be a fork of the original default variant, or an alternative implementation for the same source or destination, as long as it is actively maintained.

If you maintain or are aware of such a variant, please add it to your Meltano project as a custom plugin and make it discoverable, or file an issue so that the Meltano core team can assist you.

As a plugin’s primary maintainer, you do not have to spend a lot of time improving the plugin yourself. In fact, attracting more users and getting the community involved is likely to recude your personal maintenance burden, since you’ll receive contributions with bug fixes and new features that you will only be expected to review, not build yourself.

Local changes to `discovery.yml` #

When you need to make changes to discovery.yml, these changes are not automatically detected inside of the meltano repo during development. While there are a few ways to solve this problem, it is recommended to create a symbolic link in order ensure that changes made inside of the meltano repo appear inside the Meltano project you initialized and are testing on.

Get path for discovery.yml in the repo

Example: /Users/bencodezen/Projects/meltano/src/meltano/core/bundle/discovery.yml

Open your Meltano project in your terminal
Create a symbolic link by running the following command:

ln -s $YOUR_DISCOVERY_YML_PATH

Now, when you run the ls -l command, you should see something like:

bencodezen  staff   72 Nov 19 09:19 discovery.yml -> /Users/bencodezen/Projects/meltano/src/meltano/core/bundle/discovery.yml

Now, you can see your changes in discovery.yml live in your project during development! 🎉

`discovery.yml` version #

Whenever new functionality is introduced that changes the schema of discovery.yml (the exact properties it supports and their types), the version in src/meltano/core/bundle/discovery.yml and the VERSION constant in src/meltano/core/plugin_discovery_service.py must be incremented, so that older instances of Meltano don’t attempt to download and parse a discovery.yml its parser is not compatible with.

Changes to discovery.yml that only use existing properties do not constitute schema changes and do not require version to be incremented.

Plugin Development #

Taps & Targets Development #

Watch “How taps are built” for an explanation of how Singer taps (which form the basis for Meltano extractors) work, and what goes into building new ones or verifying and modifying existing ones for various types of APIs.

Then watch “How transforms are built” for an explanation of how DBT transforms work, and what goes into building new ones for new data sources.

For existing taps/targets #

We should be good citizen about these, and use the default workflow to contribute. Most of these are on GitHub so:

Fork (using Meltano organization)
Add a webhook to trigger the meltano/meltano pipeline.
Modify and submits PRs
If there is resistance, fork as our tap (2)

How to test a tap? #

We qualify taps with the capabilities it supports:

properties: the tap uses the old --properties format for the catalog
catalog: the tap uses the new --catalog format for the catalog
discover: the tap supports catalog extraction
state: the tap supports incremental extraction

Properties/Catalog #

You should look at the tap’s documentation to see which one is supported.

Discover #

Try to run the tap with the --discover switch, which should output a catalog on STDOUT.

State #

Try to run the tap connect and extract data first, watching for STATE messages.
Do two ELT run with target-postgres, then validate that:
1. All the tables in the schema created have a PRIMARY KEY constraint. (this is important for incremental updates)
2. There is no duplicates after multiple extractions

Troubleshooting #

Tables are lacking primary keys #

This might be a configuration issue with the catalog file that is sent to the tap. Take a look at the tap’s documentation and look for custom metadata on the catalog.

For taps/targets we create #

For tap development please use the tap cookiecutter template.
For target development please use the target cookiecutter template.
Use a separate repo (meltano/target|tap-x) in GitLab e.g. Snowflake: https://gitlab.com/meltano/target-snowflake
Publish PyPI packages of these package (not for now)
We could mirror this repo on GitHub if we want (not for now)

Transform & Models Development #

When you need to expose data to the user through Meltano UI, this often will require updating the transforms and models. At a high level:

Transforms will allow you to create the necessary PostgreSQL tables for users to query against
Models will determine the structure of what is exposed on the UI side

Transforms #

You can test local transforms in a project by adding them in a Meltano project’s transform > models > my_meltano_project directory.

Every transform file is a SQL file that will determine how the table is created. Some caveats include:

Rather than referring to the tables directly (i.e., analytics.gitlab_issues), the syntax uses ref to refer to tables
When joining two tables together, * seems to crash dbt. Instead, you should explicitly define every column. For example:

users.user_id as user_id,
users.user_name as user_name,
issues.month_closed as month_closed,
issues.year_closed as year_closed,

Once you’ve created your transforms, you can run it with the following command:

# Replace your extractors / targets with the appropriate ones
meltano elt tap-gitlab target-postgres --transform only

Models #

When updating the models that will appear in the UI, you can follow these steps:

Create table.m5o file that defines the UI columns that will appear on the UI
Update topic.m5o file to include the newly created model table
Compile model repo with python3 setup.py sdist
Go to your meltano.yml project file and replace pip_url with the file path to the targz file created
Run meltano install to fetch new settings
Refresh browser and you should now see your changes in the UI

Dashboard Development #

To create a dashboard plugin like https://gitlab.com/meltano/dashboard-google-analytics, follow these steps:

Set up the extractor and model you are creating dashboard(s) and reports for in your local Meltano instance.
Start Meltano UI.
Use the UI to create the desired reports based on the model’s designs. Name the reports appropriately, but don’t include the extractor name or label.
Create one or more new dashboard and add the reports to it. If you’re creating just one dashboard, name it after the extractor label (e.g. “Google Analytics”, not tap-google-analytics). If you’re creating multiple dashboards, add an appropriate subtitle after a colon (e.g. “Google Analytics: My Dashboard”).
Create a new plugin repository named dashboard-<data source> (e.g. dashboard-google-analytics).
Copy over setup.py, README.md, and LICENSE from https://gitlab.com/meltano/dashboard-google-analytics and edit these files as appropriate.
Move your newly created dashboards and reports from your local Meltano project’s analyze/dashboards and analyze/reports to dashboards and reports inside the new plugin repository.
Push your new plugin repository to GitLab.com. Official dashboard plugins live at https://gitlab.com/meltano/dashboard-....
Add an entry to src/meltano/core/bundle/discovery.yml under dashboards. Set namespace to the namespace of the extractor and model plugins the dashboard(s) and reports are related to (e.g. tap_google_analytics), and set name and pip_url set as appropriate.
Delete the dashboard(s) and reports from your local Meltano project’s analyze directory.
Ensure that your local Meltano instance uses the recently modified discovery.yml by following the steps under “Local changes to discovery.yml.
Run meltano add --include-related extractor <extractor name> to automatically install all plugins related to the extractor, including our new dashboard plugin. Related plugins are also installed automatically when installing an extractor using the UI, but we can’t use that flow here because the extractor has already been installed.
Verify that the dashboard(s) and reports have automatically been added to your local Meltano project’s analyze directory and show up under “Dashboards” in the UI.
Success! You can now submit a merge request to Meltano containing the changes to discovery.yml (and an appropriate CHANGELOG item, of course).

File Bundle Development #

To create a file bundle plugin like https://gitlab.com/meltano/files-dbt, follow these steps:

Create a new plugin repository named files-<service/tool> (e.g. files-airflow or files-docker).
Copy over setup.py, README.md, and LICENSE from https://gitlab.com/meltano/files.dbt and edit these files as appropriate.
Create a bundle directory with an empty __init__.py file.
Add all desired directories and files to the bundle directory. All of these files will be copied over into the Meltano project directory when the file bundle is added to the project.
Add all file paths under bundle to the package_data["bundle"] array in setup.py
Push your new plugin repository to GitLab.com. Official file bundle plugins live at https://gitlab.com/meltano/files-....
Add an entry to src/meltano/core/bundle/discovery.yml under files. Set name and pip_url as appropriate, and if applicable, set namespace to the namespace of the plugin the file bundle is related to (e.g. dbt).
If any files are to be updated automatically when meltano upgrade is run, add an update object with [file path]: True entries for each file.
Success! You can now submit a merge request to Meltano containing the changes to discovery.yml (and an appropriate CHANGELOG item, of course).